GitLab

Filesystem refactor

Implement additional data checks

Build with no cgo

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

dont fail copy on not found error

Interfaces

add contexts to commit and precommit

Fix challange counts

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* consume newly-extracted go-paramfetch

* add basic CircleCI config, and use TestDownloadParams to hydrate cache before running tests

* make deps, not make deps lotus

* use Makefile target when testing + set env variable for verbose rust logging

* drop gosumtest

* drop short mode

* add some documentation

* deals: Sending initial proposal works

* deals: Almost sealing client data

* deals: Use temp files for AddPiece

* deals: Upstream bitswap changes

* pond: Basic message display in Block window

* move poller to sector store

* sectorstore: Address review feetback

* storageminer: Initial PaymentVerify implementation

* Wire up more proper ticket generation and verification logic

* Replace most marshaling with codegen

* Command to list sealed blocks

* update sectorbuilder

* Import proofs for paramfetch

* Extract go-fil-proofs

* Fix sectorbuilder poRepProofPartitions

* retrieval: Make types more spec complaiant

* Simpler paramfetch

* Merge commit 'c57c47ffb5695f7536306c4f3ab05c9a98adb1c6' as 'extern/rleplus'

* Add rleplus

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Update sectorbuilder

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Update sectorbuilder

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Sector Commitment tracker

* jsonrpc: include method name in error log

* node: Basic graceful shutdown

* repo: Close datastore in Close

* storageminer: Better context handling

* cleaning up a few types

* Rought PoST method

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* update go-sectorbuilder

* use new sectorbuilder file interfaces

* fix tests

* Almost working new post code

* jsonrpc: Channel buffeering

* fix websocket closing

* pass those tests

* fix websocket closing again

* Devnet 3; Builtin bootstrap; NAT Port Map

* remove VDFs from tickets

* use faster bls code

* Update filebeat

Change log of rpc buffer as I want to set up alert when it goes to high

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Add more info to storage-miner info command output

* remove empty const block

* Update build scripts

Remove outdated

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Cleanup imports after rename

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Cleanup imports after rename

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* dont hang requests if websockets server shuts down

* REST file import endpoint

* on chain deals: Get things to actually run!

* on chain deals: Expose more chain state in pond

* on chain deals: Deals make it to the chain

* on chain deals: Put dealIDs in CommitSector messages

* WIP: updating to new proofs code

* WIP: updating to new proofs code

* should use the new parameters

* very basic sector seal scheduling

* Fix TestSealAndVerify

* storageminer: Handle uncommited sectors on start

* storageminer: Restart sealing on restart

* storageminer: More correct listing of sealed sectors

* fix panic when close miner

* Update sectorbuilder, v15 params

* WIP Interactive PoRep

* Some more progress on interactive porep

* use reflect select

* move select

* specific ipfs gateway

* use IPFS_GATEWAY

* more refactoring for interactive porep scheduling

* Fix sectorbuilder.VerifySeal

* Move statestore to lib

* Get interactive porep sector sealing mostly working

* Get interactive porep sector sealing mostly working

* Strip unused functionality from sectorstore

* statestore: Use reflect for mutators

* statestore: More generic keys

* Use state store for sectors

* Some smaller fixes

* INTERACTIVE PROEP IS ALIVE

* Update sectorbuilder

* Update sectorbuilder

* Put WorkerThreads on sectorbuilder.Config

* rate-limit some sectorbuilder ops

* Track down all the uses of cboripld and eliminate them

* Update go-sectorbuilder again

* events: Plumb context to callbacks

* fix retrieval protocol error by wrapping stream in peeker

* WIP fixing tests

* Fix statestore.List

* Mostly fix deals

* Improve errors around deal handling

* deals: Set correct Refs

* Create filler deals

* WIP: trying to write a test to reproduce the storage deal error

* Add method to query latest deal state

* fail test if deal errors

* deals: cleanup client state machine

* cborrpc -> cborutil

* Make multiple deals per almost work

* update go-sectorbuilder

* sectorbuilder: use standalone methods

* sectorbuilder: Also test PoSt

* sectorbuilder: Always create directories

* Wip fixing a thing

* sectorbuilder: Use StandaloneWriteWithAlignment

* Storage miner API improvements

* keep track of last used sector id across restarts

* Use the same dir in TestAcquireID

* padreader: Some more testcases

* sectorbuilder: Call destroy in DI module

* Update go-sectorbuilder with gpu fixes

* sectorbuilder: apply some review suggestions

* Test to reproduce post error after restart

* Update sectorbuilder with a fix

* Update sectorbuilder

* WorkerCount on storageminer config

* storageminer: Throttle GeneratePieceCommitment in storeGarbage

* more tracing spans

* fix tests and add some more trace attributes

* Skip slow tests

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Rename to --include-test-params

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* wip

* parallel sectorbuilder test

* sectorbuilder: Call AcquireSectorId in sync

* Skip sectorbuilder tests on slow hardware

* StateAPI: account for slashing in StateMinerPower

* sectorbuilder: open FD later in AddPiece

* sectorbuilder: Drop some unused functions

* wip remote sectorbuilder workers

* remote-worker: wire up storage miner endpoints

* support remote SealPreCommit

* Stats for remote workers

* Working remote PreCommit

* WIP remote sector CommitSseal

* WIP: election post restructuring

* WIP: election post restructuring

* fix rspco serialization

* Swtich to xerrors

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Add lotus-gen, rewire genesis mining

* Add lotus-gen, rewire genesis mining

* More correct challangeCount calc

* WIP getting post in sectorbuilder_test to work

* use the correct sector sizes in places

* sectorbuilder: Measure thigs in TestSealAndVerify

* WIP trying to get election post to compute

* sectorbuilder: Drop stateful sectorbuilder refs

* sync: correct vrfBase for ticket check

* Copy over last sector ID key when migrating sectorbuilder

* replace go-bls-sigs and go-sectorbuilder with filecoin-ffi

- remove old submodules and add new submodule
- update build tooling to consume new unified static library
- update consumers of old libs to use new package

* replace go-bls-sigs and go-sectorbuilder with filecoin-ffi

- remove old submodules and add new submodule
- update build tooling to consume new unified static library
- update consumers of old libs to use new package

* update filecoin-ffi to v18 params

* update filecoin-ffi to v18 params

* More v18 updates

* v19 parameters

* filecoin-ffi master

* filecoin-ffi master

* WIP: uncomment out windowed post code, try to make it work

* actors: Fallback post progress

* storageminer: New fallback post scheduler

* Use ProvingSet for GetSectorsForElectionPost

* Some fixes and dev utils

* seal-worker: Handle cache

* Rework miner test setups to fix tests

* self review: some cleanup

* Fix unsealing, sector based data refs

* deals: Correctly set deal ID in provider states

* actually set unsealed path in sectorbuilder

* Buch of lint fixes

* use challangeCount as sampleRate in IsTicketWinner

* Update filecoin-ffi

* Update filecoin-ffi

* Update filecoin-ffi

* worker: Use system tar for moving cache around

* worker: Use system tar for moving cache around

* worker: Fix rebaining bugs

* paramfetch: Only pull necessary params

* more statticcheck!

* Update filecoin-ffi

* sectorbuilder: update PoRepProofPartitions

* there is no real correlation between challenge count and len(winners)

* Allow no local sectorbuilder workers

* Fix AddPiece with disabled local workers

* Pre-sealing holes

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Update filecoin-ffi

* seed: Trim cache

* Fix tests, circle and make ux nicer
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* flag blocks that are received too late

* Add lazy RLE+ decoding

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* More iterative algorithms

 - Add RunIterator and decoder from RLE
 - Add BitIterator and BitsFromRuns
 - Add BitsFromSlice
 - Add RunsFromBits

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Improve bitvector performance

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Improve benchmarks and fix bitvector iterator

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Add rle encoder

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Optimize and start wrapping it up

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Remove old bitvector

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Improve complex code and comment it

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Replace rleplus with rlepluslazy

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Fix typo in overflow check

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Some cleanup

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* sectorbuilder: Allow to restrict task types

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* sectorbuilder: Allow to restrict task types

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Update to correct version

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Close files in ExtractTar

* implement sector dir aggregator

* update ffi

* use that nice function i wrote

* this will pretty much always be nil

* support copying directories

* use a package

* Add short tests

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Move api struct to a seprate pkg

* fix target for ePoSt IsTicketWinner fn

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* fix sync tests

* Update FFI

* add option to symlink to presealed sectors

* fixup

* sectorbuilder: Fix proving on RO filesystem

* Update filecoin-ffi

* use actual symlink flag

* sectors: Handle sector state reload errors more gracefully

* Use filecoin-ffi master

* Update ffi to f261762

* sectorbuilder: check free space before creating sectors

* sectorbuilder: fs: address review

* fix(sectorbuilder): always cast fsstat.Bsize

fixes compilation issue under macos

* sectorbuilder: fix getpath

* sectorbuilder: Improve not enough space error

* circle: buildall on macos

* Wire up faults in fPoSt

* tear the world asunder

* temporarily move build into lib to prepare for extraction

* consume sectorbuilder from lotus

* port sectorbuilder from lotus

* downgrade to go-datastore 0.1.1 to match lotus
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Whyrusleeping <why@ipfs.io>
Co-authored-by: Jakub Sztandera <kubuxu@protonmail.ch>
Co-authored-by: Frank <wholery@163.com>
Co-authored-by: Jack Yao <yaoh.cn@gmail.com>
Co-authored-by: Henri <3359083+sternhenri@users.noreply.github.com>
Co-authored-by: Caesar Wang <dtynn@163.com>
Co-authored-by: Friedel Ziegelmayer <me@dignifiedquire.com>

* feat(post): standalone PoSt, without a sector builder

* feat(post): support PoSt generation w/out sector builder

* fix(paramcache): install from correct branch

* ci(darwin): don't run the tests with darwin - but do verify project builds and links

* switch to feat/election-post  for rust-fil-sector-buidler

* update sector-builder

* update api to match election post

* update from upstream

* update upstream

* update upstream

* use branch for param fetch

* update upstream

* feat(candidates): provided generated candidates to PoSt generator

* ci(cache): bump cache key + selectively generate params

* fix(candidates): use generated candidates in PoSt verification

* fix(timeout): bump go test timeout

pull in latest version of rust-fil-sector-builder, with new bellperson