replace old go-sectorbuilder with lotus' sectorbuilder + paramfetch (#61)

* deals: Sending initial proposal works

* deals: Almost sealing client data

* deals: Use temp files for AddPiece

* deals: Upstream bitswap changes

* pond: Basic message display in Block window

* move poller to sector store

* sectorstore: Address review feetback

* storageminer: Initial PaymentVerify implementation

* Wire up more proper ticket generation and verification logic

* Replace most marshaling with codegen

* Command to list sealed blocks

* update sectorbuilder

* Import proofs for paramfetch

* Extract go-fil-proofs

* Fix sectorbuilder poRepProofPartitions

* retrieval: Make types more spec complaiant

* Simpler paramfetch

* Merge commit 'c57c47ffb5695f7536306c4f3ab05c9a98adb1c6' as 'extern/rleplus'

* Add rleplus

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Update sectorbuilder

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Update sectorbuilder

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Sector Commitment tracker

* jsonrpc: include method name in error log

* node: Basic graceful shutdown

* repo: Close datastore in Close

* storageminer: Better context handling

* cleaning up a few types

* Rought PoST method

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* update go-sectorbuilder

* use new sectorbuilder file interfaces

* fix tests

* Almost working new post code

* jsonrpc: Channel buffeering

* fix websocket closing

* pass those tests

* fix websocket closing again

* Devnet 3; Builtin bootstrap; NAT Port Map

* remove VDFs from tickets

* use faster bls code

* Update filebeat

Change log of rpc buffer as I want to set up alert when it goes to high

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Add more info to storage-miner info command output

* remove empty const block

* Update build scripts

Remove outdated

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Cleanup imports after rename

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Cleanup imports after rename

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* dont hang requests if websockets server shuts down

* REST file import endpoint

* on chain deals: Get things to actually run!

* on chain deals: Expose more chain state in pond

* on chain deals: Deals make it to the chain

* on chain deals: Put dealIDs in CommitSector messages

* WIP: updating to new proofs code

* WIP: updating to new proofs code

* should use the new parameters

* very basic sector seal scheduling

* Fix TestSealAndVerify

* storageminer: Handle uncommited sectors on start

* storageminer: Restart sealing on restart

* storageminer: More correct listing of sealed sectors

* fix panic when close miner

* Update sectorbuilder, v15 params

* WIP Interactive PoRep

* Some more progress on interactive porep

* use reflect select

* move select

* specific ipfs gateway

* use IPFS_GATEWAY

* more refactoring for interactive porep scheduling

* Fix sectorbuilder.VerifySeal

* Move statestore to lib

* Get interactive porep sector sealing mostly working

* Get interactive porep sector sealing mostly working

* Strip unused functionality from sectorstore

* statestore: Use reflect for mutators

* statestore: More generic keys

* Use state store for sectors

* Some smaller fixes

* INTERACTIVE PROEP IS ALIVE

* Update sectorbuilder

* Update sectorbuilder

* Put WorkerThreads on sectorbuilder.Config

* rate-limit some sectorbuilder ops

* Track down all the uses of cboripld and eliminate them

* Update go-sectorbuilder again

* events: Plumb context to callbacks

* fix retrieval protocol error by wrapping stream in peeker

* WIP fixing tests

* Fix statestore.List

* Mostly fix deals

* Improve errors around deal handling

* deals: Set correct Refs

* Create filler deals

* WIP: trying to write a test to reproduce the storage deal error

* Add method to query latest deal state

* fail test if deal errors

* deals: cleanup client state machine

* cborrpc -> cborutil

* Make multiple deals per almost work

* update go-sectorbuilder

* sectorbuilder: use standalone methods

* sectorbuilder: Also test PoSt

* sectorbuilder: Always create directories

* Wip fixing a thing

* sectorbuilder: Use StandaloneWriteWithAlignment

* Storage miner API improvements

* keep track of last used sector id across restarts

* Use the same dir in TestAcquireID

* padreader: Some more testcases

* sectorbuilder: Call destroy in DI module

* Update go-sectorbuilder with gpu fixes

* sectorbuilder: apply some review suggestions

* Test to reproduce post error after restart

* Update sectorbuilder with a fix

* Update sectorbuilder

* WorkerCount on storageminer config

* storageminer: Throttle GeneratePieceCommitment in storeGarbage

* more tracing spans

* fix tests and add some more trace attributes

* Skip slow tests

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Rename to --include-test-params

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* wip

* parallel sectorbuilder test

* sectorbuilder: Call AcquireSectorId in sync

* Skip sectorbuilder tests on slow hardware

* StateAPI: account for slashing in StateMinerPower

* sectorbuilder: open FD later in AddPiece

* sectorbuilder: Drop some unused functions

* wip remote sectorbuilder workers

* remote-worker: wire up storage miner endpoints

* support remote SealPreCommit

* Stats for remote workers

* Working remote PreCommit

* WIP remote sector CommitSseal

* WIP: election post restructuring

* WIP: election post restructuring

* fix rspco serialization

* Swtich to xerrors

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Add lotus-gen, rewire genesis mining

* Add lotus-gen, rewire genesis mining

* More correct challangeCount calc

* WIP getting post in sectorbuilder_test to work

* use the correct sector sizes in places

* sectorbuilder: Measure thigs in TestSealAndVerify

* WIP trying to get election post to compute

* sectorbuilder: Drop stateful sectorbuilder refs

* sync: correct vrfBase for ticket check

* Copy over last sector ID key when migrating sectorbuilder

* replace go-bls-sigs and go-sectorbuilder with filecoin-ffi

- remove old submodules and add new submodule
- update build tooling to consume new unified static library
- update consumers of old libs to use new package

* replace go-bls-sigs and go-sectorbuilder with filecoin-ffi

- remove old submodules and add new submodule
- update build tooling to consume new unified static library
- update consumers of old libs to use new package

* update filecoin-ffi to v18 params

* update filecoin-ffi to v18 params

* More v18 updates

* v19 parameters

* filecoin-ffi master

* filecoin-ffi master

* WIP: uncomment out windowed post code, try to make it work

* actors: Fallback post progress

* storageminer: New fallback post scheduler

* Use ProvingSet for GetSectorsForElectionPost

* Some fixes and dev utils

* seal-worker: Handle cache

* Rework miner test setups to fix tests

* self review: some cleanup

* Fix unsealing, sector based data refs

* deals: Correctly set deal ID in provider states

* actually set unsealed path in sectorbuilder

* Buch of lint fixes

* use challangeCount as sampleRate in IsTicketWinner

* Update filecoin-ffi

* Update filecoin-ffi

* Update filecoin-ffi

* worker: Use system tar for moving cache around

* worker: Use system tar for moving cache around

* worker: Fix rebaining bugs

* paramfetch: Only pull necessary params

* more statticcheck!

* Update filecoin-ffi

* sectorbuilder: update PoRepProofPartitions

* there is no real correlation between challenge count and len(winners)

* Allow no local sectorbuilder workers

* Fix AddPiece with disabled local workers

* Pre-sealing holes

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Update filecoin-ffi

* seed: Trim cache

* Fix tests, circle and make ux nicer
License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* flag blocks that are received too late

* Add lazy RLE+ decoding

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* More iterative algorithms

 - Add RunIterator and decoder from RLE
 - Add BitIterator and BitsFromRuns
 - Add BitsFromSlice
 - Add RunsFromBits

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Improve bitvector performance

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Improve benchmarks and fix bitvector iterator

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Add rle encoder

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Optimize and start wrapping it up

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Remove old bitvector

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Improve complex code and comment it

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

* Replace rleplus with rlepluslazy

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Fix typo in overflow check

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Some cleanup

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* sectorbuilder: Allow to restrict task types

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* sectorbuilder: Allow to restrict task types

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Update to correct version

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Close files in ExtractTar

* implement sector dir aggregator

* update ffi

* use that nice function i wrote

* this will pretty much always be nil

* support copying directories

* use a package

* Add short tests

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* Move api struct to a seprate pkg

* fix target for ePoSt IsTicketWinner fn

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protocol.ai>

* fix sync tests

* Update FFI

* add option to symlink to presealed sectors

* fixup

* sectorbuilder: Fix proving on RO filesystem

* Update filecoin-ffi

* use actual symlink flag

* sectors: Handle sector state reload errors more gracefully

* Use filecoin-ffi master

* Update ffi to f261762

* sectorbuilder: check free space before creating sectors

* sectorbuilder: fs: address review

* fix(sectorbuilder): always cast fsstat.Bsize

fixes compilation issue under macos

* sectorbuilder: fix getpath

* sectorbuilder: Improve not enough space error

* circle: buildall on macos

* Wire up faults in fPoSt

* tear the world asunder

* temporarily move build into lib to prepare for extraction

* consume sectorbuilder from lotus

* port sectorbuilder from lotus

* downgrade to go-datastore 0.1.1 to match lotus
Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com>
Co-authored-by: Whyrusleeping <why@ipfs.io>
Co-authored-by: Jakub Sztandera <kubuxu@protonmail.ch>
Co-authored-by: Frank <wholery@163.com>
Co-authored-by: Jack Yao <yaoh.cn@gmail.com>
Co-authored-by: Henri <3359083+sternhenri@users.noreply.github.com>
Co-authored-by: Caesar Wang <dtynn@163.com>
Co-authored-by: Friedel Ziegelmayer <me@dignifiedquire.com>

.circleci/config.yml

-version: 2.1
-
-jobs:
-  build_and_test_linux:
-    docker:
-      - image: circleci/golang:1.12.1-stretch
-    working_directory: ~/go/src/github.com/filecoin-project/go-sectorbuilder
-    resource_class: 2xlarge
-    steps:
-      - configure_environment_variables
-      - run:
-          name: Install Rust toolchain
-          command: |
-            (sudo apt-get update && sudo apt-get install -y ocl-icd-opencl-dev clang libssl-dev && which cargo && which rustc) || (curl https://sh.rustup.rs -sSf | sh -s -- -y)
-            rustc --version
-      - run:
-          name: Install jq
-          command: |
-            sudo apt-get update
-            sudo apt-get install -y jq
-            jq --version
-      - checkout
-      - update_submodules
-      - build_project
-      - lint_project
-      - restore_parameter_cache
-      - obtain_filecoin_parameters
-      - save_parameter_cache
-      - build_and_run_tests
-  build_and_test_darwin:
-    macos:
-      xcode: "10.0.0"
-    working_directory: ~/go/src/github.com/filecoin-project/go-sectorbuilder
-    resource_class: large
-    steps:
-      - configure_environment_variables
-      - run:
-          name: Install go
-          command: |
-            curl https://dl.google.com/go/go1.12.1.darwin-amd64.pkg -o /tmp/go.pkg && \
-            sudo installer -pkg /tmp/go.pkg -target /
-            go version
-      - run:
-          name: Install pkg-config and md5sum
-          command: HOMEBREW_NO_AUTO_UPDATE=1 brew install pkg-config md5sha1sum
-      - run:
-          name: Install Rust toolchain
-          command: |
-            curl https://sh.rustup.rs -sSf | sh -s -- -y
-            rustc --version
-      - run:
-          name: Install jq
-          command: |
-            HOMEBREW_NO_AUTO_UPDATE=1 brew install jq
-            jq --version
-      - checkout
-      - update_submodules
-      - build_project
-      - lint_project
-      - restore_parameter_cache
-      - obtain_filecoin_parameters
-      - save_parameter_cache
-      - build_and_compile_tests
-
-workflows:
-  version: 2
-  test_all:
-    jobs:
-      - build_and_test_linux
-      - build_and_test_darwin
-
-commands:
-  configure_environment_variables:
-    steps:
-      - run:
-          name: Configure environment variables
-          command: |
-            echo 'export PATH="/usr/local/go/bin:${HOME}/.cargo/bin:${PATH}:${HOME}/go/bin:${HOME}/.bin"' >> $BASH_ENV
-            echo 'export GOPATH="${HOME}/go"' >> $BASH_ENV
-            echo 'export FIL_PROOFS_PARAMETER_CACHE="${HOME}/filecoin-proof-parameters/"' >> $BASH_ENV
-            echo 'export GO111MODULE=on' >> $BASH_ENV
-            echo 'export RUST_LOG=info' >> $BASH_ENV
-  obtain_filecoin_parameters:
-    steps:
-      - run:
-          name: Obtain filecoin groth parameters
-          command: ./paramcache --params-for-sector-sizes=1024
-          no_output_timeout: 30m
-  update_submodules:
-    steps:
-      - run:
-          name: Update submodules
-          command: git submodule update --init --recursive
-  build_project:
-    steps:
-      - run:
-          name: Build project
-          command: make
-      - run:
-          name: Ensure paramcache is installed to project root
-          command: |
-            test -f ./paramcache \
-              || (rustup run --install nightly cargo install filecoin-proofs --force --git=https://github.com/filecoin-project/rust-fil-proofs.git --branch=master --bin=paramcache --root=./ \
-                && mv ./bin/paramcache ./paramcache)
-  lint_project:
-    steps:
-      - run:
-          name: Lint project
-          command: go run github.com/golangci/golangci-lint/cmd/golangci-lint run
-  build_and_run_tests:
-    steps:
-      - run:
-          name: Test project
-          command: RUST_LOG=info go test -p 1 -timeout 60m
-          no_output_timeout: 60m
-  build_and_compile_tests:
-    steps:
-      - run:
-          name: Build project and tests, but don't actually run the tests (used to verify that build/link works with Darwin)
-          command: RUST_LOG=info go test -run=^$
-  restore_parameter_cache:
-    steps:
-      - restore_cache:
-          keys:
-            - v17-proof-params-{{ arch }}
-  save_parameter_cache:
-    steps:
-      - save_cache:
-          key: v17-proof-params-{{ arch }}
-          paths:
-            - "~/filecoin-proof-parameters/"

.gitignore

 **/*.h
 **/*.a
 **/*.pc
-.install-rust-fil-sector-builder
-bin/
-.crates.toml
+build/.filecoin-ffi-install
+build/.update-submodules
\ No newline at end of file
-**/paramcache

.gitmodules

-[submodule "rust-fil-sector-builder"]
-	path = rust-fil-sector-builder
-	url = https://github.com/filecoin-project/rust-fil-sector-builder
+[submodule "extern/filecoin-ffi"]
+	path = extern/filecoin-ffi
+	url = git@github.com:filecoin-project/filecoin-ffi
+	branch = master

LICENSE-APACHE

-Copyright (c) 2019 Filecoin Project
-
-Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

LICENSE-MIT

-The MIT License (MIT)
-
-Copyright (c) 2019 Filecoin Project
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.

Makefile

-DEPS:=sector_builder_ffi.h sector_builder_ffi.pc libsector_builder_ffi.a
+SHELL=/usr/bin/env bash
 
-all: $(DEPS)
+all: build
 .PHONY: all
 
+# git submodules that need to be loaded
+SUBMODULES:=
 
-$(DEPS): .install-rust-fil-sector-builder  ;
+# things to clean up, e.g. libfilecoin.a
+CLEAN:=
 
-.install-rust-fil-sector-builder: rust-fil-sector-builder
-	./install-rust-fil-sector-builder
+FFI_PATH:=extern/filecoin-ffi/
+FFI_DEPS:=libfilecoin.a filecoin.pc filecoin.h
+FFI_DEPS:=$(addprefix $(FFI_PATH),$(FFI_DEPS))
+
+$(FFI_DEPS): build/.filecoin-ffi-install ;
+
+# dummy file that marks the last time the filecoin-ffi project was built
+build/.filecoin-ffi-install: $(FFI_PATH)
+	$(MAKE) -C $(FFI_PATH) $(FFI_DEPS:$(FFI_PATH)%=%)
 	@touch $@
 
+SUBMODULES+=$(FFI_PATH)
+BUILD_DEPS+=build/.filecoin-ffi-install
+CLEAN+=build/.filecoin-ffi-install
+
+$(SUBMODULES): build/.update-submodules ;
+
+# dummy file that marks the last time submodules were updated
+build/.update-submodules:
+	git submodule update --init --recursive
+	touch $@
+
+CLEAN+=build/.update-submodules
+
+# build and install any upstream dependencies, e.g. filecoin-ffi
+deps: $(BUILD_DEPS)
+.PHONY: deps
+
+test: $(BUILD_DEPS)
+	go test -v $(GOFLAGS) ./...
+.PHONY: test
+
+lint: $(BUILD_DEPS)
+	golangci-lint run -v  --concurrency 2 --new-from-rev origin/master
+.PHONY: lint
+
+build: $(BUILD_DEPS)
+	go build -v $(GOFLAGS) ./...
+.PHONY: build
 
 clean:
-	rm -rf $(DEPS) .install-rust-fil-sector-builder
+	rm -rf $(CLEAN)
+	-$(MAKE) -C $(FFI_PATH) clean
 .PHONY: clean

README.md

-# go-sectorbuilder
-
-> Go bindings for the Filecoin Sector Builder
-
-## Building
-
-> make

bindings.go

-package go_sectorbuilder
-
-import (
-	"bytes"
-	"encoding/json"
-	"os"
-	"runtime"
-	"sort"
-	"time"
-	"unsafe"
-
-	"github.com/filecoin-project/go-sectorbuilder/sealed_sector_health"
-	"github.com/filecoin-project/go-sectorbuilder/sealing_state"
-
-	logging "github.com/ipfs/go-log"
-	"github.com/pkg/errors"
-)
-
-// #cgo LDFLAGS: ${SRCDIR}/libsector_builder_ffi.a
-// #cgo pkg-config: ${SRCDIR}/sector_builder_ffi.pc
-// #include "./sector_builder_ffi.h"
-import "C"
-
-var log = logging.Logger("libsectorbuilder") // nolint: deadcode
-
-func elapsed(what string) func() {
-	start := time.Now()
-	return func() {
-		log.Debugf("%s took %v\n", what, time.Since(start))
-	}
-}
-
-// SortedPublicSectorInfo is a slice of PublicSectorInfo sorted
-// (lexicographically, ascending) by replica commitment (CommR).
-type SortedPublicSectorInfo struct {
-	f []PublicSectorInfo
-}
-
-// SortedPrivateSectorInfo is a slice of PrivateSectorInfo sorted
-// (lexicographically, ascending) by replica commitment (CommR).
-type SortedPrivateSectorInfo struct {
-	f []PrivateSectorInfo
-}
-
-// SealTicket is required for the first step of Interactive PoRep.
-type SealTicket struct {
-	BlockHeight uint64
-	TicketBytes [32]byte
-}
-
-// SealSeed is required for the second step of Interactive PoRep.
-type SealSeed struct {
-	BlockHeight uint64
-	TicketBytes [32]byte
-}
-
-type Candidate struct {
-	SectorID             uint64
-	PartialTicket        [32]byte
-	Ticket               [32]byte
-	SectorChallengeIndex uint64
-}
-
-// NewSortedPublicSectorInfo returns a SortedPublicSectorInfo
-func NewSortedPublicSectorInfo(sectorInfo ...PublicSectorInfo) SortedPublicSectorInfo {
-	fn := func(i, j int) bool {
-		return bytes.Compare(sectorInfo[i].CommR[:], sectorInfo[j].CommR[:]) == -1
-	}
-
-	sort.Slice(sectorInfo[:], fn)
-
-	return SortedPublicSectorInfo{
-		f: sectorInfo,
-	}
-}
-
-// Values returns the sorted PublicSectorInfo as a slice
-func (s *SortedPublicSectorInfo) Values() []PublicSectorInfo {
-	return s.f
-}
-
-// MarshalJSON JSON-encodes and serializes the SortedPublicSectorInfo.
-func (s SortedPublicSectorInfo) MarshalJSON() ([]byte, error) {
-	return json.Marshal(s.f)
-}
-
-// UnmarshalJSON parses the JSON-encoded byte slice and stores the result in the
-// value pointed to by s.f. Note that this method allows for construction of a
-// SortedPublicSectorInfo which violates its invariant (that its PublicSectorInfo are sorted
-// in some defined way). Callers should take care to never provide a byte slice
-// which would violate this invariant.
-func (s *SortedPublicSectorInfo) UnmarshalJSON(b []byte) error {
-	return json.Unmarshal(b, &s.f)
-}
-
-type PublicSectorInfo struct {
-	SectorID uint64
-	CommR    [CommitmentBytesLen]byte
-}
-
-// NewSortedPrivateSectorInfo returns a SortedPrivateSectorInfo
-func NewSortedPrivateSectorInfo(sectorInfo ...PrivateSectorInfo) SortedPrivateSectorInfo {
-	fn := func(i, j int) bool {
-		return bytes.Compare(sectorInfo[i].CommR[:], sectorInfo[j].CommR[:]) == -1
-	}
-
-	sort.Slice(sectorInfo[:], fn)
-
-	return SortedPrivateSectorInfo{
-		f: sectorInfo,
-	}
-}
-
-// Values returns the sorted PrivateSectorInfo as a slice
-func (s *SortedPrivateSectorInfo) Values() []PrivateSectorInfo {
-	return s.f
-}
-
-// MarshalJSON JSON-encodes and serializes the SortedPrivateSectorInfo.
-func (s SortedPrivateSectorInfo) MarshalJSON() ([]byte, error) {
-	return json.Marshal(s.f)
-}
-
-func (s *SortedPrivateSectorInfo) UnmarshalJSON(b []byte) error {
-	return json.Unmarshal(b, &s.f)
-}
-
-type PrivateSectorInfo struct {
-	SectorID         uint64
-	CommR            [CommitmentBytesLen]byte
-	CacheDirPath     string
-	SealedSectorPath string
-}
-
-// CommitmentBytesLen is the number of bytes in a CommR, CommD, CommP, and CommRStar.
-const CommitmentBytesLen = 32
-
-// StagedSectorMetadata is a sector into which we write user piece-data before
-// sealing. Note: SectorID is unique across all staged and sealed sectors for a
-// storage miner actor.
-type StagedSectorMetadata struct {
-	SectorID uint64
-}
-
-// SealedSectorMetadata represents a sector in the builder that has been sealed.
-type SealedSectorMetadata struct {
-	SectorID uint64
-	CommD    [CommitmentBytesLen]byte
-	CommR    [CommitmentBytesLen]byte
-	Proof    []byte
-	Pieces   []PieceMetadata
-	Health   sealed_sector_health.Health
-	Ticket   SealTicket
-	Seed     SealSeed
-}
-
-// SealPreCommitOutput is used to acquire a seed from the chain for the second
-// step of Interactive PoRep.
-type SealPreCommitOutput struct {
-	SectorID uint64
-	CommD    [CommitmentBytesLen]byte
-	CommR    [CommitmentBytesLen]byte
-	Pieces   []PieceMetadata
-	Ticket   SealTicket
-}
-
-// RawSealPreCommitOutput is used to acquire a seed from the chain for the
-// second step of Interactive PoRep. The PersistentAux is not expected to appear
-// on-chain, but is needed for committing. This struct is useful for standalone
-// (e.g. no sector builder) sealing.
-type RawSealPreCommitOutput struct {
-	CommC     [CommitmentBytesLen]byte
-	CommD     [CommitmentBytesLen]byte
-	CommR     [CommitmentBytesLen]byte
-	CommRLast [CommitmentBytesLen]byte
-}
-
-// SealCommitOutput is produced by the second step of Interactive PoRep.
-type SealCommitOutput struct {
-	SectorID uint64
-	CommD    [CommitmentBytesLen]byte
-	CommR    [CommitmentBytesLen]byte
-	Proof    []byte
-	Pieces   []PieceMetadata
-	Ticket   SealTicket
-	Seed     SealSeed
-}
-
-// SectorSealingStatus communicates how far along in the sealing process a
-// sector has progressed.
-type SectorSealingStatus struct {
-	SectorID     uint64
-	State        sealing_state.State
-	SealErrorMsg string                   // will be nil unless State == Failed
-	CommD        [CommitmentBytesLen]byte // will be empty unless State == Committed
-	CommR        [CommitmentBytesLen]byte // will be empty unless State == Committed
-	Proof        []byte                   // will be empty unless State == Committed
-	Pieces       []PieceMetadata          // will be empty unless State == Committed
-	Ticket       SealTicket               // will be empty unless State == Committed
-	Seed         SealSeed                 // will be empty unless State == Committed
-}
-
-// PieceMetadata represents a piece stored by the sector builder.
-type PieceMetadata struct {
-	Key   string
-	Size  uint64
-	CommP [CommitmentBytesLen]byte
-}
-
-// PublicPieceInfo is an on-chain tuple of CommP and aligned piece-size.
-type PublicPieceInfo struct {
-	Size  uint64
-	CommP [CommitmentBytesLen]byte
-}
-
-// VerifySeal returns true if the sealing operation from which its inputs were
-// derived was valid, and false if not.
-func VerifySeal(
-	sectorSize uint64,
-	commR [CommitmentBytesLen]byte,
-	commD [CommitmentBytesLen]byte,
-	proverID [32]byte,
-	ticket [32]byte,
-	seed [32]byte,
-	sectorID uint64,
-	proof []byte,
-) (bool, error) {
-	defer elapsed("VerifySeal")()
-
-	commDCBytes := C.CBytes(commD[:])
-	defer C.free(commDCBytes)
-
-	commRCBytes := C.CBytes(commR[:])
-	defer C.free(commRCBytes)
-
-	proofCBytes := C.CBytes(proof[:])
-	defer C.free(proofCBytes)
-
-	proverIDCBytes := C.CBytes(proverID[:])
-	defer C.free(proverIDCBytes)
-
-	ticketCBytes := C.CBytes(ticket[:])
-	defer C.free(ticketCBytes)
-
-	seedCBytes := C.CBytes(seed[:])
-	defer C.free(seedCBytes)
-
-	// a mutable pointer to a VerifySealResponse C-struct
-	resPtr := C.sector_builder_ffi_reexported_verify_seal(
-		C.uint64_t(sectorSize),
-		(*[CommitmentBytesLen]C.uint8_t)(commRCBytes),
-		(*[CommitmentBytesLen]C.uint8_t)(commDCBytes),
-		(*[32]C.uint8_t)(proverIDCBytes),
-		C.uint64_t(sectorID),
-		(*[32]C.uint8_t)(ticketCBytes),
-		(*[32]C.uint8_t)(seedCBytes),
-		(*C.uint8_t)(proofCBytes),
-		C.size_t(len(proof)),
-	)
-	defer C.sector_builder_ffi_reexported_destroy_verify_seal_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return false, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return bool(resPtr.is_valid), nil
-}
-
-// VerifyPoSt returns true if the PoSt-generation operation from which its
-// inputs were derived was valid, and false if not.
-func VerifyPoSt(
-	sectorSize uint64,
-	sectorInfo SortedPublicSectorInfo,
-	randomness [32]byte,
-	challengeCount uint64,
-	proof []byte,
-	winners []Candidate,
-	proverID [32]byte,
-) (bool, error) {
-	defer elapsed("VerifyPoSt")()
-
-	// CommRs and sector ids must be provided to C.verify_post in the same order
-	// that they were provided to the C.generate_post
-	sortedCommRs := make([][CommitmentBytesLen]byte, len(sectorInfo.Values()))
-	sortedSectorIds := make([]uint64, len(sectorInfo.Values()))
-	for idx, v := range sectorInfo.Values() {
-		sortedCommRs[idx] = v.CommR
-		sortedSectorIds[idx] = v.SectorID
-	}
-
-	// flattening the byte slice makes it easier to copy into the C heap
-	flattened := make([]byte, CommitmentBytesLen*len(sortedCommRs))
-	for idx, commR := range sortedCommRs {
-		copy(flattened[(CommitmentBytesLen*idx):(CommitmentBytesLen*(1+idx))], commR[:])
-	}
-
-	// copy bytes from Go to C heap
-	flattenedCommRsCBytes := C.CBytes(flattened)
-	defer C.free(flattenedCommRsCBytes)
-
-	randomnessCBytes := C.CBytes(randomness[:])
-	defer C.free(randomnessCBytes)
-
-	proofCBytes := C.CBytes(proof)
-	defer C.free(proofCBytes)
-
-	// allocate fixed-length array of uint64s in C heap
-	sectorIdsPtr, sectorIdsSize := cUint64s(sortedSectorIds)
-	defer C.free(unsafe.Pointer(sectorIdsPtr))
-
-	winnersPtr, winnersSize := cCandidates(winners)
-	defer C.free(unsafe.Pointer(winnersPtr))
-
-	proverIDCBytes := C.CBytes(proverID[:])
-	defer C.free(proverIDCBytes)
-
-	// a mutable pointer to a VerifyPoStResponse C-struct
-	resPtr := C.sector_builder_ffi_reexported_verify_post(
-		C.uint64_t(sectorSize),
-		(*[32]C.uint8_t)(randomnessCBytes),
-		C.uint64_t(challengeCount),
-		sectorIdsPtr,
-		sectorIdsSize,
-		(*C.uint8_t)(flattenedCommRsCBytes),
-		C.size_t(len(flattened)),
-		(*C.uint8_t)(proofCBytes),
-		C.size_t(len(proof)),
-		winnersPtr,
-		winnersSize,
-		(*[32]C.uint8_t)(proverIDCBytes),
-	)
-	defer C.sector_builder_ffi_reexported_destroy_verify_post_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return false, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return bool(resPtr.is_valid), nil
-}
-
-// GetMaxUserBytesPerStagedSector returns the number of user bytes that will fit
-// into a staged sector. Due to bit-padding, the number of user bytes that will
-// fit into the staged sector will be less than number of bytes in sectorSize.
-func GetMaxUserBytesPerStagedSector(sectorSize uint64) uint64 {
-	defer elapsed("GetMaxUserBytesPerStagedSector")()
-
-	return uint64(C.sector_builder_ffi_reexported_get_max_user_bytes_per_staged_sector(C.uint64_t(sectorSize)))
-}
-
-// InitSectorBuilder allocates and returns a pointer to a sector builder.
-func InitSectorBuilder(
-	sectorSize uint64,
-	poRepProofPartitions uint8,
-	lastUsedSectorID uint64,
-	metadataDir string,
-	proverID [32]byte,
-	sealedSectorDir string,
-	stagedSectorDir string,
-	sectorCacheRootDir string,
-	maxNumOpenStagedSectors uint8,
-	numWorkerThreads uint8,
-) (unsafe.Pointer, error) {
-	defer elapsed("InitSectorBuilder")()
-
-	cMetadataDir := C.CString(metadataDir)
-	defer C.free(unsafe.Pointer(cMetadataDir))
-
-	proverIDCBytes := C.CBytes(proverID[:])
-	defer C.free(proverIDCBytes)
-
-	cStagedSectorDir := C.CString(stagedSectorDir)
-	defer C.free(unsafe.Pointer(cStagedSectorDir))
-
-	cSealedSectorDir := C.CString(sealedSectorDir)
-	defer C.free(unsafe.Pointer(cSealedSectorDir))
-
-	cSectorCacheRootDir := C.CString(sectorCacheRootDir)
-	defer C.free(unsafe.Pointer(cSectorCacheRootDir))
-
-	resPtr := C.sector_builder_ffi_init_sector_builder(
-		cSectorClass(sectorSize, poRepProofPartitions),
-		C.uint64_t(lastUsedSectorID),
-		cMetadataDir,
-		(*[32]C.uint8_t)(proverIDCBytes),
-		cSealedSectorDir,
-		cStagedSectorDir,
-		cSectorCacheRootDir,
-		C.uint8_t(maxNumOpenStagedSectors),
-		C.uint8_t(numWorkerThreads),
-	)
-	defer C.sector_builder_ffi_destroy_init_sector_builder_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return nil, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return unsafe.Pointer(resPtr.sector_builder), nil
-}
-
-// DestroySectorBuilder deallocates the sector builder associated with the
-// provided pointer. This function will panic if the provided pointer is null
-// or if the sector builder has been previously deallocated.
-func DestroySectorBuilder(sectorBuilderPtr unsafe.Pointer) {
-	defer elapsed("DestroySectorBuilder")()
-
-	C.sector_builder_ffi_destroy_sector_builder((*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr))
-}
-
-// AddPiece writes the given piece into an unsealed sector and returns the id of that sector.
-func AddPiece(
-	sectorBuilderPtr unsafe.Pointer,
-	pieceKey string,
-	pieceBytes uint64,
-	piecePath string,
-) (uint64, error) {
-	defer elapsed("AddPiece")()
-
-	pieceFile, err := os.Open(piecePath)
-	if err != nil {
-		return 0, err
-	}
-
-	return AddPieceFromFile(sectorBuilderPtr, pieceKey, pieceBytes, pieceFile)
-}
-
-// AddPieceFromFile writes the given piece into an unsealed sector and returns the id of that sector.
-func AddPieceFromFile(
-	sectorBuilderPtr unsafe.Pointer,
-	pieceKey string,
-	pieceBytes uint64,
-	pieceFile *os.File,
-) (sectorID uint64, retErr error) {
-	defer elapsed("AddPieceFromFile")()
-
-	cPieceKey := C.CString(pieceKey)
-	defer C.free(unsafe.Pointer(cPieceKey))
-
-	pieceFd := pieceFile.Fd()
-
-	// TODO: The UTC time, in seconds, at which the sector builder can safely
-	// delete the piece. This allows for co-location of pieces with similar time
-	// constraints, and allows the sector builder to remove sectors containing
-	// pieces whose deals have expired.
-	//
-	// This value is currently ignored by the sector builder.
-	//
-	// https://github.com/filecoin-project/rust-fil-sector-builder/issues/32
-	pieceExpiryUtcSeconds := 0
-
-	resPtr := C.sector_builder_ffi_add_piece(
-		(*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr),
-		cPieceKey,
-		C.int(pieceFd),
-		C.uint64_t(pieceBytes),
-		C.uint64_t(pieceExpiryUtcSeconds),
-	)
-	defer C.sector_builder_ffi_destroy_add_piece_response(resPtr)
-
-	// Make sure our filedescriptor stays alive, stayin alive
-	runtime.KeepAlive(pieceFile)
-
-	if resPtr.status_code != 0 {
-		return 0, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return uint64(resPtr.sector_id), nil
-}
-
-// ReadPieceFromSealedSector produces a byte buffer containing the piece
-// associated with the provided key. If the key is not associated with any piece
-// yet sealed into a sector, an error will be returned.
-func ReadPieceFromSealedSector(sectorBuilderPtr unsafe.Pointer, pieceKey string) ([]byte, error) {
-	defer elapsed("ReadPieceFromSealedSector")()
-
-	cPieceKey := C.CString(pieceKey)
-	defer C.free(unsafe.Pointer(cPieceKey))
-
-	resPtr := C.sector_builder_ffi_read_piece_from_sealed_sector(
-		(*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr),
-		cPieceKey,
-	)
-	defer C.sector_builder_ffi_destroy_read_piece_from_sealed_sector_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return nil, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return goBytes(resPtr.data_ptr, resPtr.data_len), nil
-}
-
-// SealPreCommit pre-commits the sector with the provided id to the ticket,
-// blocking until completion. If no staged sector with the provided id exists in
-// the FullyPacked or AcceptingPieces state, an error will be returned.
-func SealPreCommit(sectorBuilderPtr unsafe.Pointer, sectorID uint64, ticket SealTicket) (SealPreCommitOutput, error) {
-	defer elapsed("SealPreCommit")()
-
-	cTicketBytes := C.CBytes(ticket.TicketBytes[:])
-	defer C.free(cTicketBytes)
-
-	cSealTicket := C.sector_builder_ffi_FFISealTicket{
-		block_height: C.uint64_t(ticket.BlockHeight),
-		ticket_bytes: *(*[32]C.uint8_t)(cTicketBytes),
-	}
-
-	resPtr := C.sector_builder_ffi_seal_pre_commit((*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr), C.uint64_t(sectorID), cSealTicket)
-	defer C.sector_builder_ffi_destroy_seal_pre_commit_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return SealPreCommitOutput{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	out, err := goSectorBuilderSealPreCommitOutput(resPtr)
-	if err != nil {
-		return SealPreCommitOutput{}, err
-	}
-
-	return out, nil
-}
-
-// ResumeSealPreCommit resumes the pre-commit operation for a sector with the
-// provided id. If no sector exists with the given id that is in the
-// PreCommittingPaused state, an error will be returned.
-func ResumeSealPreCommit(sectorBuilderPtr unsafe.Pointer, sectorID uint64) (SealPreCommitOutput, error) {
-	defer elapsed("ResumeSealPreCommit")()
-
-	resPtr := C.sector_builder_ffi_resume_seal_pre_commit((*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr), C.uint64_t(sectorID))
-	defer C.sector_builder_ffi_destroy_resume_seal_pre_commit_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return SealPreCommitOutput{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	out, err := goResumeSealPreCommitOutput(resPtr)
-	if err != nil {
-		return SealPreCommitOutput{}, err
-	}
-
-	return out, nil
-}
-
-// SealCommit commits the sector with the provided id to the seed, blocking
-// until completion. If no staged sector exists in the PreCommitted state with
-// such an id, an error will be returned.
-func SealCommit(sectorBuilderPtr unsafe.Pointer, sectorID uint64, seed SealSeed) (SealCommitOutput, error) {
-	defer elapsed("SealCommit")()
-
-	cSeedBytes := C.CBytes(seed.TicketBytes[:])
-	defer C.free(cSeedBytes)
-
-	cSealSeed := C.sector_builder_ffi_FFISealSeed{
-		block_height: C.uint64_t(seed.BlockHeight),
-		ticket_bytes: *(*[32]C.uint8_t)(cSeedBytes),
-	}
-
-	resPtr := C.sector_builder_ffi_seal_commit((*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr), C.uint64_t(sectorID), cSealSeed)
-	defer C.sector_builder_ffi_destroy_seal_commit_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return SealCommitOutput{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	out, err := goSectorBuilderSealCommitOutput(resPtr)
-	if err != nil {
-		return SealCommitOutput{}, err
-	}
-
-	return out, nil
-}
-
-// ResumeSealCommit resumes sector commit (the second stage of Interactive
-// PoRep) for a sector in the CommittingPaused state. If no staged sector exists
-// in such a state, an error will be returned.
-func ResumeSealCommit(sectorBuilderPtr unsafe.Pointer, sectorID uint64) (SealCommitOutput, error) {
-	defer elapsed("ResumeSealCommit")()
-
-	resPtr := C.sector_builder_ffi_resume_seal_commit((*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr), C.uint64_t(sectorID))
-	defer C.sector_builder_ffi_destroy_resume_seal_commit_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return SealCommitOutput{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	out, err := goResumeSealCommitOutput(resPtr)
-	if err != nil {
-		return SealCommitOutput{}, err
-	}
-
-	return out, nil
-}
-
-// GetAllStagedSectors returns a slice of all staged sector metadata for the sector builder.
-func GetAllStagedSectors(sectorBuilderPtr unsafe.Pointer) ([]StagedSectorMetadata, error) {
-	defer elapsed("GetAllStagedSectors")()
-
-	resPtr := C.sector_builder_ffi_get_staged_sectors((*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr))
-	defer C.sector_builder_ffi_destroy_get_staged_sectors_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return nil, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	meta, err := goStagedSectorMetadata(resPtr.sectors_ptr, resPtr.sectors_len)
-	if err != nil {
-		return nil, err
-	}
-
-	return meta, nil
-}
-
-// GetAllSealedSectors returns a slice of all sealed sector metadata, excluding
-// sector health.
-func GetAllSealedSectors(sectorBuilderPtr unsafe.Pointer) ([]SealedSectorMetadata, error) {
-	defer elapsed("GetAllSealedSectors")()
-
-	return getAllSealedSectors(sectorBuilderPtr, false)
-}
-
-// GetAllSealedSectorsWithHealth returns a slice of all sealed sector metadata
-// for the sector builder, including sector health info (which can be expensive
-// to compute).
-func GetAllSealedSectorsWithHealth(sectorBuilderPtr unsafe.Pointer) ([]SealedSectorMetadata, error) {
-	defer elapsed("GetAllSealedSectorsWithHealth")()
-
-	return getAllSealedSectors(sectorBuilderPtr, true)
-}
-
-// GetSectorSealingStatusByID produces sector sealing status (staged, sealing in
-// progress, sealed, failed) for the provided sector id. If no sector
-// corresponding to the provided id exists, this function returns an error.
-func GetSectorSealingStatusByID(sectorBuilderPtr unsafe.Pointer, sectorID uint64) (SectorSealingStatus, error) {
-	defer elapsed("GetSectorSealingStatusByID")()
-
-	resPtr := C.sector_builder_ffi_get_seal_status(
-		(*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr),
-		C.uint64_t(sectorID),
-	)
-	defer C.sector_builder_ffi_destroy_get_seal_status_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return SectorSealingStatus{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	if resPtr.seal_status_code == C.Failed {
-		return SectorSealingStatus{SectorID: sectorID, State: sealing_state.Failed, SealErrorMsg: C.GoString(resPtr.seal_error_msg)}, nil
-	} else if resPtr.seal_status_code == C.AcceptingPieces {
-		return SectorSealingStatus{SectorID: sectorID, State: sealing_state.AcceptingPieces}, nil
-	} else if resPtr.seal_status_code == C.Committing {
-		return SectorSealingStatus{SectorID: sectorID, State: sealing_state.Committing}, nil
-	} else if resPtr.seal_status_code == C.CommittingPaused {
-		return SectorSealingStatus{SectorID: sectorID, State: sealing_state.CommittingPaused}, nil
-	} else if resPtr.seal_status_code == C.FullyPacked {
-		return SectorSealingStatus{SectorID: sectorID, State: sealing_state.FullyPacked}, nil
-	} else if resPtr.seal_status_code == C.PreCommitted {
-		return SectorSealingStatus{SectorID: sectorID, State: sealing_state.PreCommitted}, nil
-	} else if resPtr.seal_status_code == C.PreCommitting {
-		return SectorSealingStatus{SectorID: sectorID, State: sealing_state.PreCommitting}, nil
-	} else if resPtr.seal_status_code == C.PreCommittingPaused {
-		return SectorSealingStatus{SectorID: sectorID, State: sealing_state.PreCommittingPaused}, nil
-	} else if resPtr.seal_status_code == C.Committed {
-		commRSlice := goBytes(&resPtr.comm_r[0], CommitmentBytesLen)
-		var commR [CommitmentBytesLen]byte
-		copy(commR[:], commRSlice)
-
-		commDSlice := goBytes(&resPtr.comm_d[0], CommitmentBytesLen)
-		var commD [CommitmentBytesLen]byte
-		copy(commD[:], commDSlice)
-
-		proof := goBytes(resPtr.proof_ptr, resPtr.proof_len)
-
-		ps, err := goPieceMetadata(resPtr.pieces_ptr, resPtr.pieces_len)
-		if err != nil {
-			return SectorSealingStatus{}, errors.Wrap(err, "failed to marshal from string to cid")
-		}
-
-		return SectorSealingStatus{
-			SectorID: sectorID,
-			State:    sealing_state.Committed,
-			CommD:    commD,
-			CommR:    commR,
-			Proof:    proof,
-			Pieces:   ps,
-			Ticket:   goSealTicket(resPtr.seal_ticket),
-			Seed:     goSealSeed(resPtr.seal_seed),
-		}, nil
-	} else {
-		// unknown
-		return SectorSealingStatus{}, errors.New("unexpected seal status")
-	}
-}
-
-// FinalizeTicket creates an actual ticket from a partial ticket.
-func FinalizeTicket(partialTicket [32]byte) ([32]byte, error) {
-	defer elapsed("FinalizeTicket")()
-
-	partialTicketPtr := unsafe.Pointer(&(partialTicket)[0])
-	resPtr := C.sector_builder_ffi_reexported_finalize_ticket(
-		(*[32]C.uint8_t)(partialTicketPtr),
-	)
-	defer C.sector_builder_ffi_reexported_destroy_finalize_ticket_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return [32]byte{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return goCommitment(&resPtr.ticket[0]), nil
-}
-
-// GenerateCandidates creates a list of election candidates.
-func GenerateCandidates(
-	sectorBuilderPtr unsafe.Pointer,
-	sectorInfo SortedPublicSectorInfo,
-	randomness [32]byte,
-	challengeCount uint64,
-	faults []uint64,
-) ([]Candidate, error) {
-	defer elapsed("GenerateCandidates")()
-
-	// CommRs and sector ids must be provided to C.verify_post in the same order
-	// that they were provided to the C.generate_post
-	sortedCommRs := make([][CommitmentBytesLen]byte, len(sectorInfo.Values()))
-	for idx, v := range sectorInfo.Values() {
-		sortedCommRs[idx] = v.CommR
-	}
-
-	// flattening the byte slice makes it easier to copy into the C heap
-	flattened := make([]byte, CommitmentBytesLen*len(sortedCommRs))
-	for idx, commR := range sortedCommRs {
-		copy(flattened[(CommitmentBytesLen*idx):(CommitmentBytesLen*(1+idx))], commR[:])
-	}
-
-	// copy the Go byte slice into C memory
-	cflattened := C.CBytes(flattened)
-	defer C.free(cflattened)
-
-	randomnessPtr := unsafe.Pointer(&(randomness)[0])
-
-	faultsPtr, faultsSize := cUint64s(faults)
-	defer C.free(unsafe.Pointer(faultsPtr))
-
-	// a mutable pointer to a SectorBuilderGenerateCandidatesResponse C-struct
-	resPtr := C.sector_builder_ffi_generate_candidates(
-		(*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr),
-		(*C.uint8_t)(cflattened),
-		C.size_t(len(flattened)),
-		(*[32]C.uint8_t)(randomnessPtr),
-		C.uint64_t(challengeCount),
-		faultsPtr,
-		faultsSize,
-	)
-	defer C.sector_builder_ffi_destroy_generate_candidates_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return nil, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return goCandidates(resPtr.candidates_ptr, resPtr.candidates_len)
-}
-
-// GeneratePoSt produces a proof-of-spacetime for the provided replica commitments.
-func GeneratePoSt(
-	sectorBuilderPtr unsafe.Pointer,
-	sectorInfo SortedPublicSectorInfo,
-	randomness [32]byte,
-	challengeCount uint64,
-	winners []Candidate,
-) ([]byte, error) {
-	defer elapsed("GeneratePoSt")()
-
-	// CommRs and sector ids must be provided to C.verify_post in the same order
-	// that they were provided to the C.generate_post
-	sortedCommRs := make([][CommitmentBytesLen]byte, len(sectorInfo.Values()))
-	for idx, v := range sectorInfo.Values() {
-		sortedCommRs[idx] = v.CommR
-	}
-
-	// flattening the byte slice makes it easier to copy into the C heap
-	flattened := make([]byte, CommitmentBytesLen*len(sortedCommRs))
-	for idx, commR := range sortedCommRs {
-		copy(flattened[(CommitmentBytesLen*idx):(CommitmentBytesLen*(1+idx))], commR[:])
-	}
-
-	// copy the Go byte slice into C memory
-	cflattened := C.CBytes(flattened)
-	defer C.free(cflattened)
-
-	randomnessPtr := unsafe.Pointer(&(randomness)[0])
-
-	winnersPtr, winnersSize := cCandidates(winners)
-	defer C.free(unsafe.Pointer(winnersPtr))
-
-	// a mutable pointer to a SectorBuilderGeneratePoStResponse C-struct
-	resPtr := C.sector_builder_ffi_generate_post(
-		(*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr),
-		(*C.uint8_t)(cflattened),
-		C.size_t(len(flattened)),
-		(*[32]C.uint8_t)(randomnessPtr),
-		C.uint64_t(challengeCount),
-		winnersPtr,
-		winnersSize,
-	)
-	defer C.sector_builder_ffi_destroy_generate_post_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return nil, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return goBytes(resPtr.flattened_proofs_ptr, resPtr.flattened_proofs_len), nil
-}
-
-// AcquireSectorId returns a sector ID which can be used by out-of-band sealing.
-func AcquireSectorId(
-	sectorBuilderPtr unsafe.Pointer,
-) (uint64, error) {
-	defer elapsed("AcquireSectorId")()
-
-	resPtr := C.sector_builder_ffi_acquire_sector_id(
-		(*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr),
-	)
-	defer C.sector_builder_ffi_destroy_acquire_sector_id_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return 0, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return uint64(resPtr.sector_id), nil
-}
-
-// ImportSealedSector
-func ImportSealedSector(
-	sectorBuilderPtr unsafe.Pointer,
-	sectorID uint64,
-	sectorCacheDirPath string,
-	sealedSectorPath string,
-	ticket SealTicket,
-	seed SealSeed,
-	commR [CommitmentBytesLen]byte,
-	commD [CommitmentBytesLen]byte,
-	commC [CommitmentBytesLen]byte,
-	commRLast [CommitmentBytesLen]byte,
-	proof []byte,
-	pieces []PieceMetadata,
-) error {
-	defer elapsed("ImportSealedSector")()
-
-	cSectorCacheDirPath := C.CString(sectorCacheDirPath)
-	defer C.free(unsafe.Pointer(cSectorCacheDirPath))
-
-	cSealedSectorPath := C.CString(sealedSectorPath)
-	defer C.free(unsafe.Pointer(cSealedSectorPath))
-
-	cTicketBytes := C.CBytes(ticket.TicketBytes[:])
-	defer C.free(cTicketBytes)
-
-	cSealTicket := C.sector_builder_ffi_FFISealTicket{
-		block_height: C.uint64_t(ticket.BlockHeight),
-		ticket_bytes: *(*[32]C.uint8_t)(cTicketBytes),
-	}
-
-	cSeedBytes := C.CBytes(seed.TicketBytes[:])
-	defer C.free(cSeedBytes)
-
-	cSealSeed := C.sector_builder_ffi_FFISealSeed{
-		block_height: C.uint64_t(seed.BlockHeight),
-		ticket_bytes: *(*[32]C.uint8_t)(cSeedBytes),
-	}
-
-	commDCBytes := C.CBytes(commD[:])
-	defer C.free(commDCBytes)
-
-	commRCBytes := C.CBytes(commR[:])
-	defer C.free(commRCBytes)
-
-	commCCBytes := C.CBytes(commC[:])
-	defer C.free(commCCBytes)
-
-	commRLastCBytes := C.CBytes(commRLast[:])
-	defer C.free(commRLastCBytes)
-
-	proofCBytes := C.CBytes(proof[:])
-	defer C.free(proofCBytes)
-
-	piecesPtr, piecesLen := cPieceMetadata(pieces)
-	defer C.free(unsafe.Pointer(piecesPtr))
-
-	resPtr := C.sector_builder_ffi_import_sealed_sector(
-		(*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr),
-		C.uint64_t(sectorID),
-		cSectorCacheDirPath,
-		cSealedSectorPath,
-		cSealTicket,
-		cSealSeed,
-		(*[CommitmentBytesLen]C.uint8_t)(commRCBytes),
-		(*[CommitmentBytesLen]C.uint8_t)(commDCBytes),
-		(*[CommitmentBytesLen]C.uint8_t)(commCCBytes),
-		(*[CommitmentBytesLen]C.uint8_t)(commRLastCBytes),
-		(*C.uint8_t)(proofCBytes),
-		C.size_t(len(proof)),
-		piecesPtr,
-		piecesLen,
-	)
-	defer C.sector_builder_ffi_destroy_import_sealed_sector_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return nil
-}
-
-// GeneratePieceCommitment produces a piece commitment for the provided data
-// stored at a given path.
-func GeneratePieceCommitment(piecePath string, pieceSize uint64) ([CommitmentBytesLen]byte, error) {
-	pieceFile, err := os.Open(piecePath)
-	if err != nil {
-		return [CommitmentBytesLen]byte{}, err
-	}
-
-	return GeneratePieceCommitmentFromFile(pieceFile, pieceSize)
-}
-
-// GenerateDataCommitment produces a commitment for the sector containing the
-// provided pieces.
-func GenerateDataCommitment(sectorSize uint64, pieces []PublicPieceInfo) ([CommitmentBytesLen]byte, error) {
-	cPiecesPtr, cPiecesLen := cPublicPieceInfo(pieces)
-	defer C.free(unsafe.Pointer(cPiecesPtr))
-
-	resPtr := C.sector_builder_ffi_reexported_generate_data_commitment(C.uint64_t(sectorSize), (*C.sector_builder_ffi_FFIPublicPieceInfo)(cPiecesPtr), cPiecesLen)
-	defer C.sector_builder_ffi_reexported_destroy_generate_data_commitment_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return [CommitmentBytesLen]byte{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return goCommitment(&resPtr.comm_d[0]), nil
-}
-
-// GeneratePieceCommitmentFromFile produces a piece commitment for the provided data
-// stored in a given file.
-func GeneratePieceCommitmentFromFile(pieceFile *os.File, pieceSize uint64) (commP [CommitmentBytesLen]byte, err error) {
-	pieceFd := pieceFile.Fd()
-
-	resPtr := C.sector_builder_ffi_reexported_generate_piece_commitment(C.int(pieceFd), C.uint64_t(pieceSize))
-	defer C.sector_builder_ffi_reexported_destroy_generate_piece_commitment_response(resPtr)
-
-	// Make sure our filedescriptor stays alive, stayin alive
-	runtime.KeepAlive(pieceFile)
-
-	if resPtr.status_code != 0 {
-		return [CommitmentBytesLen]byte{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return goCommitment(&resPtr.comm_p[0]), nil
-}
-
-// StandaloneWriteWithAlignment
-func StandaloneWriteWithAlignment(
-	pieceFile *os.File,
-	pieceBytes uint64,
-	stagedSectorFile *os.File,
-	existingPieceSizes []uint64,
-) (leftAlignment, total uint64, commP [CommitmentBytesLen]byte, retErr error) {
-	defer elapsed("StandaloneWriteWithAlignment")()
-
-	pieceFd := pieceFile.Fd()
-	runtime.KeepAlive(pieceFile)
-
-	stagedSectorFd := stagedSectorFile.Fd()
-	runtime.KeepAlive(stagedSectorFile)
-
-	ptr, len := cUint64s(existingPieceSizes)
-	defer C.free(unsafe.Pointer(ptr))
-
-	resPtr := C.sector_builder_ffi_reexported_write_with_alignment(
-		C.int(pieceFd),
-		C.uint64_t(pieceBytes),
-		C.int(stagedSectorFd),
-		ptr,
-		len,
-	)
-	defer C.sector_builder_ffi_reexported_destroy_write_with_alignment_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return 0, 0, [CommitmentBytesLen]byte{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return uint64(resPtr.left_alignment_unpadded), uint64(resPtr.total_write_unpadded), goCommitment(&resPtr.comm_p[0]), nil
-}
-
-// StandaloneWriteWithoutAlignment
-func StandaloneWriteWithoutAlignment(
-	pieceFile *os.File,
-	pieceBytes uint64,
-	stagedSectorFile *os.File,
-) (uint64, [CommitmentBytesLen]byte, error) {
-	defer elapsed("StandaloneWriteWithoutAlignment")()
-
-	pieceFd := pieceFile.Fd()
-	runtime.KeepAlive(pieceFile)
-
-	stagedSectorFd := stagedSectorFile.Fd()
-	runtime.KeepAlive(stagedSectorFile)
-
-	resPtr := C.sector_builder_ffi_reexported_write_without_alignment(
-		C.int(pieceFd),
-		C.uint64_t(pieceBytes),
-		C.int(stagedSectorFd),
-	)
-	defer C.sector_builder_ffi_reexported_destroy_write_without_alignment_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return 0, [CommitmentBytesLen]byte{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return uint64(resPtr.total_write_unpadded), goCommitment(&resPtr.comm_p[0]), nil
-}
-
-// StandaloneSealPreCommit
-func StandaloneSealPreCommit(
-	sectorSize uint64,
-	poRepProofPartitions uint8,
-	cacheDirPath string,
-	stagedSectorPath string,
-	sealedSectorPath string,
-	sectorID uint64,
-	proverID [32]byte,
-	ticket [32]byte,
-	pieces []PublicPieceInfo,
-) (RawSealPreCommitOutput, error) {
-	defer elapsed("StandaloneSealPreCommit")()
-
-	cCacheDirPath := C.CString(cacheDirPath)
-	defer C.free(unsafe.Pointer(cCacheDirPath))
-
-	cStagedSectorPath := C.CString(stagedSectorPath)
-	defer C.free(unsafe.Pointer(cStagedSectorPath))
-
-	cSealedSectorPath := C.CString(sealedSectorPath)
-	defer C.free(unsafe.Pointer(cSealedSectorPath))
-
-	proverIDCBytes := C.CBytes(proverID[:])
-	defer C.free(proverIDCBytes)
-
-	ticketCBytes := C.CBytes(ticket[:])
-	defer C.free(ticketCBytes)
-
-	cPiecesPtr, cPiecesLen := cPublicPieceInfo(pieces)
-	defer C.free(unsafe.Pointer(cPiecesPtr))
-
-	resPtr := C.sector_builder_ffi_reexported_seal_pre_commit(
-		cSectorClass(sectorSize, poRepProofPartitions),
-		cCacheDirPath,
-		cStagedSectorPath,
-		cSealedSectorPath,
-		C.uint64_t(sectorID),
-		(*[32]C.uint8_t)(proverIDCBytes),
-		(*[32]C.uint8_t)(ticketCBytes),
-		(*C.sector_builder_ffi_FFIPublicPieceInfo)(cPiecesPtr),
-		cPiecesLen,
-	)
-	defer C.sector_builder_ffi_reexported_destroy_seal_pre_commit_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return RawSealPreCommitOutput{}, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return goRawSealPreCommitOutput(resPtr.seal_pre_commit_output), nil
-}
-
-// StandaloneSealCommit
-func StandaloneSealCommit(
-	sectorSize uint64,
-	poRepProofPartitions uint8,
-	cacheDirPath string,
-	sectorID uint64,
-	proverID [32]byte,
-	ticket [32]byte,
-	seed [32]byte,
-	pieces []PublicPieceInfo,
-	rspco RawSealPreCommitOutput,
-) ([]byte, error) {
-	defer elapsed("StandaloneSealCommit")()
-
-	cCacheDirPath := C.CString(cacheDirPath)
-	defer C.free(unsafe.Pointer(cCacheDirPath))
-
-	proverIDCBytes := C.CBytes(proverID[:])
-	defer C.free(proverIDCBytes)
-
-	ticketCBytes := C.CBytes(ticket[:])
-	defer C.free(ticketCBytes)
-
-	seedCBytes := C.CBytes(seed[:])
-	defer C.free(seedCBytes)
-
-	cPiecesPtr, cPiecesLen := cPublicPieceInfo(pieces)
-	defer C.free(unsafe.Pointer(cPiecesPtr))
-
-	resPtr := C.sector_builder_ffi_reexported_seal_commit(
-		cSectorClass(sectorSize, poRepProofPartitions),
-		cCacheDirPath,
-		C.uint64_t(sectorID),
-		(*[32]C.uint8_t)(proverIDCBytes),
-		(*[32]C.uint8_t)(ticketCBytes),
-		(*[32]C.uint8_t)(seedCBytes),
-		(*C.sector_builder_ffi_FFIPublicPieceInfo)(cPiecesPtr),
-		cPiecesLen,
-		cSealPreCommitOutput(rspco),
-	)
-	defer C.sector_builder_ffi_reexported_destroy_seal_commit_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return nil, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return C.GoBytes(unsafe.Pointer(resPtr.proof_ptr), C.int(resPtr.proof_len)), nil
-}
-
-// StandaloneUnseal
-func StandaloneUnseal(
-	sectorSize uint64,
-	poRepProofPartitions uint8,
-	cacheDirPath string,
-	sealedSectorPath string,
-	unsealOutputPath string,
-	sectorID uint64,
-	proverID [32]byte,
-	ticket [32]byte,
-	commD [CommitmentBytesLen]byte,
-) error {
-	defer elapsed("StandaloneUnseal")()
-
-	cCacheDirPath := C.CString(cacheDirPath)
-	defer C.free(unsafe.Pointer(cCacheDirPath))
-
-	cSealedSectorPath := C.CString(sealedSectorPath)
-	defer C.free(unsafe.Pointer(cSealedSectorPath))
-
-	cUnsealOutputPath := C.CString(unsealOutputPath)
-	defer C.free(unsafe.Pointer(cUnsealOutputPath))
-
-	proverIDCBytes := C.CBytes(proverID[:])
-	defer C.free(proverIDCBytes)
-
-	ticketCBytes := C.CBytes(ticket[:])
-	defer C.free(ticketCBytes)
-
-	commDCBytes := C.CBytes(commD[:])
-	defer C.free(commDCBytes)
-
-	resPtr := C.sector_builder_ffi_reexported_unseal(
-		cSectorClass(sectorSize, poRepProofPartitions),
-		cCacheDirPath,
-		cSealedSectorPath,
-		cUnsealOutputPath,
-		C.uint64_t(sectorID),
-		(*[32]C.uint8_t)(proverIDCBytes),
-		(*[32]C.uint8_t)(ticketCBytes),
-		(*[CommitmentBytesLen]C.uint8_t)(commDCBytes),
-	)
-	defer C.sector_builder_ffi_reexported_destroy_unseal_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return nil
-}
-
-// StandaloneGenerateCandidates
-func StandaloneGenerateCandidates(
-	sectorSize uint64,
-	proverID [32]byte,
-	randomness [32]byte,
-	challengeCount uint64,
-	privateSectorInfo SortedPrivateSectorInfo,
-) ([]Candidate, error) {
-	defer elapsed("StandaloneGenerateCandidates")()
-
-	randomessCBytes := C.CBytes(randomness[:])
-	defer C.free(randomessCBytes)
-
-	proverIDCBytes := C.CBytes(proverID[:])
-	defer C.free(proverIDCBytes)
-
-	replicasPtr, replicasSize := cPrivateReplicaInfos(privateSectorInfo.Values())
-	defer C.free(unsafe.Pointer(replicasPtr))
-
-	resPtr := C.sector_builder_ffi_reexported_generate_candidates(
-		C.uint64_t(sectorSize),
-		(*[32]C.uint8_t)(randomessCBytes),
-		C.uint64_t(challengeCount),
-		replicasPtr,
-		replicasSize,
-		(*[32]C.uint8_t)(proverIDCBytes),
-	)
-	defer C.sector_builder_ffi_reexported_destroy_generate_candidates_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return nil, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return goCandidates(resPtr.candidates_ptr, resPtr.candidates_len)
-}
-
-// StandaloneGeneratePoSt
-func StandaloneGeneratePoSt(
-	sectorSize uint64,
-	proverID [32]byte,
-	privateSectorInfo SortedPrivateSectorInfo,
-	randomness [32]byte,
-	winners []Candidate,
-) ([]byte, error) {
-	defer elapsed("StandaloneGeneratePoSt")()
-
-	replicasPtr, replicasSize := cPrivateReplicaInfos(privateSectorInfo.Values())
-	defer C.free(unsafe.Pointer(replicasPtr))
-
-	winnersPtr, winnersSize := cCandidates(winners)
-	defer C.free(unsafe.Pointer(winnersPtr))
-
-	proverIDCBytes := C.CBytes(proverID[:])
-	defer C.free(proverIDCBytes)
-
-	resPtr := C.sector_builder_ffi_reexported_generate_post(
-		C.uint64_t(sectorSize),
-		(*[32]C.uint8_t)(unsafe.Pointer(&(randomness)[0])),
-		replicasPtr,
-		replicasSize,
-		winnersPtr,
-		winnersSize,
-		(*[32]C.uint8_t)(proverIDCBytes),
-	)
-	defer C.sector_builder_ffi_reexported_destroy_generate_post_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return nil, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	return goBytes(resPtr.flattened_proofs_ptr, resPtr.flattened_proofs_len), nil
-}
-
-func getAllSealedSectors(sectorBuilderPtr unsafe.Pointer, performHealthchecks bool) ([]SealedSectorMetadata, error) {
-	resPtr := C.sector_builder_ffi_get_sealed_sectors((*C.sector_builder_ffi_SectorBuilder)(sectorBuilderPtr), C.bool(performHealthchecks))
-	defer C.sector_builder_ffi_destroy_get_sealed_sectors_response(resPtr)
-
-	if resPtr.status_code != 0 {
-		return nil, errors.New(C.GoString(resPtr.error_msg))
-	}
-
-	meta, err := goSealedSectorMetadata(resPtr.meta_ptr, resPtr.meta_len)
-	if err != nil {
-		return nil, err
-	}
-
-	return meta, nil
-}

bindings_test.go

-package go_sectorbuilder_test
-
-import (
-	"bytes"
-	"crypto/rand"
-	"encoding/hex"
-	"errors"
-	"fmt"
-	"github.com/stretchr/testify/assert"
-	"io"
-	"io/ioutil"
-	"math/big"
-	"os"
-	"path/filepath"
-	"testing"
-	"time"
-	"unsafe"
-
-	sb "github.com/filecoin-project/go-sectorbuilder"
-	"github.com/filecoin-project/go-sectorbuilder/sealed_sector_health"
-	"github.com/filecoin-project/go-sectorbuilder/sealing_state"
-
-	"github.com/stretchr/testify/require"
-)
-
-func TestSectorBuilderLifecycle(t *testing.T) {
-	ticketA := sb.SealTicket{
-		BlockHeight: 0,
-		TicketBytes: [32]byte{5, 4, 2},
-	}
-
-	ticketB := sb.SealTicket{
-		BlockHeight: 10,
-		TicketBytes: [32]byte{1, 2, 3},
-	}
-
-	seedA := sb.SealSeed{
-		BlockHeight: 50,
-		TicketBytes: [32]byte{7, 4, 2},
-	}
-
-	seedB := sb.SealSeed{
-		BlockHeight: 60,
-		TicketBytes: [32]byte{9, 10, 11},
-	}
-
-	proverID := [32]byte{6, 7, 8}
-
-	metadataDir := requireTempDirPath(t, "metadata")
-	defer os.RemoveAll(metadataDir)
-
-	sealedSectorDir := requireTempDirPath(t, "sealed-sectors")
-	defer os.RemoveAll(sealedSectorDir)
-
-	stagedSectorDir := requireTempDirPath(t, "staged-sectors")
-	defer os.RemoveAll(stagedSectorDir)
-
-	sectorCacheRootDir := requireTempDirPath(t, "sector-cache-root")
-	defer os.RemoveAll(sectorCacheRootDir)
-
-	ptr, err := sb.InitSectorBuilder(1024, 2, 0, metadataDir, proverID, sealedSectorDir, stagedSectorDir, sectorCacheRootDir, 1, 2)
-	require.NoError(t, err)
-	defer sb.DestroySectorBuilder(ptr)
-
-	// verify that we've not yet sealed a sector
-	sealedSectors, err := sb.GetAllSealedSectorsWithHealth(ptr)
-	require.NoError(t, err)
-	require.Equal(t, 0, len(sealedSectors), "expected to see zero sealed sectors")
-
-	// compute the max user-bytes that can fit into a staged sector such that
-	// bit-padding ("preprocessing") expands the file to $SECTOR_SIZE
-	maxPieceSize := sb.GetMaxUserBytesPerStagedSector(1024)
-
-	// create a piece which consumes all available space in a new, staged
-	// sector
-	pieceBytes := make([]byte, maxPieceSize)
-	read, err := io.ReadFull(rand.Reader, pieceBytes)
-	require.Equal(t, uint64(read), maxPieceSize)
-
-	require.NoError(t, err)
-	pieceFileA := requireTempFile(t, bytes.NewReader(pieceBytes), maxPieceSize)
-
-	require.NoError(t, err)
-	pieceFileB := requireTempFile(t, bytes.NewReader(pieceBytes), maxPieceSize)
-
-	// generate piece commitment
-	commP, err := sb.GeneratePieceCommitmentFromFile(pieceFileA, maxPieceSize)
-	require.NoError(t, err)
-
-	publicPieceInfoA := []sb.PublicPieceInfo{{
-		Size:  maxPieceSize,
-		CommP: commP,
-	}}
-
-	preComputedCommD, err := sb.GenerateDataCommitment(1024, publicPieceInfoA)
-	require.NoError(t, err)
-
-	// seek to the beginning
-	_, err = pieceFileA.Seek(0, 0)
-	require.NoError(t, err)
-
-	// write a piece to a staged sector, reducing remaining space to 0
-	sectorIDA, err := sb.AddPieceFromFile(ptr, "snoqualmie", maxPieceSize, pieceFileA)
-	require.NoError(t, err)
-
-	stagedSectors, err := sb.GetAllStagedSectors(ptr)
-	require.NoError(t, err)
-	require.Equal(t, 1, len(stagedSectors))
-	stagedSector := stagedSectors[0]
-	require.Equal(t, uint64(1), stagedSector.SectorID)
-
-	// block until the sector is ready for us to begin sealing
-	statusA, err := pollForSectorSealingStatus(ptr, sectorIDA, sealing_state.FullyPacked, time.Minute)
-	require.NoError(t, err)
-
-	// pre-commit sector to a ticket (in a non-blocking fashion)
-	go func() {
-		out, err := sb.SealPreCommit(ptr, statusA.SectorID, ticketA)
-		require.NoError(t, err)
-		require.Equal(t, sectorIDA, out.SectorID)
-		require.Equal(t, ticketA.TicketBytes, out.Ticket.TicketBytes)
-		require.True(t, bytes.Equal(preComputedCommD[:], out.CommD[:]))
-	}()
-
-	// write a second piece to a staged sector, reducing remaining space to 0
-	sectorIDB, err := sb.AddPieceFromFile(ptr, "duvall", maxPieceSize, pieceFileB)
-	require.NoError(t, err)
-
-	// pre-commit second sector to a ticket too
-	go func() {
-		_, err := sb.SealPreCommit(ptr, sectorIDB, ticketB)
-		require.NoError(t, err)
-	}()
-
-	// block until both sectors have successfully pre-committed
-	statusA, err = pollForSectorSealingStatus(ptr, sectorIDA, sealing_state.PreCommitted, 30*time.Minute)
-	require.NoError(t, err)
-
-	statusB, err := pollForSectorSealingStatus(ptr, sectorIDB, sealing_state.PreCommitted, 30*time.Minute)
-	require.NoError(t, err)
-
-	// commit both sectors concurrently
-	go func() {
-		out, err := sb.SealCommit(ptr, sectorIDA, seedA)
-		require.NoError(t, err)
-		require.Equal(t, sectorIDA, out.SectorID)
-		require.Equal(t, ticketA.TicketBytes, out.Ticket.TicketBytes)
-		require.Equal(t, seedA.TicketBytes, out.Seed.TicketBytes)
-	}()
-
-	go func() {
-		out, err := sb.SealCommit(ptr, sectorIDB, seedB)
-		require.NoError(t, err)
-		require.Equal(t, sectorIDB, out.SectorID)
-	}()
-
-	// block until both sectors have finished sealing (successfully)
-	statusA, err = pollForSectorSealingStatus(ptr, sectorIDA, sealing_state.Committed, 30*time.Minute)
-	require.NoError(t, err)
-
-	statusB, err = pollForSectorSealingStatus(ptr, sectorIDB, sealing_state.Committed, 30*time.Minute)
-	require.NoError(t, err)
-
-	// verify that we used the tickets and seeds we'd intended to use
-	require.Equal(t, ticketA.TicketBytes, statusA.Ticket.TicketBytes)
-	require.Equal(t, ticketB.TicketBytes, statusB.Ticket.TicketBytes)
-	require.Equal(t, seedA.TicketBytes, statusA.Seed.TicketBytes)
-	require.Equal(t, seedB.TicketBytes, statusB.Seed.TicketBytes)
-
-	// verify the seal proof
-	isValid, err := sb.VerifySeal(1024, statusA.CommR, statusA.CommD, proverID, ticketA.TicketBytes, seedA.TicketBytes, sectorIDA, statusA.Proof)
-	require.NoError(t, err)
-	require.True(t, isValid)
-
-	// enforces sort ordering of PublicSectorInfo tuples
-	sectorInfo := sb.NewSortedPublicSectorInfo(sb.PublicSectorInfo{
-		SectorID: statusA.SectorID,
-		CommR:    statusA.CommR,
-	})
-
-	candidates, err := sb.GenerateCandidates(ptr, sectorInfo, [32]byte{}, 2, []uint64{})
-	require.NoError(t, err)
-
-	// generate a PoSt
-	proofs, err := sb.GeneratePoSt(ptr, sectorInfo, [32]byte{}, 2, candidates)
-	require.NoError(t, err)
-
-	// verify the PoSt
-	isValid, err = sb.VerifyPoSt(1024, sectorInfo, [32]byte{}, 2, proofs, candidates, proverID)
-	require.NoError(t, err)
-	require.True(t, isValid)
-
-	sealedSectors, err = sb.GetAllSealedSectorsWithHealth(ptr)
-	require.NoError(t, err)
-	require.Equal(t, 2, len(sealedSectors), "expected to see two sealed sectors")
-	for _, sealedSector := range sealedSectors {
-		require.Equal(t, sealed_sector_health.Ok, sealedSector.Health)
-	}
-
-	// both sealed sectors contain the same data, so either will suffice
-	require.Equal(t, commP, sealedSectors[0].CommD)
-
-	// unseal the sector and retrieve the client's piece, verifying that the
-	// retrieved bytes match what we originally wrote to the staged sector
-	unsealedPieceBytes, err := sb.ReadPieceFromSealedSector(ptr, "snoqualmie")
-	require.NoError(t, err)
-	require.Equal(t, pieceBytes, unsealedPieceBytes)
-}
-
-func TestImportSector(t *testing.T) {
-	challengeCount := uint64(2)
-	poRepProofPartitions := uint8(2)
-	proverID := [32]byte{6, 7, 8}
-	randomness := [32]byte{9, 9, 9}
-	sectorSize := uint64(1024)
-
-	ticket := sb.SealTicket{
-		BlockHeight: 0,
-		TicketBytes: [32]byte{5, 4, 2},
-	}
-
-	seed := sb.SealSeed{
-		BlockHeight: 50,
-		TicketBytes: [32]byte{7, 4, 2},
-	}
-
-	// initialize a sector builder
-	metadataDir := requireTempDirPath(t, "metadata")
-	defer os.RemoveAll(metadataDir)
-
-	sealedSectorsDir := requireTempDirPath(t, "sealed-sectors")
-	defer os.RemoveAll(sealedSectorsDir)
-
-	stagedSectorsDir := requireTempDirPath(t, "staged-sectors")
-	defer os.RemoveAll(stagedSectorsDir)
-
-	sectorCacheRootDir := requireTempDirPath(t, "sector-cache-root-dir")
-	defer os.RemoveAll(sectorCacheRootDir)
-
-	ptr, err := sb.InitSectorBuilder(sectorSize, 2, 0, metadataDir, proverID, sealedSectorsDir, stagedSectorsDir, sectorCacheRootDir, 1, 1)
-	require.NoError(t, err)
-	defer sb.DestroySectorBuilder(ptr)
-
-	sectorID, err := sb.AcquireSectorId(ptr)
-	require.NoError(t, err)
-
-	sectorCacheDirPath := requireTempDirPath(t, "sector-cache-dir")
-	defer os.RemoveAll(sectorCacheDirPath)
-
-	stagedSectorFile := requireTempFile(t, bytes.NewReader([]byte{}), 0)
-	defer stagedSectorFile.Close()
-
-	sealedSectorFile := requireTempFile(t, bytes.NewReader([]byte{}), 0)
-	defer sealedSectorFile.Close()
-
-	unsealOutputFile := requireTempFile(t, bytes.NewReader([]byte{}), 0)
-	defer unsealOutputFile.Close()
-
-	// some rando bytes
-	someBytes := make([]byte, 1016)
-	_, err = io.ReadFull(rand.Reader, someBytes)
-	require.NoError(t, err)
-
-	// write first piece
-	require.NoError(t, err)
-	pieceFileA := requireTempFile(t, bytes.NewReader(someBytes[0:127]), 127)
-
-	commPA, err := sb.GeneratePieceCommitmentFromFile(pieceFileA, 127)
-	require.NoError(t, err)
-
-	// seek back to head (generating piece commitment moves offset)
-	_, err = pieceFileA.Seek(0, 0)
-	require.NoError(t, err)
-
-	// write the first piece using the alignment-free function
-	n, commP, err := sb.StandaloneWriteWithoutAlignment(pieceFileA, 127, stagedSectorFile)
-	require.NoError(t, err)
-	require.Equal(t, int(n), 127)
-	require.Equal(t, commP, commPA)
-
-	// write second piece + alignment
-	require.NoError(t, err)
-	pieceFileB := requireTempFile(t, bytes.NewReader(someBytes[0:508]), 508)
-
-	commPB, err := sb.GeneratePieceCommitmentFromFile(pieceFileB, 508)
-	require.NoError(t, err)
-
-	// seek back to head
-	_, err = pieceFileB.Seek(0, 0)
-	require.NoError(t, err)
-
-	// second piece relies on the alignment-computing version
-	left, tot, commP, err := sb.StandaloneWriteWithAlignment(pieceFileB, 508, stagedSectorFile, []uint64{127})
-	require.NoError(t, err)
-	require.Equal(t, int(left), 381)
-	require.Equal(t, int(tot), 889)
-	require.Equal(t, commP, commPB)
-
-	publicPieces := []sb.PublicPieceInfo{{
-		Size:  127,
-		CommP: commPA,
-	}, {
-		Size:  508,
-		CommP: commPB,
-	}}
-
-	privatePieces := make([]sb.PieceMetadata, len(publicPieces))
-	for i, v := range publicPieces {
-		privatePieces[i] = sb.PieceMetadata{
-			Key:   hex.EncodeToString(v.CommP[:]),
-			Size:  v.Size,
-			CommP: v.CommP,
-		}
-	}
-
-	// pre-commit the sector
-	output, err := sb.StandaloneSealPreCommit(sectorSize, poRepProofPartitions, sectorCacheDirPath, stagedSectorFile.Name(), sealedSectorFile.Name(), sectorID, proverID, ticket.TicketBytes, publicPieces)
-	require.NoError(t, err)
-
-	// commit the sector
-	proof, err := sb.StandaloneSealCommit(sectorSize, poRepProofPartitions, sectorCacheDirPath, sectorID, proverID, ticket.TicketBytes, seed.TicketBytes, publicPieces, output)
-	require.NoError(t, err)
-
-	// verify the 'ole proofy
-	isValid, err := sb.VerifySeal(sectorSize, output.CommR, output.CommD, proverID, ticket.TicketBytes, seed.TicketBytes, sectorID, proof)
-	require.NoError(t, err)
-	require.True(t, isValid, "proof wasn't valid")
-
-	// unseal and verify that things went as we planned
-	require.NoError(t, sb.StandaloneUnseal(sectorSize, poRepProofPartitions, sectorCacheDirPath, sealedSectorFile.Name(), unsealOutputFile.Name(), sectorID, proverID, ticket.TicketBytes, output.CommD))
-	contents, err := ioutil.ReadFile(unsealOutputFile.Name())
-	require.NoError(t, err)
-
-	// unsealed sector includes a bunch of alignment NUL-bytes
-	alignment := make([]byte, 381)
-
-	// verify that we unsealed what we expected to unseal
-	require.Equal(t, someBytes[0:127], contents[0:127])
-	require.Equal(t, alignment, contents[127:508])
-	require.Equal(t, someBytes[0:508], contents[508:1016])
-
-	// verify that the sector builder owns no sealed sectors
-	var sealedSectorPaths []string
-	require.NoError(t, filepath.Walk(sealedSectorsDir, visit(&sealedSectorPaths)))
-	assert.Equal(t, 1, len(sealedSectorPaths), sealedSectorPaths)
-
-	// no sector cache dirs, either
-	var sectorCacheDirPaths []string
-	require.NoError(t, filepath.Walk(sectorCacheRootDir, visit(&sectorCacheDirPaths)))
-	assert.Equal(t, 1, len(sectorCacheDirPaths), sectorCacheDirPaths)
-
-	// generate a PoSt over the proving set before importing, just to exercise
-	// the new API
-	privateInfo := sb.NewSortedPrivateSectorInfo(sb.PrivateSectorInfo{
-		SectorID:         sectorID,
-		CommR:            output.CommR,
-		CacheDirPath:     sectorCacheDirPath,
-		SealedSectorPath: sealedSectorFile.Name(),
-	})
-
-	publicInfo := sb.NewSortedPublicSectorInfo(sb.PublicSectorInfo{
-		SectorID: sectorID,
-		CommR:    output.CommR,
-	})
-
-	candidatesA, err := sb.StandaloneGenerateCandidates(sectorSize, proverID, randomness, challengeCount, privateInfo)
-	require.NoError(t, err)
-
-	proofA, err := sb.StandaloneGeneratePoSt(sectorSize, proverID, privateInfo, randomness, candidatesA)
-	require.NoError(t, err)
-
-	isValid, err = sb.VerifyPoSt(sectorSize, publicInfo, randomness, challengeCount, proofA, candidatesA, proverID)
-	require.NoError(t, err)
-	require.True(t, isValid, "VerifyPoSt rejected the (standalone) proof as invalid")
-
-	// import the sealed sector, transferring ownership to the sector builder
-	err = sb.ImportSealedSector(ptr, sectorID, sectorCacheDirPath, sealedSectorFile.Name(), ticket, seed, output.CommR, output.CommD, output.CommC, output.CommRLast, proof, privatePieces)
-	require.NoError(t, err)
-
-	// it should now have a sealed sector!
-	var sealedSectorPathsB []string
-	require.NoError(t, filepath.Walk(sealedSectorsDir, visit(&sealedSectorPathsB)))
-	assert.Equal(t, 2, len(sealedSectorPathsB), sealedSectorPathsB)
-
-	// it should now have a cache dir and a bunch of goodies in the cache
-	var sectorCacheDirPathsB []string
-	require.NoError(t, filepath.Walk(sectorCacheRootDir, visit(&sectorCacheDirPathsB)))
-	assert.Less(t, 2, len(sectorCacheDirPathsB), sectorCacheDirPathsB)
-
-	// verify that it shows up in sealed sector list
-	metadata, err := sb.GetAllSealedSectorsWithHealth(ptr)
-	require.NoError(t, err)
-	require.Equal(t, 1, len(metadata))
-	require.Equal(t, output.CommD, metadata[0].CommD)
-	require.Equal(t, output.CommR, metadata[0].CommR)
-
-	candidatesB, err := sb.GenerateCandidates(ptr, publicInfo, randomness, challengeCount, []uint64{})
-	require.NoError(t, err)
-	require.Less(t, 0, len(candidatesB))
-
-	// finalize the ticket, but don't do anything with the results (simply
-	// exercise the API)
-	_, err = sb.FinalizeTicket(candidatesB[0].PartialTicket)
-	require.NoError(t, err)
-
-	proofB, err := sb.GeneratePoSt(ptr, publicInfo, randomness, challengeCount, candidatesB)
-	require.NoError(t, err)
-
-	isValid, err = sb.VerifyPoSt(sectorSize, publicInfo, randomness, challengeCount, proofB, candidatesB, proverID)
-	require.NoError(t, err)
-	require.True(t, isValid, "VerifyPoSt rejected the proof as invalid")
-}
-
-func TestJsonMarshalSymmetry(t *testing.T) {
-	for i := 0; i < 100; i++ {
-		xs := make([]sb.PublicSectorInfo, 10)
-		for j := 0; j < 10; j++ {
-			var x sb.PublicSectorInfo
-			_, err := io.ReadFull(rand.Reader, x.CommR[:])
-			require.NoError(t, err)
-
-			n, err := rand.Int(rand.Reader, big.NewInt(500))
-			require.NoError(t, err)
-			x.SectorID = n.Uint64()
-			xs[j] = x
-		}
-		toSerialize := sb.NewSortedPublicSectorInfo(xs...)
-
-		serialized, err := toSerialize.MarshalJSON()
-		require.NoError(t, err)
-
-		var fromSerialized sb.SortedPublicSectorInfo
-		err = fromSerialized.UnmarshalJSON(serialized)
-		require.NoError(t, err)
-
-		require.Equal(t, toSerialize, fromSerialized)
-	}
-}
-
-func pollForSectorSealingStatus(ptr unsafe.Pointer, sectorID uint64, targetState sealing_state.State, timeout time.Duration) (status sb.SectorSealingStatus, retErr error) {
-	timeoutCh := time.After(timeout)
-	lastState := sealing_state.Unknown
-
-	tick := time.Tick(1 * time.Second)
-
-	for {
-		select {
-		case <-timeoutCh:
-			retErr = fmt.Errorf("timed out waiting for sector hit desired state (last state: %s)", lastState)
-			return
-		case <-tick:
-			sealingStatus, err := sb.GetSectorSealingStatusByID(ptr, sectorID)
-			if err != nil {
-				retErr = err
-				return
-			}
-
-			lastState = sealingStatus.State
-
-			if sealingStatus.State == targetState {
-				status = sealingStatus
-				return
-			} else if sealingStatus.State == sealing_state.Failed {
-				retErr = errors.New(sealingStatus.SealErrorMsg)
-				return
-			}
-		}
-	}
-}
-
-func requireTempFile(t *testing.T, fileContentsReader io.Reader, size uint64) *os.File {
-	file, err := ioutil.TempFile("", "")
-	require.NoError(t, err)
-
-	written, err := io.Copy(file, fileContentsReader)
-	require.NoError(t, err)
-	// check that we wrote everything
-	require.Equal(t, int(size), int(written))
-
-	require.NoError(t, file.Sync())
-
-	// seek to the beginning
-	_, err = file.Seek(0, 0)
-	require.NoError(t, err)
-
-	return file
-}
-
-func requireTempDirPath(t *testing.T, prefix string) string {
-	dir, err := ioutil.TempDir("", prefix)
-	require.NoError(t, err)
-
-	return dir
-}
-
-func visit(paths *[]string) filepath.WalkFunc {
-	return func(path string, info os.FileInfo, err error) error {
-		if err != nil {
-			panic(err)
-		}
-		*paths = append(*paths, path)
-		return nil
-	}
-}

filecoin-ffi @ bb699517

+Subproject commit bb699517a5904b3d2549ac97e2b0005ab6471dce

files.go

+package sectorbuilder
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+
+	"golang.org/x/xerrors"
+)
+
+func (sb *SectorBuilder) SectorName(sectorID uint64) string {
+	return fmt.Sprintf("s-%s-%d", sb.Miner, sectorID)
+}
+
+func (sb *SectorBuilder) StagedSectorPath(sectorID uint64) string {
+	return filepath.Join(sb.filesystem.pathFor(dataStaging), sb.SectorName(sectorID))
+}
+
+func (sb *SectorBuilder) unsealedSectorPath(sectorID uint64) string {
+	return filepath.Join(sb.filesystem.pathFor(dataUnsealed), sb.SectorName(sectorID))
+}
+
+func (sb *SectorBuilder) stagedSectorFile(sectorID uint64) (*os.File, error) {
+	return os.OpenFile(sb.StagedSectorPath(sectorID), os.O_RDWR|os.O_CREATE, 0644)
+}
+
+func (sb *SectorBuilder) SealedSectorPath(sectorID uint64) (string, error) {
+	path := filepath.Join(sb.filesystem.pathFor(dataSealed), sb.SectorName(sectorID))
+
+	return path, nil
+}
+
+func (sb *SectorBuilder) sectorCacheDir(sectorID uint64) (string, error) {
+	dir := filepath.Join(sb.filesystem.pathFor(dataCache), sb.SectorName(sectorID))
+
+	err := os.Mkdir(dir, 0755)
+	if os.IsExist(err) {
+		err = nil
+	}
+
+	return dir, err
+}
+
+func (sb *SectorBuilder) GetPath(typ string, sectorName string) (string, error) {
+	_, found := overheadMul[dataType(typ)]
+	if !found {
+		return "", xerrors.Errorf("unknown sector type: %s", typ)
+	}
+
+	return filepath.Join(sb.filesystem.pathFor(dataType(typ)), sectorName), nil
+}
+
+func (sb *SectorBuilder) TrimCache(sectorID uint64) error {
+	dir, err := sb.sectorCacheDir(sectorID)
+	if err != nil {
+		return xerrors.Errorf("getting cache dir: %w", err)
+	}
+
+	files, err := ioutil.ReadDir(dir)
+	if err != nil {
+		return xerrors.Errorf("readdir: %w", err)
+	}
+
+	for _, file := range files {
+		if !strings.HasSuffix(file.Name(), ".dat") { // _aux probably
+			continue
+		}
+		if strings.HasSuffix(file.Name(), "-data-tree-r-last.dat") { // Want to keep
+			continue
+		}
+
+		if err := os.Remove(filepath.Join(dir, file.Name())); err != nil {
+			return xerrors.Errorf("rm %s: %w", file.Name(), err)
+		}
+	}
+
+	return nil
+}
+
+func toReadableFile(r io.Reader, n int64) (*os.File, func() error, error) {
+	f, ok := r.(*os.File)
+	if ok {
+		return f, func() error { return nil }, nil
+	}
+
+	var w *os.File
+
+	f, w, err := os.Pipe()
+	if err != nil {
+		return nil, nil, err
+	}
+
+	var wait sync.Mutex
+	var werr error
+
+	wait.Lock()
+	go func() {
+		defer wait.Unlock()
+
+		var copied int64
+		copied, werr = io.CopyN(w, r, n)
+		if werr != nil {
+			log.Warnf("toReadableFile: copy error: %+v", werr)
+		}
+
+		err := w.Close()
+		if werr == nil && err != nil {
+			werr = err
+			log.Warnf("toReadableFile: close error: %+v", err)
+			return
+		}
+		if copied != n {
+			log.Warnf("copied different amount than expected: %d != %d", copied, n)
+			werr = xerrors.Errorf("copied different amount than expected: %d != %d", copied, n)
+		}
+	}()
+
+	return f, func() error {
+		wait.Lock()
+		return werr
+	}, nil
+}

fs.go

+package sectorbuilder
+
+import (
+	"os"
+	"path/filepath"
+	"sync"
+	"syscall"
+
+	"golang.org/x/xerrors"
+)
+
+type dataType string
+
+const (
+	dataCache    dataType = "cache"
+	dataStaging  dataType = "staging"
+	dataSealed   dataType = "sealed"
+	dataUnsealed dataType = "unsealed"
+)
+
+var overheadMul = map[dataType]uint64{ // * sectorSize
+	dataCache:    11, // TODO: check if true for 32G sectors
+	dataStaging:  1,
+	dataSealed:   1,
+	dataUnsealed: 1,
+}
+
+type fs struct {
+	path string
+
+	// in progress actions
+
+	reserved map[dataType]uint64
+
+	lk sync.Mutex
+}
+
+func openFs(dir string) *fs {
+	return &fs{
+		path:     dir,
+		reserved: map[dataType]uint64{},
+	}
+}
+
+func (f *fs) init() error {
+	for _, dir := range []string{f.path,
+		f.pathFor(dataCache),
+		f.pathFor(dataStaging),
+		f.pathFor(dataSealed),
+		f.pathFor(dataUnsealed)} {
+		if err := os.Mkdir(dir, 0755); err != nil {
+			if os.IsExist(err) {
+				continue
+			}
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (f *fs) pathFor(typ dataType) string {
+	_, found := overheadMul[typ]
+	if !found {
+		panic("unknown data path requested")
+	}
+
+	return filepath.Join(f.path, string(typ))
+}
+
+func (f *fs) reservedBytes() int64 {
+	var out int64
+	for _, r := range f.reserved {
+		out += int64(r)
+	}
+	return out
+}
+
+func (f *fs) reserve(typ dataType, size uint64) error {
+	f.lk.Lock()
+	defer f.lk.Unlock()
+
+	var fsstat syscall.Statfs_t
+
+	if err := syscall.Statfs(f.pathFor(typ), &fsstat); err != nil {
+		return err
+	}
+
+	fsavail := int64(fsstat.Bavail) * int64(fsstat.Bsize)
+
+	avail := fsavail - f.reservedBytes()
+
+	need := overheadMul[typ] * size
+
+	if int64(need) > avail {
+		return xerrors.Errorf("not enough space in '%s', need %dB, available %dB (fs: %dB, reserved: %dB)",
+			f.path,
+			need,
+			avail,
+			fsavail,
+			f.reservedBytes())
+	}
+
+	f.reserved[typ] += need
+
+	return nil
+}
+
+func (f *fs) free(typ dataType, sectorSize uint64) {
+	f.lk.Lock()
+	defer f.lk.Unlock()
+
+	f.reserved[typ] -= overheadMul[typ] * sectorSize
+}

go.mod

 module github.com/filecoin-project/go-sectorbuilder
 
-go 1.12
+go 1.13
 
 require (
-	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/ipfs/go-log v0.0.1
-	github.com/kr/pretty v0.1.0 // indirect
-	github.com/mattn/go-colorable v0.1.4 // indirect
-	github.com/pkg/errors v0.8.1
+	github.com/GeertJohan/go.rice v1.0.0
+	github.com/fatih/color v1.7.0 // indirect
+	github.com/filecoin-project/filecoin-ffi v0.0.0-20191219131535-bb699517a590
+	github.com/filecoin-project/go-address v0.0.0-20191219011437-af739c490b4f
+	github.com/gogo/protobuf v1.3.1 // indirect
+	github.com/gopherjs/gopherjs v0.0.0-20190812055157-5d271430af9f // indirect
+	github.com/ipfs/go-cid v0.0.4 // indirect
+	github.com/ipfs/go-datastore v0.1.1
+	github.com/ipfs/go-ipld-format v0.0.2 // indirect
+	github.com/ipfs/go-log v1.0.0
+	github.com/jbenet/goprocess v0.1.3 // indirect
+	github.com/mattn/go-colorable v0.1.2 // indirect
+	github.com/mattn/go-isatty v0.0.9 // indirect
+	github.com/mattn/go-runewidth v0.0.4 // indirect
+	github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1
+	github.com/minio/sha256-simd v0.1.1 // indirect
+	github.com/mr-tron/base58 v1.1.3 // indirect
+	github.com/otiai10/copy v1.0.2
+	github.com/polydawn/refmt v0.0.0-20190809202753-05966cbd336a // indirect
+	github.com/smartystreets/assertions v1.0.1 // indirect
+	github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337 // indirect
 	github.com/stretchr/testify v1.4.0
-	golang.org/x/net v0.0.0-20190923162816-aa69164e4478 // indirect
-	golang.org/x/sys v0.0.0-20190922100055-0a153f010e69 // indirect
-	gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
-	gopkg.in/yaml.v2 v2.2.4 // indirect
+	github.com/warpfork/go-wish v0.0.0-20190328234359-8b3e70f8e830 // indirect
+	go.opencensus.io v0.22.2
+	go.uber.org/multierr v1.4.0
+	golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413 // indirect
+	golang.org/x/sys v0.0.0-20191210023423-ac6580df4449 // indirect
+	golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
+	gopkg.in/cheggaaa/pb.v1 v1.0.28
 )
+
+replace github.com/golangci/golangci-lint => github.com/golangci/golangci-lint v1.18.0
+
+replace github.com/filecoin-project/filecoin-ffi => ./extern/filecoin-ffi

go.sum

-github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
+cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
+github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/GeertJohan/go.incremental v1.0.0/go.mod h1:6fAjUhbVuX1KcMD3c8TEgVUqmo4seqhv0i0kdATSkM0=
+github.com/GeertJohan/go.rice v1.0.0 h1:KkI6O9uMaQU3VEKaj01ulavtF7o1fWT7+pk/4voiMLQ=
+github.com/GeertJohan/go.rice v1.0.0/go.mod h1:eH6gbSOAUv07dQuZVnBmoDP8mgsM1rtixis4Tib9if0=
+github.com/akavel/rsrc v0.8.0/go.mod h1:uLoCtb9J+EyAqh+26kdrTgmzRBFPGOolLWKpdxkKq+c=
+github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/daaku/go.zipexe v1.0.0 h1:VSOgZtH418pH9L16hC/JrgSNJbbAL26pj7lmD1+CGdY=
+github.com/daaku/go.zipexe v1.0.0/go.mod h1:z8IiR6TsVLEYKwXAoE/I+8ys/sDkgTzSL0CLnGVd57E=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE=
+github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
+github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
+github.com/filecoin-project/go-address v0.0.0-20191219011437-af739c490b4f h1:L2jaVU8TvWTx7iZPhlYvUE8vkoOnj778XuKavz8W36g=
+github.com/filecoin-project/go-address v0.0.0-20191219011437-af739c490b4f/go.mod h1:rCbpXPva2NKF9/J4X6sr7hbKBgQCxyFtRj7KOZqoIms=
+github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03 h1:2pMXdBnCiXjfCYx/hLqFxccPoqsSveQFxVLvNxy9bus=
+github.com/filecoin-project/go-crypto v0.0.0-20191218222705-effae4ea9f03/go.mod h1:+viYnvGtUTgJRdy6oaeF4MTFKAfatX071MPDPBL11EQ=
+github.com/go-check/check v0.0.0-20180628173108-788fd7840127 h1:0gkP6mzaMqkmpcJYCFOLkIBwI7xFExG03bbkOkCvUPI=
+github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98=
 github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
-github.com/ipfs/go-log v0.0.1 h1:9XTUN/rW64BCG1YhPK9Hoy3q8nr4gOmHHBpgFdfw6Lc=
-github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM=
+github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls=
+github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o=
+github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
+github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I=
+github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
+github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
+github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
+github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
+github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/gopherjs/gopherjs v0.0.0-20190812055157-5d271430af9f h1:KMlcu9X58lhTA/KrfX8Bi1LQSO4pzoVjTiL3h4Jk+Zk=
+github.com/gopherjs/gopherjs v0.0.0-20190812055157-5d271430af9f/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU=
+github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48=
+github.com/ipfs/go-block-format v0.0.2 h1:qPDvcP19izTjU8rgo6p7gTXZlkMkF5bz5G3fqIsSCPE=
+github.com/ipfs/go-block-format v0.0.2/go.mod h1:AWR46JfpcObNfg3ok2JHDUfdiHRgWhJgCQF+KIgOPJY=
+github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
+github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
+github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
+github.com/ipfs/go-cid v0.0.4 h1:UlfXKrZx1DjZoBhQHmNHLC1fK1dUJDN20Y28A7s+gJ8=
+github.com/ipfs/go-cid v0.0.4/go.mod h1:4LLaPOQwmk5z9LBgQnpkivrx8BJjUyGwTXCd5Xfj6+M=
+github.com/ipfs/go-datastore v0.1.1 h1:F4k0TkTAZGLFzBOrVKDAvch6JZtuN4NHkfdcEZL50aI=
+github.com/ipfs/go-datastore v0.1.1/go.mod h1:w38XXW9kVFNp57Zj5knbKWM2T+KOZCGDRVNdgPHtbHw=
+github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw=
+github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50=
+github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc=
+github.com/ipfs/go-ipld-cbor v0.0.3 h1:ENsxvybwkmke7Z/QJOmeJfoguj6GH3Y0YOaGrfy9Q0I=
+github.com/ipfs/go-ipld-cbor v0.0.3/go.mod h1:wTBtrQZA3SoFKMVkp6cn6HMRteIB1VsmHA0AQFOn7Nc=
+github.com/ipfs/go-ipld-format v0.0.1/go.mod h1:kyJtbkDALmFHv3QR6et67i35QzO3S0dCDnkOJhcZkms=
+github.com/ipfs/go-ipld-format v0.0.2 h1:OVAGlyYT6JPZ0pEfGntFPS40lfrDmaDbQwNHEY2G9Zs=
+github.com/ipfs/go-ipld-format v0.0.2/go.mod h1:4B6+FM2u9OJ9zCV+kSbgFAZlOrv1Hqbf0INGQgiKf9k=
+github.com/ipfs/go-log v1.0.0 h1:BW3LQIiZzpNyolt84yvKNCd3FU+AK4VDw1hnHR+1aiI=
+github.com/ipfs/go-log v1.0.0/go.mod h1:JO7RzlMK6rA+CIxFMLOuB6Wf5b81GDiKElL7UPSIKjA=
+github.com/ipsn/go-secp256k1 v0.0.0-20180726113642-9d62b9f0bc52 h1:QG4CGBqCeuBo6aZlGAamSkxWdgWfZGeE49eUOWJPA4c=
+github.com/ipsn/go-secp256k1 v0.0.0-20180726113642-9d62b9f0bc52/go.mod h1:fdg+/X9Gg4AsAIzWpEHwnqd+QY3b7lajxyjE1m4hkq4=
+github.com/jbenet/go-cienv v0.1.0/go.mod h1:TqNnHUmJgXau0nCzC7kXWeotg3J9W34CUv5Djy1+FlA=
+github.com/jbenet/goprocess v0.0.0-20160826012719-b497e2f366b8/go.mod h1:Ly/wlsjFq/qrU3Rar62tu1gASgGw6chQbSh/XgIIXCY=
+github.com/jbenet/goprocess v0.1.3 h1:YKyIEECS/XvcfHtBzxtjBBbWK+MbvA6dG8ASiqwvr10=
+github.com/jbenet/goprocess v0.1.3/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4=
+github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
+github.com/jtolds/gls v4.2.1+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
+github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
-github.com/kisielk/gotool v1.0.0 h1:AV2c/EiW3KqPNT9ZKl07ehoAGi4C5/01Cfbblndcapg=
+github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
-github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg=
-github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
-github.com/mattn/go-colorable v0.1.4 h1:snbPLB8fVfU9iwbbo30TPtbLRzwWu6aJS6Xh4eaaviA=
-github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
-github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw=
-github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
-github.com/mattn/go-isatty v0.0.8 h1:HLtExJ+uU2HOZ+wI0Tt5DtUDrx8yhUqDcp7fYERX4CE=
+github.com/mattn/go-colorable v0.1.2 h1:/bC9yWikZXAL9uJdulbSfyVNIR3n3trXl+v8+1sx8mU=
+github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
 github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
-github.com/opentracing/opentracing-go v1.0.2 h1:3jA2P6O1F9UOrWVpwrIo17pu01KWvNWg4X946/Y5Zwg=
-github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
+github.com/mattn/go-isatty v0.0.9 h1:d5US/mDsogSGW37IV293h//ZFaeajb69h+EHFsv2xGg=
+github.com/mattn/go-isatty v0.0.9/go.mod h1:YNRxwqDuOph6SZLI9vUUz6OYw3QyUt7WiY2yME+cCiQ=
+github.com/mattn/go-runewidth v0.0.4 h1:2BvfKmzob6Bmd4YsL0zygOqfdFnK7GR4QL06Do4/p7Y=
+github.com/mattn/go-runewidth v0.0.4/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
+github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g=
+github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ=
+github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
+github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
+github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8=
+github.com/mr-tron/base58 v1.1.2 h1:ZEw4I2EgPKDJ2iEw0cNmLB3ROrEmkOtXIkaG7wZg+78=
+github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
+github.com/mr-tron/base58 v1.1.3 h1:v+sk57XuaCKGXpWtVBX8YJzO7hMGx4Aajh4TQbdEFdc=
+github.com/mr-tron/base58 v1.1.3/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
+github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI=
+github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA=
+github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA=
+github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs=
+github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U=
+github.com/multiformats/go-multihash v0.0.10 h1:lMoNbh2Ssd9PUF74Nz008KGzGPlfeV6wH3rit5IIGCM=
+github.com/multiformats/go-multihash v0.0.10/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew=
+github.com/multiformats/go-varint v0.0.2 h1:6sUvyh2YHpJCb8RZ6eYzj6iJQ4+chWYmyIHxszqlPTA=
+github.com/multiformats/go-varint v0.0.2/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
+github.com/nkovacs/streamquote v0.0.0-20170412213628-49af9bddb229/go.mod h1:0aYXnNPJ8l7uZxf45rWW1a/uME32OF0rhiYGNQ2oF2E=
+github.com/opentracing/opentracing-go v1.1.0 h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsqf19k25Ur8rU=
+github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
+github.com/otiai10/copy v1.0.2 h1:DDNipYy6RkIkjMwy+AWzgKiNTyj2RUI9yEMeETEpVyc=
+github.com/otiai10/copy v1.0.2/go.mod h1:c7RpqBkwMom4bYTSkLSym4VSJz/XtncWRAj/J4PEIMY=
+github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95 h1:+OLn68pqasWca0z5ryit9KGfp3sUsW4Lqg32iRMJyzs=
+github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95/go.mod h1:9qAhocn7zKJG+0mI8eUu6xqkFDYS2kb2saOteoSB3cE=
+github.com/otiai10/mint v1.3.0 h1:Ady6MKVezQwHBkGzLFbrsywyp09Ah7rkmfjV3Bcr5uc=
+github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT91xUo=
 github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/polydawn/refmt v0.0.0-20190221155625-df39d6c2d992/go.mod h1:uIp+gprXxxrWSjjklXD+mN4wed/tMfjMMmN/9+JsA9o=
+github.com/polydawn/refmt v0.0.0-20190807091052-3d65705ee9f1/go.mod h1:uIp+gprXxxrWSjjklXD+mN4wed/tMfjMMmN/9+JsA9o=
+github.com/polydawn/refmt v0.0.0-20190809202753-05966cbd336a h1:hjZfReYVLbqFkAtr2us7vdy04YWz3LVAirzP7reh8+M=
+github.com/polydawn/refmt v0.0.0-20190809202753-05966cbd336a/go.mod h1:uIp+gprXxxrWSjjklXD+mN4wed/tMfjMMmN/9+JsA9o=
+github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/assertions v1.0.1 h1:voD4ITNjPL5jjBfgR/r8fPIIBrliWrWHeiJApdr3r4w=
+github.com/smartystreets/assertions v1.0.1/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM=
+github.com/smartystreets/goconvey v0.0.0-20190222223459-a17d461953aa/go.mod h1:2RVY1rIf+2J2o/IM9+vPq9RzmHDSseB7FoXiSNIUsoU=
+github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337 h1:WN9BUFbdyOsSH/XohnWpXOlq9NBD5sGAB2FciQMUEe8=
+github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
+github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
+github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
-github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
-github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc h1:9lDbC6Rz4bwmou+oE6Dt4Cb2BGMur5eR/GYptkKUVHo=
-github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM=
+github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
+github.com/valyala/fasttemplate v1.0.1/go.mod h1:UQGH1tvbgY+Nz5t2n7tXsz52dQxojPUpymEIMZ47gx8=
+github.com/warpfork/go-wish v0.0.0-20180510122957-5ad1f5abf436/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw=
+github.com/warpfork/go-wish v0.0.0-20190328234359-8b3e70f8e830 h1:8kxMKmKzXXL4Ru1nyhvdms/JjWt+3YLpvRb/bAjO/y0=
+github.com/warpfork/go-wish v0.0.0-20190328234359-8b3e70f8e830/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw=
+github.com/whyrusleeping/cbor-gen v0.0.0-20191216205031-b047b6acb3c0 h1:efb/4CnrubzNGqQOeHErxyQ6rIsJb7GcgeSDF7fqWeI=
+github.com/whyrusleeping/cbor-gen v0.0.0-20191216205031-b047b6acb3c0/go.mod h1:xdlJQaiqipF0HW+Mzpg7XRM3fWbGvfgFlcppuvlkIvY=
+go.opencensus.io v0.22.2 h1:75k/FF0Q2YM8QYo07VPddOLBslDt1MZOdEslOHvmzAs=
+go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
+go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
+go.uber.org/atomic v1.5.0 h1:OI5t8sDa1Or+q8AeE+yKeB/SDYioSHAgcVljj9JIETY=
+go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
+go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
+go.uber.org/multierr v1.4.0 h1:f3WCSC2KzAcBXGATIxAB1E2XuCpNU255wNKZ505qi3E=
+go.uber.org/multierr v1.4.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
+go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4=
+go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
+go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM=
+go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
+golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/net v0.0.0-20190227160552-c95aed5357e7 h1:C2F/nMkR/9sfUTpvR3QrjBuTdvMUC/cFajkphs1YLQo=
-golang.org/x/net v0.0.0-20190227160552-c95aed5357e7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/net v0.0.0-20190923162816-aa69164e4478 h1:l5EDrHhldLYb3ZRHDUhXF7Om7MvYXnkV9/iQNo1lX6g=
-golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413 h1:ULYEB3JvPRE/IfO+9uO7vKV/xzVTO7XPAwm8xbf4w2g=
+golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
+golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
+golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
+golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
+golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
+golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223 h1:DH4skfRX4EBpamg7iV4ZlCpblAHI6s6TDM39bFZumv8=
+golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190922100055-0a153f010e69 h1:rOhMmluY6kLMhdnrivzec6lLgaVbMHMn2ISQXJeJ5EM=
-golang.org/x/sys v0.0.0-20190922100055-0a153f010e69/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191210023423-ac6580df4449 h1:gSbV7h1NRL2G1xTg/owz62CST1oJBmxy4QpMMregXVQ=
+golang.org/x/sys v0.0.0-20191210023423-ac6580df4449/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/tools v0.0.0-20180221164845-07fd8470d635 h1:2eB4G6bDQDeP69ZXbOKC00S2Kf6TIiRS+DzfKsKeQU0=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
+golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5 h1:hKsoRgsbwY1NafxrwTs+k64bikrLBkAgPir1TNCj3Zs=
+golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
+google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
+google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
+google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
+google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
+google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/cheggaaa/pb.v1 v1.0.28 h1:n1tBJnnK2r7g9OW2btFH91V92STTUevLXYFb8gy9EMk=
+gopkg.in/cheggaaa/pb.v1 v1.0.28/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw=
+gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
+honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=

install-rust-fil-sector-builder

-#!/usr/bin/env bash
-
-set -Eeo pipefail
-
-cd "$(dirname "${BASH_SOURCE[0]}")"
-source "install-shared.bash"
-
-subm_dir="rust-fil-sector-builder"
-
-git submodule update --init --recursive $subm_dir
-
-if [ "${SB_BUILD_FROM_SOURCE}" != "1" ] && download_release_tarball tarball_path "${subm_dir}"; then
-    tmp_dir=$(mktemp -d)
-    tar -C "$tmp_dir" -xzf "$tarball_path"
-
-    cp "${tmp_dir}/include/sector_builder_ffi.h" .
-    cp "${tmp_dir}/lib/libsector_builder_ffi.a" .
-    cp "${tmp_dir}/lib/pkgconfig/sector_builder_ffi.pc" .
-
-    cp "${tmp_dir}/bin/paramcache" .
-
-    (>&2 echo "successfully installed prebuilt libsector_builder")
-else
-    (>&2 echo "building libsector_builder from local sources (dir = ${subm_dir})")
-    build_from_source "${subm_dir}"
-
-    mkdir -p include
-    mkdir -p lib/pkgconfig
-
-    find "${subm_dir}/target/release" -type f -name sector_builder_ffi.h -exec cp -- "{}" . \;
-    find "${subm_dir}/target/release" -type f -name libsector_builder_ffi.a -exec cp -- "{}" . \;
-    find "${subm_dir}" -type f -name sector_builder_ffi.pc -exec cp -- "{}" . \;
-
-    if [[ ! -f "./sector_builder_ffi.h" ]]; then
-        (>&2 echo "failed to install sector_builder_ffi.h")
-        exit 1
-    fi
-
-    if [[ ! -f "./libsector_builder_ffi.a" ]]; then
-        (>&2 echo "failed to install libsector_builder_ffi.a")
-        exit 1
-    fi
-
-    if [[ ! -f "./sector_builder_ffi.pc" ]]; then
-        (>&2 echo "failed to install sector_builder_ffi.pc")
-        exit 1
-    fi
-
-    (>&2 echo "WARNING: paramcache was not installed - you may wish to 'cargo install' it")
-
-    (>&2 echo "successfully built and installed libsector_builder from source")
-fi

install-shared.bash

-#!/usr/bin/env bash
-
-download_release_tarball() {
-    __resultvar=$1
-    __submodule_path=$2
-    __repo_name=$(echo $2 | cut -d '/' -f 1)
-    __release_name="${__repo_name}-$(uname)"
-    __release_sha1=$(git rev-parse HEAD:"${__submodule_path}")
-    __release_tag="${__release_sha1:0:16}"
-    __release_tag_url="https://api.github.com/repos/filecoin-project/${__repo_name}/releases/tags/${__release_tag}"
-
-    echo "acquiring release @ ${__release_tag}"
-
-    __release_response=$(curl \
-        --retry 3 \
-        --location $__release_tag_url)
-
-    __release_url=$(echo $__release_response | jq -r ".assets[] | select(.name | contains(\"${__release_name}\")) | .url")
-
-    if [[ -z "$__release_url" ]]; then
-        (>&2 echo "failed to download release (tag URL: ${__release_tag_url}, response: ${__release_response})")
-        return 1
-    fi
-
-    __tar_path="/tmp/${__release_name}_$(basename ${__release_url}).tar.gz"
-
-    __asset_url=$(curl \
-        --head \
-        --retry 3 \
-        --header "Accept:application/octet-stream" \
-        --location \
-        --output /dev/null \
-        -w %{url_effective} \
-        "$__release_url")
-
-    curl --retry 3 --output "${__tar_path}" "$__asset_url"
-    if [[ $? -ne "0" ]]; then
-        (>&2 echo "failed to download release asset (tag URL: ${__release_tag_url}, asset URL: ${__asset_url})")
-        return 1
-    fi
-
-    eval $__resultvar="'$__tar_path'"
-}
-
-build_from_source() {
-    __submodule_path=$1
-    __submodule_sha1=$(git rev-parse @:"${__submodule_path}")
-    __submodule_sha1_truncated="${__submodule_sha1:0:16}"
-
-    echo "building from source @ ${__submodule_sha1_truncated}"
-
-    if ! [ -x "$(command -v cargo)" ]; then
-        (>&2 echo 'Error: cargo is not installed.')
-        (>&2 echo 'Install Rust toolchain to resolve this problem.')
-        exit 1
-    fi
-
-    if ! [ -x "$(command -v rustup)" ]; then
-        (>&2 echo 'Error: rustup is not installed.')
-        (>&2 echo 'Install Rust toolchain installer to resolve this problem.')
-        exit 1
-    fi
-
-    pushd $__submodule_path
-
-    cargo --version
-
-    if [[ -f "./scripts/build-release.sh" ]]; then
-        ./scripts/build-release.sh $(cat rust-toolchain)
-    else
-        cargo build --release --all
-    fi
-
-    popd
-}

mock.go

+package sectorbuilder
+
+import (
+	"github.com/filecoin-project/go-address"
+	"github.com/ipfs/go-datastore"
+)
+
+func TempSectorbuilderDir(dir string, sectorSize uint64, ds datastore.Batching) (*SectorBuilder, error) {
+	addr, err := address.NewFromString("t3vfxagwiegrywptkbmyohqqbfzd7xzbryjydmxso4hfhgsnv6apddyihltsbiikjf3lm7x2myiaxhuc77capq")
+	if err != nil {
+		return nil, err
+	}
+
+	sb, err := New(&Config{
+		SectorSize: sectorSize,
+
+		Dir: dir,
+
+		WorkerThreads: 2,
+		Miner:         addr,
+	}, ds)
+	if err != nil {
+		return nil, err
+	}
+
+	return sb, nil
+}

paramfetch/paramfetch.go

+package paramfetch
+
+import (
+	"encoding/hex"
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+
+	rice "github.com/GeertJohan/go.rice"
+	logging "github.com/ipfs/go-log"
+	"github.com/minio/blake2b-simd"
+	"go.uber.org/multierr"
+	"golang.org/x/xerrors"
+	pb "gopkg.in/cheggaaa/pb.v1"
+)
+
+var log = logging.Logger("paramfetch")
+
+//const gateway = "http://198.211.99.118/ipfs/"
+const gateway = "https://ipfs.io/ipfs/"
+const paramdir = "/var/tmp/filecoin-proof-parameters"
+const dirEnv = "FIL_PROOFS_PARAMETER_CACHE"
+
+type paramFile struct {
+	Cid        string `json:"cid"`
+	Digest     string `json:"digest"`
+	SectorSize uint64 `json:"sector_size"`
+}
+
+type fetch struct {
+	wg      sync.WaitGroup
+	fetchLk sync.Mutex
+
+	errs []error
+}
+
+func getParamDir() string {
+	if os.Getenv(dirEnv) == "" {
+		return paramdir
+	}
+	return os.Getenv(dirEnv)
+}
+
+func GetParams(storageSize uint64) error {
+	if err := os.Mkdir(getParamDir(), 0755); err != nil && !os.IsExist(err) {
+		return err
+	}
+
+	var params map[string]paramFile
+
+	paramBytes := rice.MustFindBox("proof-params").MustBytes("parameters.json")
+	if err := json.Unmarshal(paramBytes, &params); err != nil {
+		return err
+	}
+
+	ft := &fetch{}
+
+	for name, info := range params {
+		if storageSize != info.SectorSize && strings.HasSuffix(name, ".params") {
+			continue
+		}
+
+		ft.maybeFetchAsync(name, info)
+	}
+
+	return ft.wait()
+}
+
+func (ft *fetch) maybeFetchAsync(name string, info paramFile) {
+	ft.wg.Add(1)
+
+	go func() {
+		defer ft.wg.Done()
+
+		path := filepath.Join(getParamDir(), name)
+
+		err := ft.checkFile(path, info)
+		if !os.IsNotExist(err) && err != nil {
+			log.Warn(err)
+		}
+		if err == nil {
+			return
+		}
+
+		ft.fetchLk.Lock()
+		defer ft.fetchLk.Unlock()
+
+		if err := doFetch(path, info); err != nil {
+			ft.errs = append(ft.errs, xerrors.Errorf("fetching file %s failed: %w", path, err))
+			return
+		}
+		err = ft.checkFile(path, info)
+		if err != nil {
+			ft.errs = append(ft.errs, xerrors.Errorf("checking file %s failed: %w", path, err))
+			err := os.Remove(path)
+			if err != nil {
+				ft.errs = append(ft.errs, xerrors.Errorf("remove file %s failed: %w", path, err))
+			}
+		}
+	}()
+}
+
+func (ft *fetch) checkFile(path string, info paramFile) error {
+	if os.Getenv("TRUST_PARAMS") == "1" {
+		log.Warn("Assuming parameter files are ok. DO NOT USE IN PRODUCTION")
+		return nil
+	}
+
+	f, err := os.Open(path)
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+
+	h := blake2b.New512()
+	if _, err := io.Copy(h, f); err != nil {
+		return err
+	}
+
+	sum := h.Sum(nil)
+	strSum := hex.EncodeToString(sum[:16])
+	if strSum == info.Digest {
+		log.Infof("Parameter file %s is ok", path)
+		return nil
+	}
+
+	return xerrors.Errorf("checksum mismatch in param file %s, %s != %s", path, strSum, info.Digest)
+}
+
+func (ft *fetch) wait() error {
+	ft.wg.Wait()
+	return multierr.Combine(ft.errs...)
+}
+
+func doFetch(out string, info paramFile) error {
+	gw := os.Getenv("IPFS_GATEWAY")
+	if gw == "" {
+		gw = gateway
+	}
+	log.Infof("Fetching %s from %s", out, gw)
+
+	outf, err := os.OpenFile(out, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
+	if err != nil {
+		return err
+	}
+	defer outf.Close()
+
+	fStat, err := outf.Stat()
+	if err != nil {
+		return err
+	}
+	header := http.Header{}
+	header.Set("Range", "bytes="+strconv.FormatInt(fStat.Size(), 10)+"-")
+	url, err := url.Parse(gw + info.Cid)
+	if err != nil {
+		return err
+	}
+	log.Infof("GET %s", url)
+
+	req := http.Request{
+		Method: "GET",
+		URL:    url,
+		Header: header,
+		Close:  true,
+	}
+
+	resp, err := http.DefaultClient.Do(&req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+
+	bar := pb.New64(resp.ContentLength)
+	bar.Units = pb.U_BYTES
+	bar.ShowSpeed = true
+	bar.Start()
+
+	_, err = io.Copy(outf, bar.NewProxyReader(resp.Body))
+
+	bar.Finish()
+
+	return err
+}

paramfetch/proof-params/parameters.json

+{
+  "v20-proof-of-spacetime-election-5f585aca354eb68e411c8582ed0efd800792430e4e76d73468c4fc03f1a8d6d2.params": {
+    "cid": "QmX7tYeNPWae2fjZ3Am6GB9dmHvLqvoz8dKo3PR98VYxH9",
+    "digest": "39a9edec3355516674f0d12b926be493",
+    "sector_size": 34359738368
+  },
+  "v20-proof-of-spacetime-election-5f585aca354eb68e411c8582ed0efd800792430e4e76d73468c4fc03f1a8d6d2.vk": {
+    "cid": "QmbNGx7pNbGiEr8ykoHxVXHW2LNSmGdsxKtj1onZCyguCX",
+    "digest": "0227ae7df4f2affe529ebafbbc7540ee",
+    "sector_size": 34359738368
+  },
+  "v20-proof-of-spacetime-election-a4e18190d4b4657ba1b4d08a341871b2a6f398e327cb9951b28ab141fbdbf49d.params": {
+    "cid": "QmRGZsNp4mp1cZshcXqt3VMuWscAEsiMa2iepF4CsWWoiv",
+    "digest": "991041a354b12c280542741f58c7f2ca",
+    "sector_size": 1024
+  },
+  "v20-proof-of-spacetime-election-a4e18190d4b4657ba1b4d08a341871b2a6f398e327cb9951b28ab141fbdbf49d.vk": {
+    "cid": "QmWpmrhCGVcfqLyqp5oGAnhPmCE5hGTPaauHi25mpQwRSU",
+    "digest": "91fac550e1f9bccab213830bb0c85bd6",
+    "sector_size": 1024
+  },
+  "v20-proof-of-spacetime-election-a9eb6d90b896a282ec2d3a875c6143e3fcff778f0da1460709e051833651559b.params": {
+    "cid": "QmenSZXh1EsSyHiSRvA6wb8yaPhYBTjrKehJw96Px5HnN4",
+    "digest": "6322eacd2773163ddd51f9ca7d645fc4",
+    "sector_size": 1073741824
+  },
+  "v20-proof-of-spacetime-election-a9eb6d90b896a282ec2d3a875c6143e3fcff778f0da1460709e051833651559b.vk": {
+    "cid": "QmPvZoMKofw6eDhDg5ESJA2QAZP8HvM6qMQk7fw4pq9bQf",
+    "digest": "0df62745fceac922e3e70847cfc70b52",
+    "sector_size": 1073741824
+  },
+  "v20-proof-of-spacetime-election-bf872523641b1de33553db2a177df13e412d7b3b0103e6696ae0a1cf5d525259.params": {
+    "cid": "QmVibFqzkZoL8cwQmzj8njPokCQGCCx4pBcUH77bzgJgV9",
+    "digest": "de9d71e672f286706a1673bd57abdaac",
+    "sector_size": 16777216
+  },
+  "v20-proof-of-spacetime-election-bf872523641b1de33553db2a177df13e412d7b3b0103e6696ae0a1cf5d525259.vk": {
+    "cid": "QmZa5FX27XyiEXQQLQpHqtMJKLzrcY8wMuj3pxzmSimSyu",
+    "digest": "7f796d3a0f13499181e44b5eee0cc744",
+    "sector_size": 16777216
+  },
+  "v20-proof-of-spacetime-election-ffc3fb192364238b60977839d14e3154d4a98313e30d46694a12af54b6874975.params": {
+    "cid": "Qmbt2SWWAmMcYoY3DAiRDXA8fAuqdqRLWucJMSxYmzBCmN",
+    "digest": "151ae0ae183fc141e8c2bebc28e5cc10",
+    "sector_size": 268435456
+  },
+  "v20-proof-of-spacetime-election-ffc3fb192364238b60977839d14e3154d4a98313e30d46694a12af54b6874975.vk": {
+    "cid": "QmUxvPu4xdVmjMFihUKoYyEdXBqxsXkvmxRweU7KouWHji",
+    "digest": "95eb89588e9d1832aca044c3a13178af",
+    "sector_size": 268435456
+  },
+  "v20-stacked-proof-of-replication-117839dacd1ef31e5968a6fd13bcd6fa86638d85c40c9241a1d07c2a954eb89b.params": {
+    "cid": "QmQZe8eLo2xXbhSDxtyYZNqEjqjdcWGdADywECRvNEZQdX",
+    "digest": "fcd50e2e08a8560a6bb3418e883567ed",
+    "sector_size": 268435456
+  },
+  "v20-stacked-proof-of-replication-117839dacd1ef31e5968a6fd13bcd6fa86638d85c40c9241a1d07c2a954eb89b.vk": {
+    "cid": "Qme1hn6QT1covfoUFGDZkqoE1pMTax9FNW3nWWmTNqFe7y",
+    "digest": "872e244d86499fd659082e3bcf3f13e7",
+    "sector_size": 268435456
+  },
+  "v20-stacked-proof-of-replication-b46f3a1051afbb67f70aae7082da95def62eee943662f3e1bf69837fb08aaae4.params": {
+    "cid": "QmSfrPDC9jwY4MKrjzhCqDBBAG44wSDM8oE5NuDwWSh2xN",
+    "digest": "0a338b941c5f17946340de5fc95cab30",
+    "sector_size": 34359738368
+  },
+  "v20-stacked-proof-of-replication-b46f3a1051afbb67f70aae7082da95def62eee943662f3e1bf69837fb08aaae4.vk": {
+    "cid": "QmTDGynCmnbaZNBP3Bv3F3duC3ecKRubCKeMUiQQZYbGpF",
+    "digest": "c752e070a6b7aa8b79aa661a6b600b55",
+    "sector_size": 34359738368
+  },
+  "v20-stacked-proof-of-replication-e71093863cadc71de61f38311ee45816633973bbf34849316b147f8d2e66f199.params": {
+    "cid": "QmXjSSnMUnc7EjQBYtTHhvLU3kXJTbUyhVhJRSTRehh186",
+    "digest": "efa407fd09202dffd15799a8518e73d3",
+    "sector_size": 1024
+  },
+  "v20-stacked-proof-of-replication-e71093863cadc71de61f38311ee45816633973bbf34849316b147f8d2e66f199.vk": {
+    "cid": "QmYHW3zhQouDP4okFbXSsRMcZ8bokKGvzxqbv7ZrunPMiG",
+    "digest": "b2f09a0ccb62da28c890d5b881c8dcd2",
+    "sector_size": 1024
+  },
+  "v20-stacked-proof-of-replication-e99a585174b6a45b254ba4780d72c89ad808c305c6d11711009ade4f39dba8e9.params": {
+    "cid": "QmUhyfNeLb32LfSkjsUwTFYLXQGMj6JQ8daff4DdVMt79q",
+    "digest": "b53c1916a63839ec345aa2224e9198b7",
+    "sector_size": 1073741824
+  },
+  "v20-stacked-proof-of-replication-e99a585174b6a45b254ba4780d72c89ad808c305c6d11711009ade4f39dba8e9.vk": {
+    "cid": "QmWReGfbuoozNErbskmFvqV4q36BY6F2WWb4cVFc3zoYkA",
+    "digest": "20d58a3fae7343481f8298a2dd493dd7",
+    "sector_size": 1073741824
+  },
+  "v20-stacked-proof-of-replication-f571ee2386f4c65a68e802747f2d78691006fc81a67971c4d9641403fffece16.params": {
+    "cid": "QmSAHu14Pe8iav6BYCt9XkpHJ73XM7tcpY4d9JK9BST9HU",
+    "digest": "7698426202c7e07b26ef056d31485b3a",
+    "sector_size": 16777216
+  },
+  "v20-stacked-proof-of-replication-f571ee2386f4c65a68e802747f2d78691006fc81a67971c4d9641403fffece16.vk": {
+    "cid": "QmaKtFLShnhMGVn7P9UsHjkgqtqRFSwCStqqykBN7u8dax",
+    "digest": "834408e5c3fce6ec5d1bf64e64cee94e",
+    "sector_size": 16777216
+  }
+}
+