Commit Graph

1103 Commits

Author SHA1 Message Date
Joonas Koivunen
9e695e02cb fix: go back to std::sync::Mutex
Turns out #174 was wrong, as the alternative required sprinkling
block_on which made the whole thing worse. we shouldn't risk a deadlock
with the mutex as we are not (can not) holding it across awaits.

unwrap as we are not expecting subscriptions to panic and poison the
mutex.
2020-08-11 13:03:45 +03:00
bors[bot]
1b1aab339b
Merge #295
295: Merge and adapt the libipld dependency r=koivunej a=ljedrz

Our [libipld](https://crates.io/crates/libipld) dependency has been out of date with _its_ dependencies for a while now and since we don't need to maintain a shadow fork of the entire repo in order to be able to keep up, we might as well merge the bits we need until the upstream repo(s) are updated and we can depend on them directly again.

All the copied (with minor adjustments) code resides in the `ipld` module where the source repo is credited.

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-08-10 12:31:35 +00:00
ljedrz
b1bc672559 chore: merge and adapt the libipld dependency
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-08-10 13:15:11 +02:00
bors[bot]
16ae43011b
Merge #286
286: Ipfs object tweaks r=ljedrz a=ljedrz

Removes `SwarmTypes`, drops the generic from `IpfsOptions`, `RepoOptions` and `SwarmOptions` and simplifies the creation of `Ipfs`/`UnitializedIpfs` via more pattern-matching `move`s.

Rationale: since we already need to specify the type in order to create `IpfsOptions`, we might as well do that directly for `Ipfs` instead, without dragging the generic through all the options that don't even use it (they just carry `PhantomData`).

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-08-10 10:24:42 +00:00
ljedrz
f8674a0504 refactor: remove the generic from IpfsOptions, RepoOptions and SwarmOptions
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-08-10 11:49:43 +02:00
ljedrz
b95cb29650 refactor: simplify Ipfs object creation
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-08-10 11:46:18 +02:00
ljedrz
8e42d203c0 refactor: remove SwarmTypes
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-08-10 11:46:18 +02:00
bors[bot]
e00198a4d6
Merge #284
284: unixfs: feat tree building r=koivunej a=koivunej

Includes a `BufferingTreeBuilder` which kind of follows the js-ipfs implementation: a *complete* tree is built from path segments and later it's converted into a dag-pb by an "iterator" which walks the tree in post order, converting (rendering) the most nested elements first. Communication back to the "parent" is done using a `HashMap<u64, Vec<Option<(String, Cid, usize)>>>` which relies on initial ordering provided by `BufferingTreeBuilder` by using BTreeMap's on each level.

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
Co-authored-by: Joonas Koivunen <joonas.koivunen@gmail.com>
2020-08-10 09:18:51 +00:00
Joonas Koivunen
aaabc37562 doc: suggestions and notes from code review
Co-authored-by: ljedrz <ljedrz@users.noreply.github.com>
2020-08-10 12:04:03 +03:00
Joonas Koivunen
276e4b7441 bench: filter symlinks earlier 2020-08-10 11:45:58 +03:00
Joonas Koivunen
14c27fd1b8 bench: move buffer and fix path growing outside 2020-08-10 11:44:10 +03:00
Joonas Koivunen
665c0c4f85 doc: comment touch ups 2020-08-10 09:23:11 +03:00
Joonas Koivunen
bfc1bf167c refactor: rename put_file => put_link 2020-08-10 09:22:58 +03:00
Joonas Koivunen
9df95ad1e8 refactor: remove duplicated ingest-tar example, it remains as bench 2020-08-10 09:13:44 +03:00
Joonas Koivunen
7446810982 doc: minor docs and fmt 2020-08-09 22:24:36 +03:00
Joonas Koivunen
71b02cf370 chore: cargo keeps changing the index file, clippy
not sure what is the index file hash change.
2020-08-09 22:24:36 +03:00
Joonas Koivunen
5484f3ed8f fix: make the benchmark more generic 2020-08-09 22:24:36 +03:00
Joonas Koivunen
28392da979 chore: cleanup, doc 2020-08-09 22:24:36 +03:00
Joonas Koivunen
3d48caa079 refactor: custom pb types to own module 2020-08-09 22:24:36 +03:00
Joonas Koivunen
3876717e5d doc: cleanup, rename, docs 2020-08-09 22:24:36 +03:00
Joonas Koivunen
56f1970d7b fix: use knowledge of longest path
this saves a few resizes but no effect in bench.
2020-08-09 22:24:36 +03:00
Joonas Koivunen
19a62e23dc refactor: use BTreeMap in dir_builder
this allows us to get the unique and sorted links right away, allowing
to use Vec<Option<(...)>> later.
2020-08-09 22:24:36 +03:00
Joonas Koivunen
eb58b8b15e test: output "actual" on test failure 2020-08-09 22:24:36 +03:00
Joonas Koivunen
bcab2bd3b1 refactor: move Visited under iter.rs 2020-08-09 22:24:36 +03:00
Joonas Koivunen
dc4ff9cee8 add: tar ingestion example, bench
copypaste sadly
2020-08-09 22:24:20 +03:00
Caio
09f44d18f2 Use mutable ref instead of taking ownership
Walker was being moved for each iteration and this was being a problem
for proper `Entry` matching.

Removes `Item` indirection and `continue_walk(self, ...)` is now
`inspect(&mut self, ...)`.
2020-08-09 13:52:14 -03:00
Caio
a032c195d8 Sort elements (walk.rs)
1. Private declarations come after public declarations
2. Implementation blocks are next to its declarations
3. Elements are sorted in ascending order
2020-08-09 13:42:09 -03:00
Joonas Koivunen
25e56fd51d add: missing fmt::Debug impl 2020-08-08 11:52:31 +03:00
Joonas Koivunen
9a8267d5b5 chore: update stack limit notes 2020-08-08 11:52:15 +03:00
Joonas Koivunen
07c41c0573 fix: tests after going from Infallible => Rejection
the change was made to allow boxing the v0::routes return value.
2020-08-07 21:15:48 +03:00
Joonas Koivunen
a3f27e5b46 fix: move all path matching to v0, box handlers
This patch changes the warp filter boxing strategy so, that:

 1. paths are nested with a "prefix" matcher, if possible
 2. the actual handler for the matched path is boxed
 3. the non implemented but recognized paths are unified

This strategy might be a good compromise between optimizations and
compilation time but such measurements are yet to be done.

Previous boxing strategy ended up being quite unfortunate. Upon adding
the full `/add` implementation, the test cases using GNU binutils linker
or ld 2.33 (ubuntu) started failing failing with a thread overflowing
its stack.

Most of the stack frames were warp filters for matching the path. This
happened because the combine! macro used in http/src/v0.rs boxed
(mostly) each filter, meaning the compiler could never see through any
of the and requiring enough many frames at runtime to cause a stack
overflow.

An alternative of opt-level = 1 turned out to be easy and simple and was
explored in #293 while this solution was being searched for. The
downside is increased compilation time.
2020-08-07 21:07:11 +03:00
Joonas Koivunen
8c10a0ad0d fix: remove calling ulimit
while this may have succeeded on github actions it probably isn't wise
to run it every time at least automatically.
2020-08-07 21:05:38 +03:00
Joonas Koivunen
e38c1ff04c refactor: try splitting the huge func 2020-08-07 15:06:55 +03:00
Joonas Koivunen
fdc5f8aa84 refactor: make PostOrderIterator 'static
kind of forgot the String and Vec<u8> while trying to impl the Iterator
trait to return non-static values.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
41566e4759 refactor(postorderiter): remove redundant visibilities 2020-08-07 15:06:55 +03:00
Joonas Koivunen
684ecdc9d3 doc: note lld-9 ulimit -s bisection 2020-08-07 15:06:55 +03:00
Joonas Koivunen
f24f9bda42 refactor: reorder items, fix doc 2020-08-07 15:06:55 +03:00
Joonas Koivunen
761124884f feat: block size limit 2020-08-07 15:06:55 +03:00
Joonas Koivunen
b134593708 doc: minor too short sentence 2020-08-07 15:06:55 +03:00
Joonas Koivunen
5d8b124367 doc: suggestions from code review
Thanks to ljedrz for the many many fixes.

Co-authored-by: ljedrz <ljedrz@users.noreply.github.com>
2020-08-07 15:06:55 +03:00
Joonas Koivunen
7d7915a9da doc: remove mention of wrap in => wrap with 2020-08-07 15:06:55 +03:00
Joonas Koivunen
7bbff65bdd fix: DirBuilder::is_empty 2020-08-07 15:06:55 +03:00
Joonas Koivunen
39583a77f1 refactor: remove extra initialization 2020-08-07 15:06:55 +03:00
Joonas Koivunen
1c71e3c08d refactor: tests repeated the verify with block dump fn 2020-08-07 15:06:55 +03:00
Joonas Koivunen
f3b92d48d6 fix: replace todo! with Err(PathEndsInSlash)
this can leave empty nodes in the tree.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
9e948f943f perf: render Cids to bytes without allocating a Vec
not sure how much sense does this make, given that the byte writing fns
might be less inlined. would be nice to have some benchmarks.

this also adds a test to verify directory cid when using cidv1 even
though it's quite unclear to me if those are allowed in unixfs dirs.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
3e499e161e perf: custom dag-pb serialization
use the BTreeMap<String, Leaf> directly to output the PBLink alike
bytes without going through the mapping into Vec. Does not support Cid
v1 though yet.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
653d079cc9 doc: add note on HAMT sharding, adjust 2020-08-07 15:06:55 +03:00
Joonas Koivunen
a90826b48d perf: avoid clearing block_buffer
this should lead to less zero extending but it's questionable if this is
more performant.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
dc2c0217b9 feat: /add?progress=true
adds the progress reporting, which is a progress notification to the
response stream on every block write *in addition to* a notification
*after* the file has been read (order with the `Added` message doesn't
seem to have been specified).
2020-08-07 15:06:55 +03:00