Commit Graph

1092 Commits

Author SHA1 Message Date
Joonas Koivunen
665c0c4f85 doc: comment touch ups 2020-08-10 09:23:11 +03:00
Joonas Koivunen
bfc1bf167c refactor: rename put_file => put_link 2020-08-10 09:22:58 +03:00
Joonas Koivunen
9df95ad1e8 refactor: remove duplicated ingest-tar example, it remains as bench 2020-08-10 09:13:44 +03:00
Joonas Koivunen
7446810982 doc: minor docs and fmt 2020-08-09 22:24:36 +03:00
Joonas Koivunen
71b02cf370 chore: cargo keeps changing the index file, clippy
not sure what is the index file hash change.
2020-08-09 22:24:36 +03:00
Joonas Koivunen
5484f3ed8f fix: make the benchmark more generic 2020-08-09 22:24:36 +03:00
Joonas Koivunen
28392da979 chore: cleanup, doc 2020-08-09 22:24:36 +03:00
Joonas Koivunen
3d48caa079 refactor: custom pb types to own module 2020-08-09 22:24:36 +03:00
Joonas Koivunen
3876717e5d doc: cleanup, rename, docs 2020-08-09 22:24:36 +03:00
Joonas Koivunen
56f1970d7b fix: use knowledge of longest path
this saves a few resizes but no effect in bench.
2020-08-09 22:24:36 +03:00
Joonas Koivunen
19a62e23dc refactor: use BTreeMap in dir_builder
this allows us to get the unique and sorted links right away, allowing
to use Vec<Option<(...)>> later.
2020-08-09 22:24:36 +03:00
Joonas Koivunen
eb58b8b15e test: output "actual" on test failure 2020-08-09 22:24:36 +03:00
Joonas Koivunen
bcab2bd3b1 refactor: move Visited under iter.rs 2020-08-09 22:24:36 +03:00
Joonas Koivunen
dc4ff9cee8 add: tar ingestion example, bench
copypaste sadly
2020-08-09 22:24:20 +03:00
Caio
09f44d18f2 Use mutable ref instead of taking ownership
Walker was being moved for each iteration and this was being a problem
for proper `Entry` matching.

Removes `Item` indirection and `continue_walk(self, ...)` is now
`inspect(&mut self, ...)`.
2020-08-09 13:52:14 -03:00
Caio
a032c195d8 Sort elements (walk.rs)
1. Private declarations come after public declarations
2. Implementation blocks are next to its declarations
3. Elements are sorted in ascending order
2020-08-09 13:42:09 -03:00
Joonas Koivunen
25e56fd51d add: missing fmt::Debug impl 2020-08-08 11:52:31 +03:00
Joonas Koivunen
9a8267d5b5 chore: update stack limit notes 2020-08-08 11:52:15 +03:00
Joonas Koivunen
07c41c0573 fix: tests after going from Infallible => Rejection
the change was made to allow boxing the v0::routes return value.
2020-08-07 21:15:48 +03:00
Joonas Koivunen
a3f27e5b46 fix: move all path matching to v0, box handlers
This patch changes the warp filter boxing strategy so, that:

 1. paths are nested with a "prefix" matcher, if possible
 2. the actual handler for the matched path is boxed
 3. the non implemented but recognized paths are unified

This strategy might be a good compromise between optimizations and
compilation time but such measurements are yet to be done.

Previous boxing strategy ended up being quite unfortunate. Upon adding
the full `/add` implementation, the test cases using GNU binutils linker
or ld 2.33 (ubuntu) started failing failing with a thread overflowing
its stack.

Most of the stack frames were warp filters for matching the path. This
happened because the combine! macro used in http/src/v0.rs boxed
(mostly) each filter, meaning the compiler could never see through any
of the and requiring enough many frames at runtime to cause a stack
overflow.

An alternative of opt-level = 1 turned out to be easy and simple and was
explored in #293 while this solution was being searched for. The
downside is increased compilation time.
2020-08-07 21:07:11 +03:00
Joonas Koivunen
8c10a0ad0d fix: remove calling ulimit
while this may have succeeded on github actions it probably isn't wise
to run it every time at least automatically.
2020-08-07 21:05:38 +03:00
Joonas Koivunen
e38c1ff04c refactor: try splitting the huge func 2020-08-07 15:06:55 +03:00
Joonas Koivunen
fdc5f8aa84 refactor: make PostOrderIterator 'static
kind of forgot the String and Vec<u8> while trying to impl the Iterator
trait to return non-static values.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
41566e4759 refactor(postorderiter): remove redundant visibilities 2020-08-07 15:06:55 +03:00
Joonas Koivunen
684ecdc9d3 doc: note lld-9 ulimit -s bisection 2020-08-07 15:06:55 +03:00
Joonas Koivunen
f24f9bda42 refactor: reorder items, fix doc 2020-08-07 15:06:55 +03:00
Joonas Koivunen
761124884f feat: block size limit 2020-08-07 15:06:55 +03:00
Joonas Koivunen
b134593708 doc: minor too short sentence 2020-08-07 15:06:55 +03:00
Joonas Koivunen
5d8b124367 doc: suggestions from code review
Thanks to ljedrz for the many many fixes.

Co-authored-by: ljedrz <ljedrz@users.noreply.github.com>
2020-08-07 15:06:55 +03:00
Joonas Koivunen
7d7915a9da doc: remove mention of wrap in => wrap with 2020-08-07 15:06:55 +03:00
Joonas Koivunen
7bbff65bdd fix: DirBuilder::is_empty 2020-08-07 15:06:55 +03:00
Joonas Koivunen
39583a77f1 refactor: remove extra initialization 2020-08-07 15:06:55 +03:00
Joonas Koivunen
1c71e3c08d refactor: tests repeated the verify with block dump fn 2020-08-07 15:06:55 +03:00
Joonas Koivunen
f3b92d48d6 fix: replace todo! with Err(PathEndsInSlash)
this can leave empty nodes in the tree.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
9e948f943f perf: render Cids to bytes without allocating a Vec
not sure how much sense does this make, given that the byte writing fns
might be less inlined. would be nice to have some benchmarks.

this also adds a test to verify directory cid when using cidv1 even
though it's quite unclear to me if those are allowed in unixfs dirs.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
3e499e161e perf: custom dag-pb serialization
use the BTreeMap<String, Leaf> directly to output the PBLink alike
bytes without going through the mapping into Vec. Does not support Cid
v1 though yet.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
653d079cc9 doc: add note on HAMT sharding, adjust 2020-08-07 15:06:55 +03:00
Joonas Koivunen
a90826b48d perf: avoid clearing block_buffer
this should lead to less zero extending but it's questionable if this is
more performant.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
dc2c0217b9 feat: /add?progress=true
adds the progress reporting, which is a progress notification to the
response stream on every block write *in addition to* a notification
*after* the file has been read (order with the `Added` message doesn't
seem to have been specified).
2020-08-07 15:06:55 +03:00
Joonas Koivunen
5534fef2b8 refactor: wrap_in_directory => wrap_with_directory
just aligning the terminology and trying not to invent new terms.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
14112c3dca feat: /add?wrap_with_directory=true
previous implementation included "defaulting the name for the wrapper
dir" which was extra and not expected by the interface tests.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
65acedae1d fix: normalize prefix slash away to enable more tests 2020-08-07 15:06:55 +03:00
Joonas Koivunen
4077de1a0f fix: report adderrors so that js-ipfs-http-client errors
the mechanism is to just push a new line of MessageResponse. it's a
hack, but at least the js-ipfs-http-client fails.
2020-08-07 15:06:55 +03:00
Joonas Koivunen
8319e8bd97 chore: remove extra clone (clippy) 2020-08-07 15:06:55 +03:00
Joonas Koivunen
dd8d9d9bfe conformance: enable more directory tests 2020-08-07 15:06:55 +03:00
Joonas Koivunen
57737b49c6 refactor: detupleify into (Owned)?TreeNode 2020-08-07 15:06:55 +03:00
Joonas Koivunen
a5b52b3682 doc: add initial documentation 2020-08-07 15:06:55 +03:00
Joonas Koivunen
6ba4195500 refactor: split dir/builder.rs up
* split off dir_builder.rs
* split off PostOrderIterator
* split bufferedtreebuilder off
2020-08-07 15:06:55 +03:00
Joonas Koivunen
548dbe2873 chore: appease clippy
single while let Some(_) = ... changed to is_some().
2020-08-07 15:06:55 +03:00
Joonas Koivunen
bb9555b14e doc: explain anyhow dependency 2020-08-07 15:06:55 +03:00