676 Commits

Author SHA1 Message Date
Joonas Koivunen
40c7b414e3 add: serde Display/FromStr wrapper
not sure why this isn't included in serde?
2020-07-09 14:14:23 +03:00
bors[bot]
e37c9ab8d5
Merge #231
231: Streaming multiparts: pubsub r=ljedrz a=koivunej

This was seen originally but at least it seemed some months ago that there wasn't a way to process multiparts streaming. Luckily the older js-ipfs-http-client puts the message in query. Later versions related to #228 use multipart bodies, but the topic is still a query argument.

Depends on #230.

 * Adds test for the older query part message parsing
 * Uses `&'static str` instead of `&'static [u8]` (fixes error messages)

The latest version should leave us failing only the conformance tests related to:

* pinning
* `?timeout=<not sure what unit>`
  * refs
  * cat
  * get
  * block/get
  * dag/get
* wantlist

This is:

```
  146 passing (3m)
  21 pending
  8 failing
```


Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
2020-07-09 08:27:56 +00:00
bors[bot]
c3b92c7640
Merge #230
230: Streaming multiparts at block/put and dag/put r=ljedrz a=koivunej

Supercedes #225 to cover `dag/put`. I'll close the earlier and keep this as a draft while the temp git dependency is in place.

This allows accepting multipart bodies without Content-Length header by using mpart-async. Uploads are limited to 1MB.

As minor fixes this includes checking `dag/put?input-enc=raw` as we don't support any other input-enc at the moment (rust-ipld could read dag-json, this was implemented during phase1 already).

As more minor fixes this includes transformation of LengthRequired rejection to "Missing header: content-length" instead of "UNHANDLED REJECTION".

Fix is needed for #228.

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
2020-07-09 08:04:44 +00:00
Joonas Koivunen
7498e7ebda doc: remove unneeded allows, comments 2020-07-08 22:10:55 +03:00
Joonas Koivunen
ce939c3036 fix: wanted error for missing pubsub message 2020-07-08 21:34:13 +03:00
Joonas Koivunen
1c9b31cf76 fix: get pubsub/pub message from multipart body 2020-07-08 21:09:54 +03:00
Joonas Koivunen
6ddd7774ea fix: bad use of [u8] where str would do 2020-07-08 20:33:13 +03:00
Joonas Koivunen
7637874087 refactor: reorder pub fn before inner impl 2020-07-08 19:42:47 +03:00
Joonas Koivunen
6f2bf990cb fix: check input-enc in dag/put
this is minor change but good for interop. the conformance tests only
exercise the input-enc=raw as js-ipfs-http-client does most of the heavy
work. not checking input-enc could end up someone trying to put a json
dag document expecting it to be transformed to cbor, which we currently
do not support.
2020-07-08 19:26:01 +03:00
Joonas Koivunen
fef741c750 fix: allow multipart bodies without content-len in dag/put 2020-07-08 19:25:43 +03:00
Joonas Koivunen
4101383855 refactor: move try_only_named.. to support 2020-07-08 19:24:17 +03:00
Joonas Koivunen
087776bda1 refactor: try_only_named_multipart out of put_block 2020-07-08 19:17:58 +03:00
Joonas Koivunen
06ba40e8f2 fix: add descriptive error for LengthRequired 2020-07-08 16:03:10 +03:00
Joonas Koivunen
0a9ee274ca temp: upgrade to mpart-async with fix 2020-07-08 13:52:01 +03:00
Joonas Koivunen
76486f8ecb fix: rewrite block/put to support streaming multiparts
streaming multiparts are something we dont yet support, which is keeping
us back in an earlier version of js-ipfs-http-client for conformance
tests.
2020-07-08 13:52:01 +03:00
bors[bot]
2ac9b3562a
Merge #227
227: Assorted improvements, continued r=koivunej a=ljedrz

~~Builds on https://github.com/rs-ipfs/rust-ipfs/pull/226, only the last 5 commits are new.~~
The commit messages describe the specific changes, of which the following are the most interesting:
- don't clone the `Subscription`'s result unless it's ready
- enum-ify the `Subscription` object

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-07-08 09:21:58 +00:00
ljedrz
9e03b9aac6 refactor: enum-ify the Subscription object
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-08 11:18:57 +02:00
ljedrz
29e8a855bc refactor: remove a barely used getter
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-08 10:45:26 +02:00
ljedrz
92f2d45722 perf: don't clone the Subscriber's result unless it's Ready
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-08 10:45:26 +02:00
ljedrz
fb1742a38e refactor: minify empty structs
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-08 10:45:26 +02:00
ljedrz
cc5a266d2b refactor: use the Mutex from futures instead of async_std
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-08 10:45:26 +02:00
ljedrz
67749cafda fix: fix a currently-broken rocksdb test
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-08 10:45:26 +02:00
bors[bot]
c78aca5881
Merge #226
226: Assorted improvements to Repo and SubscriptionRegistry r=koivunej a=ljedrz

Individual commits describe the specific changes; notable ones:
- remove the `Arc` and its clones where not needed
- reduce allocations around subscription wakers
- use `futures:🔒:Mutex` instead of `std::sync::Mutex` where applicable
- add a `wipe` method to the stores (useful for tests/benches)

cc https://github.com/rs-ipfs/rust-ipfs/issues/174

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-07-08 08:29:04 +00:00
ljedrz
26d99119d7 feat: add a wipe method to RocksDataStore
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 15:31:59 +02:00
ljedrz
70e58675f5 fix: use an async Mutex inside the RocksDataStore
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 15:31:41 +02:00
ljedrz
4290a287f2 fix: add a missing match branch in ResolveColumnFamily::resolve
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 15:31:10 +02:00
ljedrz
4c3cd0a909 perf: remove the Arc in RocksDataStore
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 15:30:32 +02:00
ljedrz
437584ac19 feat: add a wipe method to BlockStore and DataStore
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 15:12:13 +02:00
ljedrz
e5c838a2b5 perf: remove the Arc in MemBlockStore and MemDataStore
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 15:11:50 +02:00
ljedrz
c430458fa0 perf: remove the Arc in FsBlockStore and an unneeded stray clone
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 15:08:24 +02:00
ljedrz
8cb4967fdb fix: use an async Mutex inside the FsBlockStore
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 15:08:24 +02:00
ljedrz
970bbbf7cb perf: reduce allocations around subscription wakers
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 14:17:51 +02:00
ljedrz
f76999018b refactor: prefer the Default impl when no params are given
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 14:04:53 +02:00
ljedrz
c0e999aaa0 refactor: rename SubscriptionRegistry::cancelled to ::shutting_down
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 13:25:47 +02:00
ljedrz
636a1e188a fix: use an async Mutex over the SubscriptionRegistry
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-07 11:25:06 +02:00
bors[bot]
f757458474
Merge #223
223: Arc-ify the Ipfs struct r=koivunej a=ljedrz

This allows us to pass the central `Ipfs` object around when needed, improving what we can do within its child objects. In addition:
- `impl Default for Bitswap` (now possible)
- replace `Arc<Repo>` with a reference to `Ipfs` in `IpldDag` and `Ipns`, which are now created on-the-fly when needed
- poll for `BitswapEvent`s using `NetworkBehaviourEventProcess`
- un`Arc` the `Repo`

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-07-06 10:01:53 +00:00
ljedrz
11cb434d68 refactor: create Dag and Ipns on the fly
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-06 11:32:45 +02:00
ljedrz
d5f6f5e300 chore: add a FIXME from the former PR
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-06 11:32:45 +02:00
ljedrz
872913ce3f refactor: remove Bitswap.io_events, use NetworkBehaviourEventProcess for BitswapEvents
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-06 11:32:45 +02:00
ljedrz
8b618aab7c refactor: wrap Ipfs in an Arc
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-06 11:32:45 +02:00
bors[bot]
65f884c849
Merge #221
221: remove Strategy, improve event handling r=koivunej a=ljedrz

The scope of changes is far greater than I've anticipated, but bear with me.

Key, planned changes:
- remove `bitswap::strategy`
- move `bitswap::Stats` to `ledger`
- introduce `BitswapEvents` emitted by `Bitswap::poll`
- move `Message` into `Ledger`

Drive-by changes forced by current design:
- wrap `Stats` in an `Arc`, make individual stats `AtomicU64`
- add a `Bitswap.queued_blocks` collection populated from `IpfsFuture::poll`
- make some child `bitswap` objects `pub` or `pub(crate)`
- update `Stats` from `IpfsFuture::poll` (good causality-wise, bad encapsulation-wise)
- introduce a `Bitswap.io_events` sender and its `IpfsFuture.bitswap_events` counterpart receiver

Observations:
- `Repo` is something we want to have access to both in `bitswap` (in order to find `want`ed `Block`s) and in `Ipfs` (to handle the higher-level API requests)
- `Ledger`s for the different `PeerId`s contain `Stats` that are updated at `IpfsFuture` level
- `BitswapEvents` don't currently result in anything of interest, due to `Behaviour` not containing the `Repo`

Key takeaway: we need to do some re-designing in order for key objects to have better access to one another.

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-07-06 09:10:43 +00:00
bors[bot]
933b52d97f
Merge #220
220: Initial file adder r=ljedrz a=koivunej

Creates multiblock balanced trees. Is quite slow and uses quite a lot of heap.

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
Co-authored-by: Joonas Koivunen <joonas.koivunen@gmail.com>
2020-07-03 09:33:28 +00:00
ljedrz
33935bc109 refactor: better ledger stat handling
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-03 11:25:19 +02:00
Joonas Koivunen
2553de32de refactor: go back to moving builder 2020-07-03 12:20:58 +03:00
Joonas Koivunen
2d31efafb2 chore: changelog 2020-07-03 12:20:58 +03:00
Joonas Koivunen
77657806d7 doc: enhance, document lore, suggestions
Co-authored-by: ljedrz <ljedrz@users.noreply.github.com>
2020-07-03 12:20:58 +03:00
Joonas Koivunen
13e5036126 refactor: use Default::default() struct init 2020-07-03 12:20:58 +03:00
Joonas Koivunen
28065973ad refactor: name the 4-tuple as Link 2020-07-03 12:20:58 +03:00
Joonas Koivunen
a409095509 doc: suggestions from code review
Co-authored-by: ljedrz <ljedrz@users.noreply.github.com>
2020-07-03 12:20:58 +03:00
Joonas Koivunen
c47bfc1933 fix: debug intermediate levels as well 2020-07-03 12:20:58 +03:00