895 Commits

Author SHA1 Message Date
bors[bot]
9e491a68f3
Merge #259
259: Multihash-based repo r=koivunej a=ljedrz

We currently have an ad-hoc `Cid`-upgrading mechanism in place for the legacy `v0` version that is complicated and error-prone; when working on the `SubscriptionRegistry` I had to disable some of those upgrades, but some of them were still applied in the `Repo` (and necessary in order for the conformance tests to pass), making it hard to work on further functionalities like content discovery.

The solution I came up with is to introduce a `RepoCid` wrapper in order to make the keys in `Repo` objects operate on the associated `Multihash` instead of the whole `Cid`. Changing them to plain `Multihash`es wouldn't work, as we are still expected to return `Cid`s for the purposes of e.g. `Ipfs::local_refs` and I'm pretty sure some of the conformance tests still expect the full `Cid` to be available.

The only drawback is some extra `clone()`ing, but the improvement in code clarity and the unblocking of further changes is well worth it. This change also simplifies one of our conformance test patches (local refs).

~~I'm marking this PR as a draft in hopes that I can still come up with some solution to the extra cloning.~~ This is hard; I'm not able to do this at the moment, but this shouldn't be a blocker.

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-07-28 07:30:09 +00:00
bors[bot]
1367a30a8c
Merge #265
265: Use async macro for tests r=ljedrz a=c410-f3r

Fixes #248 

Co-authored-by: Caio <c410.f3r@gmail.com>
2020-07-27 18:09:01 +00:00
Caio
1107b29114 Rustfmt 2020-07-27 13:16:54 -03:00
Caio
e42b88c6a6 Use async macro for tests 2020-07-27 13:00:40 -03:00
ljedrz
6188425c59 chore: remove unused Ledger tests
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-27 11:14:32 +02:00
ljedrz
b1aa5a2a7d refactor: move bitswap stats directly under the Bitswap object
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-27 11:11:56 +02:00
ljedrz
af3e539444 fix: adjust the existing local_refs conformance test patch
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-23 14:28:12 +02:00
ljedrz
7201b1b82f fix: adjust test_inner_local
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-23 13:58:49 +02:00
ljedrz
fc5d63319e perf: make Block equality depend only on its Multihash
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-23 12:52:23 +02:00
ljedrz
e9e788fce2 feat: disregard Cid upgrades; only care about the Multihash in the Repo
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-23 12:52:23 +02:00
bors[bot]
6bee0b57e6
Merge #256
256: some groundwork for peer/content discovery r=koivunej a=ljedrz

Some groundwork for peer and content discovery using `libp2p::kad`.

~~Very much a WIP with plenty of prototyping and wild `println!`s.~~

Cc https://github.com/rs-ipfs/rust-ipfs/issues/10

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-07-22 14:23:42 +00:00
ljedrz
5b02b62b85 fix: move bootstrap and add_peer methods to Node
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-22 15:59:58 +02:00
ljedrz
b41490fe31 fix: classic Default::default() fun plus remove explicit kad protocol names
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-22 15:47:25 +02:00
ljedrz
c14f9e60cb chore: document new DHT-related Ipfs methods
Co-authored-by: Joonas Koivunen <joonas.koivunen@gmail.com>
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-22 15:40:13 +02:00
ljedrz
962f389e46 fix: move get_closest_peers from Ipfs to Node
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-22 15:37:37 +02:00
ljedrz
ff1765ba10 fix: remove unused params in Node::new and IpfsOptions::inmemory_with_generated_keys
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-22 15:35:25 +02:00
ljedrz
499744b20c chore: remove a stray commented-out block
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-22 15:23:12 +02:00
ljedrz
e2297fd53a fix: update a few tests
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-22 13:31:46 +02:00
ljedrz
3bb0a65e04 feat: add and test peer discovery
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-22 12:00:56 +02:00
ljedrz
8e957c2423 feat: initial Kademlia extensions
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-22 10:15:40 +02:00
bors[bot]
dc5e18f7e6
Merge #257
257: ci: stop building pushes to master r=aphelionz a=koivunej

bors should only fast forward or reset the master to HEAD of staging at the end of each run, so if there is a credit system on GH we are just wasting those credits on each merged PR.

if nothing else, this should be more energy efficient :) if bors fast forwarded something the commit should have the status badge.

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
2020-07-21 18:33:06 +00:00
Joonas Koivunen
8971c9d05e ci: stop building pushes to master
bors should only fast forward or reset the master to HEAD of staging
at the end of each run, so if there is a credit system on GH we are just
wasting those credits on each merged PR.
2020-07-21 21:26:22 +03:00
bors[bot]
7a74c74c35
Merge #255
255: update libp2p again r=koivunej a=ljedrz

Needed for our `substrate-ipfs` project, as `substrate` just updated it.

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-07-21 14:50:47 +00:00
bors[bot]
5bbaf0633d
Merge #254
254: Fix bitswap ready block wakeup r=koivunej a=koivunej

The previous version relied on block_on for the block reads, lets see if this still timeouts on `exchange_block`.

This changes the `queued_blocks` to be `UnboundedSender<(PeerId, Block)>` instead of `Arc<Mutex<Vec<(PeerId, Block)>>>`. An alternative would had been to store the waker but we concluded the channel would be more appropriate.

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
Co-authored-by: Joonas Koivunen <joonas.koivunen@gmail.com>
2020-07-21 14:24:46 +00:00
Joonas Koivunen
578cebc820
doc: apply suggestions from code review
Co-authored-by: ljedrz <ljedrz@users.noreply.github.com>
2020-07-21 16:39:27 +03:00
ljedrz
cc26f40d14 feat: update libp2p again
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-21 14:50:06 +02:00
Joonas Koivunen
b6b14e93ba doc: add fixmes to bitswap 2020-07-21 15:44:21 +03:00
Joonas Koivunen
cb6e57a8a6 fix: use unbounded ch instead of Arc<Mutex<Vec<_>>>
Arc<Mutex<Vec<_>>> depended on the fact that the blocks were read
blockingly. This way we will get wakeups when the reads complete.
2020-07-21 15:44:21 +03:00
Joonas Koivunen
7dbe83fb70 fix: remove task::block_on from Repo::get_block_now 2020-07-21 15:44:21 +03:00
bors[bot]
91979c22d6
Merge #253
253: update some dependencies r=koivunej a=ljedrz

We have been behind in terms of `cid` and `multihash` for a while now, but with the recent update of `libp2p` (as long as we want to follow it closely), those need to be updated too. The biggest obstacle to updating our dependencies is currently the pinned revision of `libipld` and while it'll take more work to adapt our code to its recent upstream version, that revision can easily be forked with updated dependencies, enabling other updates.

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-07-21 09:45:39 +00:00
ljedrz
856c41f8dc fix: async-ify and simplify bounded_retry
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-21 11:26:42 +02:00
ljedrz
88c7219790 feat: update some dependencies
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-21 10:31:55 +02:00
bors[bot]
81dd57e06f
Merge #250
250: Migrating http symlink creation to conformance/setup.sh + CI Update r=koivunej a=aphelionz

This PR removes the symlink from source control, and adds some lines to validate and create the symlink to `conformance/setup.sh`. It also updates CI to build the project _before_ trying to create the symlink, as per the warning in `setup.sh`

The reasoning behind this is for GitHub Pages + Jekyll, which will break upon encountering a symlink that doesn't resolve properly in the repo.

Co-authored-by: Mark Robert Henderson <henderson.mark@gmail.com>
2020-07-20 19:30:46 +00:00
Mark Robert Henderson
fba5399fc1 Removing the http symlink
Update setup.sh

Update ci.yml
2020-07-20 11:09:19 -04:00
bors[bot]
8c82178bac
Merge #252
252: introduce a local get_block_now functionality r=koivunej a=ljedrz

This allows us not to call the `async` counterpart (that spawns a potentially long-lasting `SubscriptionFuture`) when being asked for a `Cid` by other peers.

In addition, expand the `wantlist_cancellation` test a little bit - these changes don't affect it after all, but the extra checks are useful.

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-07-20 14:11:31 +00:00
ljedrz
4b60684f73 refactor: use the Node object in the wantlist cancellation test
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-20 15:56:09 +02:00
ljedrz
9e013101a3 fix: increase some wantlist test timeouts
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-20 15:30:09 +02:00
ljedrz
6a646d73d6 feat: test wantlist/subscriptions coop
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-20 15:30:09 +02:00
ljedrz
d494e3dfde perf: introduce get_block_now to speed up the wantlist
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-07-20 15:30:09 +02:00
bors[bot]
1078e85e57
Merge #251
251: Fix hanging interface-ipfs-core tests r=ljedrz a=koivunej

This should add the diagnostic, but not hang the builds. Lets see how lucky we get. This is following the test win/mac failures on #249 and #250.

In this PR:
- why-is-node-running triggered 1min after interface-ipfs-core tests complete (so visible only when stuck)
- ipfsd-ctl patched to always await for subprocesses (this seems healthy either way)
- tests ignored:
    - miscellanious.stop for surely causing the hang
    - add from invalid url for likely causing a hang (DNSWRAP)

Need to investigate further what is causing the hangs with the stop -- it seemed like a deadlock. Not sure what happens with the DNSWRAP; perhaps the invalid url is good enough, but our rejection causes it to be "forgotten" and as such it seems to linger as it's completion is never checked once we fail the url add.

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
2020-07-20 13:13:03 +00:00
Joonas Koivunen
1b870c2d7f fix: skip the stop tests as they are the final hanging ones 2020-07-20 15:24:32 +03:00
Joonas Koivunen
153da866f0 fix: ipfsd-ctl should await for subprocess 2020-07-20 15:00:39 +03:00
Joonas Koivunen
d20a84b164 add: at least some shutdown logging 2020-07-20 13:36:09 +03:00
Joonas Koivunen
96752cc5ac fix: enable debug output for npm tests (ipfsd-ctl) 2020-07-20 13:11:57 +03:00
Joonas Koivunen
9b9f49d75a fix: ignore possibly hanging test case 2020-07-20 12:40:45 +03:00
Joonas Koivunen
0b6e3dddad add: why-is-node-running diagnostics for win/mac builds 2020-07-20 11:55:57 +03:00
bors[bot]
62131cf445
Merge #249
249: Fix full link blocks r=koivunej a=koivunej

Rather a big rewrite on the balanced tree builder. The failing test case showed that compression happened too eagerly. This meant adjusting the `<` to `<=` in the two `work <= self.branching_factor` conditions, and making the "compression" a loop.

After pondering this a while, I wrote the loop internals to be "inplace mutating" instead of "drain" and "insert", which moved complexity a bit around. Also refactored the N argument fn out.

Fixes #242.

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
2020-07-19 15:22:07 +00:00
Joonas Koivunen
fedf176e3d fix: full link block generation
this included making the compression to loop on the lower level, making
the algorithm in place modifying instead of draining (and thus moving
yet to be processed links). added comments and asserts as well.

Co-authored-by: ljedrz <ljedrz@users.noreply.github.com>
2020-07-17 18:00:35 +03:00
Mark Robert Henderson
7bbfbd7354 Set theme jekyll-theme-cayman 2020-07-17 10:57:04 -04:00
Joonas Koivunen
27068d1e4e fix: Link was missing fmt::Debug 2020-07-17 15:38:54 +03:00