1306 Commits

Author SHA1 Message Date
ljedrz
70b1c735e7 chore: rename test_dht_with_go to test_go_interop
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-09-07 17:14:41 +02:00
ljedrz
5628024208 test: introduce tests/common and move go interop there
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-09-07 17:14:41 +02:00
ljedrz
9341d014f4 test: add go-ipfs DHT interop tests to the CI
Signed-off-by: ljedrz <ljedrz@gmail.com>
2020-09-07 17:14:41 +02:00
bors[bot]
7c4e652a59
Merge #354
354: add DHT get and put r=koivunej a=ljedrz

While I'm looking at the curious Windows crashes with `/dns` and `/resolve`, we can expand territories elsewhere: I've had these functionalities implemented in a branch building one the recently merged PRs; there's no counterpart HTTP endpoints yet, as the conformance tests use functionalities that we're not implementing yet; these features are less interesting for the HTTP API anyway.

Co-authored-by: ljedrz <ljedrz@gmail.com>
2020-09-07 14:47:53 +00:00
bors[bot]
4c719d49c2
Merge #359
359: This PR uses libp2p dns, so that /dns4 and /dns6 multiaddrs can be dialed. r=koivunej a=rklaehn

Not quite sure how to best test this without having a test that uses the dns of the CI server. Any ideas?

Co-authored-by: Rüdiger Klaehn <rklaehn@protonmail.com>
2020-09-07 14:20:07 +00:00
bors[bot]
2365748147
Merge #360
360: test: common fs and mem pinstore tests r=aphelionz a=koivunej

Continues the long chain of pin PRs from #358. Refactors the previously custom tempdir hack and system polluting repo tests into a macro based which can be re-generated for each datastore implementation. Of course this revealed issues on both fs and mem pinstores but they were more about me not understanding the rules before three implementations :) Lets fix later if we find out more inconsistencies.

The macro approach is the only one I can see, looks pretty bad and if there are errors those will be even uglier.

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
2020-09-04 15:47:06 +00:00
Joonas Koivunen
2b82d88702 refactor: rename tests_for_ds_impl, add doc 2020-09-04 18:30:19 +03:00
Joonas Koivunen
0672a3d57a chore: clippy 2020-09-04 18:30:19 +03:00
Joonas Koivunen
6fc846e1e3 fix: align mem and fs pinstore impl and tests 2020-09-04 18:30:19 +03:00
Joonas Koivunen
e684f2a30e fix: allow cidv0 and cidv1 indirect storage 2020-09-04 18:30:19 +03:00
Joonas Koivunen
bb5b756bfc refactor: remove the old repo tests 2020-09-04 18:30:19 +03:00
Joonas Koivunen
c38673ff3e refactor: the pinstore "interface tests"
my first attempt with macro generated tests: lots of quite confusing to
debug compiler errors but the final outcome looks ok.
2020-09-04 18:30:19 +03:00
bors[bot]
77bfbd3441
Merge #358
358: feat: fs pinstore r=aphelionz a=koivunej

Adds filesystem based pin store. See adec0a8a75a6512e9ace4a953240a399962da25e for TL;DR.

Builds upon #357. Only last N commits are new after including 159a1c5ca7fcb478b029eb031c6f2012c7506822. Even windows seems ok. base32lower is used to encode the cid names.

TODO:
- [x] at least the tests written for ipfs::repo::mem but will post another PR

The differences to other impls are not explained as I am not sure if I am up to date with the other implementations.



Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
2020-09-04 15:29:30 +00:00
Joonas Koivunen
eef03b5138 fix: botched copypaste 2020-09-04 17:32:37 +03:00
Joonas Koivunen
fbce11d2d4 test: add invalid cid path silently ignored tests 2020-09-04 17:32:37 +03:00
Joonas Koivunen
001bcb46ba chore: use ordering, typos and minor docs 2020-09-04 17:32:37 +03:00
Joonas Koivunen
04e96ec42e fix: move to semaphore owned permits for locking
this allows using the owned permit and more importantly moving it to the
threadpool thread, which is not necessarily bound to the task lifetime.
there is just one permit per datastore at the moment, but making this
per cid for example shouldn't be too difficult.

having a global lock will be needed later on, not sure how to implement
it though.
2020-09-04 17:32:37 +03:00
Joonas Koivunen
be71e302da doc: explain why the DataStore column ops are not implemented 2020-09-04 17:32:37 +03:00
Joonas Koivunen
db710b962a refactor: split paths module 2020-09-04 17:32:37 +03:00
Joonas Koivunen
7a4ada12cd test: add tests for path2cid2path fns 2020-09-04 17:32:37 +03:00
Joonas Koivunen
c8cc20bc06 refactor: reorder the shard and path fns, add docs 2020-09-04 17:32:37 +03:00
Joonas Koivunen
e2c098c4c2 refactor: split fsblockstore to separate file 2020-09-04 17:32:37 +03:00
Joonas Koivunen
64e3e49e01 refactor: split pinstore 2020-09-04 17:32:37 +03:00
Joonas Koivunen
ee7357aca6 chore: clippy 2020-09-04 17:32:37 +03:00
Joonas Koivunen
112d46dc5a refactor(http): change some printlns to tracing 2020-09-04 17:32:37 +03:00
Joonas Koivunen
ed235c44f6 refactor: remove use of tracing:: in front of macros 2020-09-04 17:32:37 +03:00
Joonas Koivunen
2e1f9b7568 add: tracing for pinning operations 2020-09-04 17:32:37 +03:00
Joonas Koivunen
630e0823ad refactor(refs): tracing adjustments 2020-09-04 17:32:37 +03:00
Joonas Koivunen
bc3fcd7334 refactor: minor tracing adjustments 2020-09-04 17:32:37 +03:00
Joonas Koivunen
d781c0c452 feat: initial fs backed datastore, persistent pinning
this is implemented using a blockstore alike sharding implementation.
for recursive pins, a json Vec<String> is stored and used mostly
linearly (eh, something to improve). for direct pins, empty files are
used. the pin files have different file extensions.

lets see if this works in windows as well.
2020-09-04 17:32:37 +03:00
Joonas Koivunen
266e95efa5 refactor: filestem to cid into free fn 2020-09-04 17:32:37 +03:00
bors[bot]
b90ec1d0ac
Merge #357
357: refactor:  move recursive pinning details below r=aphelionz a=koivunej

Builds on #355; does not select the strategy for fs pinstore.

This does change the locking on ipfs::repo::mem but not so that it would be caught in the conformance tests. Probably the change is for the better.

Relevant 4 commits: 588e020236..159a1c5ca7

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
2020-09-04 14:09:31 +00:00
Joonas Koivunen
827dc1b35b refactor: move PinDocument and friends to repo::mem 2020-09-04 16:52:27 +03:00
Joonas Koivunen
ab4d9f1f52 refactor: get rid of accidentially added indirect_to
did not mean to commit this
2020-09-04 16:52:27 +03:00
Joonas Koivunen
5588ef7348 doc: add missing fsync FIXME 2020-09-04 16:52:27 +03:00
Joonas Koivunen
08f6473c7b refactor: move pin handling deeper from ipfs::Ipfs
this was a fault in the first inmem run. now the APIs are looking a
bit more suitable for a fs implementation: instead of walking over the
refs in ipfs::Ipfs level a stream is passed over to repo level for the
operation.
2020-09-04 16:52:27 +03:00
bors[bot]
e8902b4fd0
Merge #355
355: fix: fs blockstore r=aphelionz a=koivunej

This fixes the racing writers issue which the implementation has had for a long time; multiple writers would come in and all would write over the same file, multiple times, allowing readers to observe partial writes in the meantime. This covers most of the changes.

In addition ipfs-http now uses the file block store. There was some transient macosx failures.

### Checklist (can be deleted from PR description once items are checked)

- [x] **New** code is “linted” i.e. code formatting via rustfmt and language idioms via clippy
- [x] There are no extraneous changes like formatting, line reordering, etc. Keep the patch sizes small!
- [x] There are functional and/or unit tests written, and they are passing
- [x] There is suitable documentation. In our case, this means:
    - [ ] Each command has a usage example and API specification
    - [ ] Rustdoc tests are passing on all code-level comments
    - [ ] Differences between Rust’s IPFS implementation and the Go or JS implementations are explained

Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
Co-authored-by: Joonas Koivunen <joonas.koivunen@gmail.com>
2020-09-04 12:59:50 +00:00
Rüdiger Klaehn
08c911b970 Add support for dialing dns4, dns6 and dns multiaddrs 2020-09-04 10:30:01 +02:00
Joonas Koivunen
588e020236 doc: add comments, remove forgotten allow(clippy::type_cmplx) 2020-09-03 14:17:03 +03:00
Joonas Koivunen
247384aabb add: panic on wipe 2020-09-03 14:08:52 +03:00
Joonas Koivunen
c8e0de4dfd add: tracing all over fs blockstore 2020-09-03 14:08:44 +03:00
Joonas Koivunen
befd945cbd chore: clippy 2020-09-03 03:55:50 +03:00
Joonas Koivunen
bb71e032eb test: add case which uses remove and list 2020-09-03 03:14:35 +03:00
Joonas Koivunen
53a883537c refactor: simplify write_completion 2020-09-03 03:14:35 +03:00
Joonas Koivunen
4b893ce1f2 fix: synchronize on remove as well 2020-09-03 03:14:35 +03:00
Joonas Koivunen
53def20cb6 fix: shuffle things around, think about drop safety
quite similar issues as with subscriptionfutures, how to get the
synchronization key removed from the map. trying with similar but
perhaps simpler drop guard.
2020-09-03 03:14:30 +03:00
Joonas Koivunen
59758ca353
doc: suggestions from code review 2020-09-03 00:36:01 +03:00
Joonas Koivunen
af7b58424e fix: use repocid (partial fix)
using repocid wrapper is more correct than not having one. there is
still a chance in this to fail, since we are writing the cidv1 to disk
but the repocid will block on multihash. Hmm.. This might be ok
after all.
2020-09-03 00:22:31 +03:00
Joonas Koivunen
5604f6010a doc: comment on writes, allow clippy 2020-09-03 00:15:10 +03:00
Joonas Koivunen
f9a2634280 feat: use fsblockstore in ipfs-http 2020-09-03 00:06:09 +03:00