rust-ipfs/conformance

Conformance testing for Rust IPFS

This directory contains the scripts used to run interface conformance testing for Rust IPFS. It uses js-ipfsd-ctl, ipfs-http-client, and interface-js-ipfs-core. This code used to live at https://github.com/rs-ipfs/ipfs-rust-conformance, but was integrated into https://github.com/rs-ipfs/rust-ipfs, as working with multiple repositories got a bit tedious.

Usage

Use setup.sh to do npm install and patch any of the dependencies:

$ cargo build -p ipfs-http
$ cd conformance
$ ./setup.sh

By default, there is a http symlink to ../target/debug/ipfs-http. You can change this to the release binary by modifying the symlink or use a custom binary via the environment variable IPFS_RUST_EXEC. The default rust.sh wrapper will record all actions taken by the tests into a single large log file. It's not recommended to trust it to keep all log lines especially for tests with multiple processes.

$ IPFS_RUST_EXEC="$(pwd)/rust.sh" npm test
$ cat /tmp/rust.log

Obtaining logs for tests with multiple processes

Patch the rust.sh as follows:

-./http "$@" 2>&1 | tee -a /tmp/rust.log || retval=$?
+./http "$@" 2>&1 | tee -a /tmp/rust.log.$$ || retval=$?

Now the /tmp/rust.log will contain only the "pointers" to other log files, for example:

>>>> new execution 24132 with args: daemon
<<<< exiting 24132 with 0

This means there is now a log file /tmp/rust.log.24132 for that invocation.

Additionally, it helps to clear out the logs often with rm -f /tmp/rust.log* and only run selected tests using IPFS_RUST_EXEC="$(pwd)/rust.sh" npm test -- --grep 'should do foo'. If it's impossible to limit the number of tests to one with --grep, you can comment out the undesired tests in test/index.js.

Patch management

We are currently pinned to interface-ipfs-core@0.137.0 and the fixes we have upstreamed are kept under patches/.

To create a new patch:

1. Fork https://github.com/ipfs/js-ipfs
2. Clone locally
3. Checkout a new branch based on the tag for the interface-ipfs-core version we are currently depending on (see package.json)
4. Apply all of our patches from patches/ with git apply $ipfs_rust_conformance/patches/*
5. Fix any new tests
6. When done, cherry-pick your commits to a new branch based off the latest main js-ipfs branch
7. Submit PR
8. Squash your work, refer to your PR in the message
9. Remove existing patches/
10. Recreate the patches/ with git format-patch -o $ipfs-rust-conformance/patches $the_tag_you_based_your_work

$variable definitions:

• $ipfs_rust_conformance points to your checkout of this repository
• $the_tag_you_based_your_work is the tag where you started working on js-ipfs

Previously this has been done by first working on a patch and then backporting it, which might be easier. Luckily the tests are low traffic. If you need help wrestling with git, don't hesitate to ask for guidance; this is simpler than its step by step explanation looks like.

Troubleshooting hangs

If the tests hang because of some deadlock on the ipfs-http side the tests will print the final summary from mocha and then seem to wait forever. This is true at least for the hangs seen so far, but you could run into a different issues. Note that in addition to test timeouts the http client also has a timeout, which can keep the npm test alive for less than ten seconds after the summary has been printed.

What has worked previously is:

1. keep disabling some tests until you find the one which causes the hang
2. rerun tests using the rust.sh wrapper which gives you logs at /tmp/rust.log
3. continue debugging over at ipfs-http

To disable other test means to comment them or to use IPFS_RUST_EXEC=... npm test -- --grep '<suspected test>'. You should get to a single running test which hangs. If the test does a lot, you can refactor that into a smaller one which will only cause the hang and nothing else.

The use of rust.sh wrapper is critical as it'll give you the logging output. If the test has multiple running instances you might be better off separating logs into files per invocation of rust.sh by appending .$$ to the log files name, which will expand to the process id of the shell running the script.

To "continue debugging" is trickier. What has worked previously is:

1. attaching a gdb to a running process in case of livelocks
2. adding debugging in case everything stalls

Livelocks (I might be using a wrong term here) happen when a task (running on either tokio or async-std) never returns from std::future::Future::poll but stays busy all of the time. You can spot these by seeing a core being utilitized by ipfs-http constantly. These are easy to track down by:

1. attach gdb -p $process_id path/to/your/ipfs-http
2. find the interesting thread with info threads or by looking at threads' stack traces

In the "everything stalled" case, a std::future::Future::poll has completed with Poll::Pending without waking up the task for a new poll. These mistakes are quite simple to make. Good indications of such issues:

• custom poll methods without the [std::task::Context] parameter: these methods will never be able to schedule wake-ups
• polling some nested "pollable" thing and returning Poll::Pending following the nested poll returning Poll::Ready(_)
  • if the inner "pollable" didn't return Poll::Pending, it means it had "more values to bubble up"
  • see this hastly written issue https://github.com/libp2p/rust-libp2p/issues/1516 and the linked commit(s)
• custom [std::future::Future] which cannot return errors on drop like with the early SubscriptionFuture (see https://github.com/ipfs-rust/rust-ipfs/pull/130)

License

Same as the repository.

Trademarks

The Rust logo and wordmark are trademarks owned and protected by the Mozilla Foundation. The Rust and Cargo logos (bitmap and vector) are owned by Mozilla and distributed under the terms of the Creative Commons Attribution license (CC-BY).