13 Commits

Author SHA1 Message Date
Colin Walters
df5cbc9be9 core: Support building with OpenSSL for checksums
Add an OpenSSL backend to the checksum input stream, which is where we do a lot
of checksumming (object commit, static deltas).

The raw OpenSSL performance is
[approximately double](https://gist.github.com/cgwalters/169349fd1c06fd4fb4d3a7ce33303222) on
my laptop; not only does OpenSSL have e.g. hand-tuned x86_64 assembly, the
current implementation uses the
[Intel SHA extensions](https://en.wikipedia.org/wiki/Intel_SHA_extensions).

Another reason to do this is I was idly thinking about adding
[Curve25519](https://en.wikipedia.org/wiki/Curve25519) signatures (like e.g.
Alpine does) instead of/in addition to GPG.  The rationale for that is
that GPG is pretty heavyweight, both in code footprint and the simple
fact that EC keys are way smaller.

I didn't benchmark ostree with this; we have bigger performance problems
really like the fact we just malloc way too much.  But, it's a step
in the right direction I think in combination with the libcurl work
where we're linking to openssl anyways.

Closes: #738
Approved by: jlebon
2017-03-20 18:32:40 +00:00
Colin Walters
515f832067 ci: Hard error on all -fsanitize=undefined warnings
I saw in a recent test log a ton of spam
```
libglnx/glnx-dirfd.c:253:3: runtime error: null pointer passed as argument 1, which is declared to never be null
```
which actually turned out to be libglnx getting reverted.  But
let's be sure now we actually bomb out quickly on UBSAN warnings
in general.

Closes: #693
Approved by: jlebon
2017-02-21 16:59:19 +00:00
Colin Walters
361aa449fb libcurl backend
For rpm-ostree, we already link to libcurl indirectly via librepo, and
only having one HTTP library in process makes sense.

Further, libcurl is (I think) more popular in the embedded space.  It
also supports HTTP/2.0 today, which is a *very* nice to have for OSTree.

This seems to be working fairly well for me in my local testing, but it's
obviously brand new nontrivial code, so it's going to need some soak time.

The ugliest part of this is having to vendor in the soup-url code. With
Oxidation we could follow the path of Firefox and use the
[Servo URL parser](https://github.com/servo/rust-url).  Having to redo
cookie parsing also sucked, and that would also be a good oxidation target.

But that's for the future.

Closes: #641
Approved by: jlebon
2017-02-09 16:37:45 +00:00
Colin Walters
d894f609db oxidation: Add implementation of bupsplit in Rust
This is an initial drop of "oxidation", or adding implementation
of components in Rust.  The bupsplit code is a good target - no
dependencies, just computation.

Translation into Rust had a few twists -

 - The C code relies a lot on overflowing unsigned ints, and
   also on the C promotion rules for e.g. `uint8_t -> int32_t`
 - There were some odd loops that I introduced bugs in while
   translating...in particular, the function always returns `len`,
   but I mistakenly translated to `len+1`, resulting in an OOB
   read on the C side, which was hard to debug.

On the plus side, an off-by-one array indexing in the Rust code paniced nicely.

In practice, we'll need a lot more build infrastructure to make this work, such
as using `cargo vendor` when producing build artifacts for example. Also, Cargo
is yet another thing we need to cache.

Where do we go with this? Well, I think we should merge this, it's not a lot of
code. We can just have it be an alternative CI target. Should we do a lot more
right now? Probably not immediately, but I find the medium/long term prospects
pretty exciting!

Closes: #656
Approved by: jlebon
2017-02-03 14:29:00 +00:00
Colin Walters
5aefe17ee9 ci: Combine UBSAN and ASAN by default
I only recently realized this was possible.  While we're still seeing
leaks in the CI environment for some reason, adding ASAN gives us
use-after-free detection etc., which is obviously still very useful
even if we're not doing leak checking.

Closes: #622
Approved by: jlebon
2016-12-09 18:05:53 +00:00
Colin Walters
67ce5ec917 ci: Drop sudo installed tests
This conflicts with the ASAN work...and in general, I think I'd like
to make a new format for tests that require root, and have them be
defined to be in mutable containers or VMs.

Our coverage loss from this isn't much because some of these tests
already required `CAP_SYS_ADMIN` which we didn't have in Docker
anyways.

While we have the patient open, parallelize the regular installed
tests.

Closes: #622
Approved by: jlebon
2016-12-09 18:05:53 +00:00
Colin Walters
daf01b27d4 ci: Make all ci tests gating for Homu
See the rhci docs.

Closes: #617
Approved by: jlebon
2016-12-07 15:42:14 +00:00
Jonathan Lebon
4b7ab5167c .redhat-ci.yml: no longer install libubsan & clang
Since they're now part of the auto-built image.

Closes: #572
Approved by: cgwalters
2016-11-10 18:10:58 +00:00
Jonathan Lebon
39e7293e3f .redhat-ci.yml: use new build key
This allows us to more concisely separate building from testing, which
in turn gives us a nicer inheritance pattern in our case.

See also: https://github.com/jlebon/redhat-ci/issues/11

Closes: #569
Approved by: cgwalters
2016-11-09 17:35:40 +00:00
Colin Walters
a15dc7f191 ci: Use -fsanitize=undefined by default
It's fast enough to use for CI testing by default, and it
can catch a lot of bad things.

Closes: #552
Approved by: jlebon
2016-10-28 17:47:50 +00:00
Jonathan Lebon
cde7928496 .redhat-ci.yml: add clang
Clang has better detection for unused vars when using auto cleanup
functions. We should eventually just fold this back into the first
testsuite. But let's just turn it on for now, at least until it's
satisfied with the whole codebase.

Closes: #549
Approved by: cgwalters
2016-10-27 18:12:05 +00:00
Jonathan Lebon
64568bc706 .redhat-ci.yml: use projectatomic/ostree-tester
Same Dockerfile, but automated to rebuild on pushes.

Closes: #536
Approved by: giuseppe
2016-10-20 08:35:13 +00:00
Jonathan Lebon
e62defa1a4 add .redhat-ci.yml and .redhat-ci.Dockerfile
Add a YAML file for the new Red Hat CI framework.

Rather than re-installing all the build deps everytime, which takes
time, I added a Dockerfile that we can wire up to the Docker Hub. For
now it lives at `jlebon/ostree-tester:rhci`, but we can move it under
the `projectatomic` org (or a new `ostree` org).

Closes: #535
Approved by: cgwalters
2016-10-19 20:35:23 +00:00