IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Unfortunately, introspection uses dlopen(), which doesn't quite
work when the DSO is compiled with ASAN but the outer executable
isn't.
Trying to inject LD_PRELOAD=libasan means the outer executable has to
be leak free...which, yeah, I'm not going to get into running ASAN
today on gjs or pygobject.
So, let's skip those tests - ideally, we still run them in some other
context without the sanitizers. The coverage we have from them is
middling anyways.
Closes: #622
Approved by: jlebon
The "remote cookies" code broke this. While I'm not sure anyone is
actually using ostree-without-http, it isn't too hard to keep the
build time conditional going. Further, this work is preparatory for
libcurl porting.
Closes: #621
Approved by: jlebon
I am trying to track down a warning I see in `test-keyfile-utils`
which turned out to be the installed case only, but let's inject
this here too.
(The GLib default is broken, but it's hard to fix upstream without
breaking the world)
Closes: #610
Approved by: jlebon
If we're using LD_LIBRARY_PATH for some locally-built library, we want
to search those after OSTree's own libraries.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #606
Approved by: cgwalters
This is necessary for "make distcheck" on Travis-CI.
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters
Some deployments may want to gate access to content based on things
like OAuth. In this model, the client system would normally compute a
token and pass it to the server via an API.
We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.
Closes: #574
Approved by: giuseppe
This fixes a "make dist" tarball produced on a minimal system and run on a
non-minimal system. Automake knows that files that are only conditionally
included in dist_whatever_WHATEVER are to be distributed, but it does not
do the same for files that are only conditionally included in EXTRA_DIST,
which is how glib-tap.mk's various variables like dist_test_scripts work.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #557
Approved by: cgwalters
These tests use unshare and mount to prepare a fake initrd/early boot
directory structure so we can then test ostree-prepare-root.
Things that are tested:
* Running in an initrd environment
* Running without initrd
* /var and /sysroot being mounted correctly
* /usr being mounted read-only
Things not tested (yet):
* Running as init - this could be accomplished by unsharing the pid
namespace too.
* mounting/unmounting `/proc` if `/proc/cmdline` isn't available
* Persistent overlayfs for `/usr`
* Probably other things
The tests are basic but can be extended in the future as we do more work
on `ostree-prepare-root`.
These tests must be run as root as they require the ability to `mount`
and to `unshare` the mount namespace. Perhaps in the future we can use
user namespaces for this test once they are more widely available.
Closes: #403
Approved by: cgwalters
The tests suite was failing locally as it was using the installed
version of rofiles-fuse, instead of the built one. Create the needed
symlinks in tests/ as we are already doing for the "ostree" binary.
ostree-prepare-root and ostree-remount added for completeness.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #395
Approved by: cgwalters
Systems like pulp may want to keep retrying in a loop if the server
throws a (hopefully transient) 500, and we need test coverage of
handling these errors versus our existing 404 and 206 coverage.
Closes: #383
Approved by: mbarnes
libtoolize creates a version of libtool for the right architecture
in $(top_builddir), which is guaranteed to be present, and is
guaranteed to match what we are compiling (even during
cross-compilation).
Packaging systems sometimes separate /usr/bin/libtool, which is
specific to one architecture, from the libtool development files
such as libtoolize and ltmain.sh, which are architecture-independent.
For example, in Debian, libtool_*_all.deb contains the files necessary
to libtoolize a package and is depended on by the dh-autoreconf package,
but libtool-bin_*_amd64.deb (or whatever architecture) contains
/usr/bin/libtool and is not normally necessary to depend on.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #367
Approved by: cgwalters
It's unfortunate that in automake one has to explicitly include the globa
`$(AM_CFLAGS)` if one sets `CFLAGS`, and similarly for other variables.
I'm trying to use `-fsanitize=address`, and not including it was
causing linker failures.
We also weren't inheriting the global warnings etc., so I had to fix a
decl-after-statement.
Closes: #351
Approved by: jlebon
The filename of the real ostree executable could be either .libs/ostree
or .libs/lt-ostree.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #337
Approved by: cgwalters
It uses dlsym(). There's no point in being extra-portable here
because OSTree only targets Linux anyway.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #336
Approved by: cgwalters
The recent memleak fixes motivated me to look at the bitrotted code to
run invocations of `ostree` in the test suite underneath valgrind.
There are a few things here. First, update suppressions file from
libhif, since I recently worked on it.
When running *uninstalled* as we now support, we need
`libtool --mode=execute` in the mix so it expands out to
the uninstalled binary and we don't valgrind the intermediate shell.
However, it's harder than that because we chdir into a tmpdir,
which defeats the libtool logic. AFAICS, the only fix for this
is to determine the realbin path before we chdir, and then unfortunately
we need to change every use of `ostree` to `${OSTREE}` =(
Then this immediately breaks for me on RHEL7 because my ancient
copy of `valgrind-3.10.0-16.el7.x86_64` is unaware of syscall 306, i.e.
`syncfs`.
But let's do this first before I dive into that.
Closes: #292
Approved by: krnowak
We have had in the past issues with running `ostree_repo_pull()`
multiple times in the same process, embarassingly enough. Nothing in
the current test suite covers this, so let's start.
Closes: #322
Approved by: jlebon
This was already supported by the commit modifier API, just needed to
expose it. This will also be used to test the libarchive API in a future
test.
Closes: #275
Approved by: cgwalters
We had a policy of cleaning up all files in `$repo/tmp` older
than one day, but we should really clean up previous bootid staging
directories too, as they can potentially take up a lot of disk space.
https://bugzilla.gnome.org/show_bug.cgi?id=760531Closes: #170
Approved by: jlebon
We keep forgetting to update `apidoc/ostree-sections.txt`, so let's
start enforcing it. Of course it turns out we had some bugs here
like symbols marked as public but never implemented, etc. Those
are fixed in the prior commits.
Closes: #263
Approved by: giuseppe
ostree-grub-generator can be used to customize
the generated grub.cfg file. Compile time
decision ostree-grub-generator vs grub2-mkconfig
can be overwritten with the OSTREE_GRUB2_EXEC
envvar - useful for auto tests and OS installers.
Why this alternative approach:
1) The current approach is less flexible than using a
custom 'ostree-grub-generator' script. Each system can
adjust this script for its needs, instead of using the
hardcoded values from ostree-bootloader-grub2.c.
2) Too much overhead on embedded to generate grub.cfg
via /etc/grub.d/ configuration files. It is still
possible to do so, even with this patch applied.
No need to install grub2 package on a target device.
3) The grub2-mkconfig code path has other issues:
https://bugzilla.gnome.org/show_bug.cgi?id=761180
Task: https://bugzilla.gnome.org/show_bug.cgi?id=762220Closes: #228
Approved by: cgwalters
If installed-tests are disabled, it would normally be a static
(convenience) library, which isn't something we can LD_PRELOAD.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #232
Approved by: cgwalters
Previously, the build-time tests would only pass if the g-i bindings to
OSTree were already installed, with a reasonably similar version.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #232
Approved by: cgwalters
Otherwise $(installed_testdir) is empty so we try to put content in
`/`, which I noticed when trying to build an RPM (it works works fine
`sudo make install`).
This will allow daemons like rpm-ostree to detect if there are any new
deployments efficiently, in combination with using inotify. If there
are any changes, rpm-ostree wants publish them on DBus.
While we're here, add some changes to start doing unit C testing of
the sysroot API.
I want to be able to easily test the C API on actual data in an OSTree
repo. The shell `libtest.sh` has code to generate it. Bridge the two
worlds by introducing a little `libostreetest` library which has a C
API which spawns a shell that runs things in `libtest.sh`.
Yes, this is about as beautiful as it sounds, which is to say, it's
not. But it works!
Note while we were here, I realized we were actually now creating
*two* tmpdirs per test in `make check` because the tap driver was
already doing that. Unify it so we know the C code can rely on it.
OSTree's code for testing predates the `glib-tap.mk` making its
way into GLib. Let's switch to it, as it provides a number
of advantages.
By far the biggest advantage is that `make check` can start to run
most of the tests *in addition* to having them work installed.
This commit keeps the installed tests working, but `make check` turns
out to be really broken because...our TAP usage has bitrotted to say
the least. Fix that all up.
Do some hacks so that the tests work uninstalled as well - in
particular, `glib-tap.mk` and the bits encoded into
`g_test_build_filename()` assume *recursive* Automake (blah). Work
around that by creating a symlink when installed to loop back.
If the average object size is greater than 4GiB, let's assume we're
dealing with opposite endianness. I'm fairly confident no one is
going to be shipping peta- or exa- byte size ostree deltas, period.
Past the gigabyte scale you really want bittorrent or something.
This is a more flexible version of the previous
ostree_repo_write_archive_to_mtree() which took a file reference.
This has an extensible options structure, and in particular
now supports `ignore_unsupported_content`.
I plan to use this for importing Docker images which contain device
nodes. (There's no reason for container images to have those, so
we'll just ignore them).
Also here, just like the export variant, the caller is responsible for
setting up libarchive.
At the moment I'm looking at using rpm-ostree to manage RPM inputs
which can then be converted into Docker images. It's most convenient
if we can stream directly out of libostree rather than doing a
checkout + tar combination.
There are also backup/debugging etc. reasons to implement `export` as
well.
While it's not strictly tied to OSTree, let's move
https://github.com/cgwalters/rofiles-fuse in here because:
- It's *very* useful in concert with OSTree
- It's tiny
- We can reuse OSTree's test, documentation, etc. infrastructure
One thing to consider also is that at some point we could experiment
with writing a FUSE filesystem for OSTree. This could internalize a
better equivalent of `--link-checkout-speedup`, but on the other hand,
the cost of walking filesystem trees for these types of operations is
really quite small.
But if we did decide to do more FUSE things in OSTree, this is a step
towards that too.
I was working on a different test, and ended up being very confused at
the behavior where removing the last deployment didn't remove the last
`ostree/X/X/X` ref pointing to its commit.
There's no reason to special case the last undeployment AFAIK, and the
existing code handles this.
test-lzma builds a copy of the compressor and decompressor directly, so
the compiler needs access to the LZMA headers and the linker needs to
link the program with liblzma.
This test uses gpgme directly to verify the signatures, so it needs to
find the gpgme headers and link with gpgme to ensure the linker can
resolve the symbols.
Having undefined (but in practice rarely changing) ordering for
`readdir()` ended up screwing us over for bootloader config
generation; see https://bugzilla.redhat.com/show_bug.cgi?id=1226520
Let's make things significantly more likely to fail more quickly in
the future if similar bugs are introduced. We accomplish this by
introducing a little `LD_PRELOAD` library that randomizes the results
of `readdir()`.
Having undefined ordering (but in practice rarely changing)
ordering for `readdir()` ended up screwing us over with respect
to bootloader config file read ordering.
Let's make things significantly more likely to fail more quickly in
the future if similar bugs are introduced. We accomplish this by
introducing a little `LD_PRELOAD` library that randomizes the results
of `readdir()`.
An OSTree user noticed that `ostree fsck` would produce `missing
object` errors in the case of interrupted pulls.
It's possible to do e.g. `ostree pull --subpath=/usr/share/rpm ...`,
which gets you just that portion of the commit. The use case for this
was being able to see what changes would appear in an update before
actually downloading all of it.
(I think this would be better covered by static deltas, but those
aren't final yet, and `--subpath` predates it)
Further, `.commitpartial` is used as a successor to the `transaction`
symlink for more precise knowledge in the case where a pull was
interrupted that we needed to resume scanning.
So it makes sense for `ostree fsck` to be aware of it.
If a system administrator happens to type `ostree admin upgrade`
multiple times, currently that will lead to a potentially corrupted
system.
I originally attempted to do locking *internally* in `libostree`, but
that didn't work out because currently a number of the commands
perform multi-step operations that all need to be serialized. All of
the current code in `ostree admin deploy` is an example.
Therefore, allow callers to perform locking, as most of the higher
level logic is presently implemented there.
At some point, we can revisit having internal locking, but it will be
difficult. A more likely approach would be similar to Java's approach
with concurrency on iterators - a "fail fast" method.
First, git doesn't do this, and whatever Linus thinks is right or
something.
Second specifically to OSTree, it's quite common to not have
intermediate commits. If one wants to reset a ref in order to prune
data after a deployment, the parentage check will fail.
Closes: https://github.com/GNOME/ostree/pull/87
Wrappers a referenced gpgme_verify_result_t so detailed verify results
can be examined independently of executing a verify operation.
_ostree_gpg_verifier_check_signature() now returns this object instead
of a single valid/invalid boolean, but the idea is for OstreeRepo to also
return this object for commit signature verification so it can be utilized
at the CLI layer (and possibly by other programs).
This does an rsync-style prepared delta basically. On my test data,
it shaves ~6MB of uncompressed data. Not a huge amount, but I expect
this to be more useful for things like binaries which embed data, etc.
This creates a archive-z2 repo, pull-locals it to bare-user and then
again back to archive-z2 making sure things fsck along the way.
Then it checks out all repos and makes sure each one reproduces
the same result.
Unfortunately we can't install this as a real test because
it doesn't work in the test-runner because tmpfs doesn't support
user xattrs.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
This just does whatever test-basic.sh does, but on a bare-user
repo.
This works standalone, but unfortunately it breaks in
gnome-desktop-testing-runner as /tmp doesn't support
xattrs, so it is not installed atm.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
fixes a coredump when using a command like:
$ ostree --repo=repo checkout -U --subpath=/usr/lib/passwd \
fedora-atomic/rawhide/x86_64/docker-host usrlib-new
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Some operating systems may come with external tools for subscription
management that drive access to the content. In that case, the origin
file may not be useful (for example, it could refer to an installer
ISO).
This patch will allow OS installers to inject that state, with a
useful error message, directing the system administrator to an
external tool.
See: https://github.com/projectatomic/rpm-ostree/issues/31https://bugzilla.gnome.org/show_bug.cgi?id=737686
Now that we have a summary file, we can use it to allow a simple:
ostree pull --mirror
To download the latest commit on every branch. Also, for a case I'm
dealing with there's only one branch, but I don't want mirror users to
have to hardcode it.
https://bugzilla.gnome.org/show_bug.cgi?id=737807
Recursive over ostree and all subcommands, and check that --help
is supported, properly outputs to standard out, and exits
with a 0 exit status. Check that for commands with subcommands,
they produce the help output to standard error when run with no arguments.
https://bugzilla.gnome.org/show_bug.cgi?id=737194
For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.
In order to make metalink work though, it needs *one* file which can
be checksummed. (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).
This commit adds a new command:
$ ostree summary -u
To regenerate the summary file. Can only be run by one process at a
time.
After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.
https://bugzilla.gnome.org/show_bug.cgi?id=729585
Changes the pull API to allow pulling only a single directory instead
of the whole deployment. This option is utilized by the check-diff
option in rpm-ostree.
Add a new state directory to hold <checksum>.commitpartial files, so
we know that we've only downloaded partial state.
If fetching GPG-signed commits over plain HTTP, a MitM attacker can
fill up the drive of targets by simply returning an enormous stream
for the commit object.
Related to this, an attacker can also cause OSTree to perform large
memory allocations by returning enormous GVariants in the metadata.
This helps close that attack by limiting all metadata objects to 10
MiB, so the initial fetch will be truncated.
But now the attack is only slightly more difficult as the attacker
will have to return a correctly formed commit object, then return a
large stream of < 10 MiB dirmeta/dirtree objects.
https://bugzilla.gnome.org/show_bug.cgi?id=725921
We don't want to allow MITM attackers to intercept upgrade requests
and provide clients with older OS versions vulnerable to security
flaws.
Only "ostree admin upgrade" gets this behavior for now - whether we
want to do it for "ostree admin switch" is another question.
These GPG tests were failing for me on EL7 - it appears to be because
we had only one directory for both private and public keys, and we
were giving that to ostree for verification, which passed them onto
gpgv.
In EL7 beta at least, gpgv now barfs if it finds a private key where
it is just expecting to find public keys.
Fix this by splitting out the public trusted directory from the
private key directory. Except now for signing, we still need the
public key there, so symlink it. Whee!
This has a very basic level of functionality (deltas can be generated,
and applied offline). There is only some stubbed out pull code to
fetch them via HTTP.
But, better to commit this now and improve it from a known starting
point, rather than have it languish in a branch.
This test had some nondeterminism because we chose a random
object to corrupt, but because there were multiple commits, it
was possible that we chose an object that was not being pulled.
Fix this by writing some custom GJS code to find an explicitly random
object that exists in a given ref, an change a random byte offset.
This adds a lot more randomness to the testing too.
The test-admin-deploy-1 was copied into -uboot at some point. But
really they should be testing exactly the same thing, except for
the bootloader backend.
Unify these tests by extracting a common test core.
The official way to add bootloader arguments to the current deployment
is to redeploy with --karg. However, doing so tripped up an
optimization made inside the deployment code to just swap the
bootlinks if we're keeping the same "bootcsum".
Change this optimization to look at the pair of (bootcsum, options).
This large patch moves the core xattr logic down into libgsystem,
which allows the gs_shutil_cp_a() API to copy them. In turn, this
allows us to just use that API instead of rolling our own recursive
copy here.
As noted in the new comment though, one case that we are explicitly
regressing is where the new /etc removes a parent directory that's
needed by a modified file. This seems unlikely for most vendors now,
but let's do that as a separate bug.
https://bugzilla.gnome.org/show_bug.cgi?id=711058
Add a --generate-sizes option to commit to add size information to the
commit metadata. This will be used by higher level code which wants
to determine the total size necessary for downloading.
Adapted from Google protobufs. For several cases, we want to support
e.g. file sizes up to guint64, but paying the cost of 8 bytes for each
number is too high.
This will be used for static deltas and sizes metadata.
This uses gpgv for verification against DATADIR/ostree/pubring.gpg by
default. The keyring can be overridden by specifying OSTREE_GPG_HOME.
Add a unit test for commit signing with gpg key and verifying on pull;
to implement this we ship a test GPG key generated with no password
for Ostree Tester <test@test.com>.
Change all of the existing tests to disable GPG verification.
Add an optional dependency on gpgme to add GPG signatures into the
detached metadata, with the key "ostree.gpgsigs", as an "aay", an
array of signatures (treated as binary data).
The commit command gains a --gpg-sign=<key-id> argument. Also add an
argument --gpg-homedir to set the GPG homedir where we look for
keyrings.
Use a consistent temporary filename to download uri's.
Check for downloaded files before fetching from uri.
Download to hash.part file, then copy/move to hash.done when complete.
Add argument support to setup_fake_remote_repo1 function.
Add test for pull resume.
To implement this, pass --force-range-requests into the trivial-httpd,
which will only serve half of the objects to clients at a time.
https://bugzilla.gnome.org/show_bug.cgi?id=706344
We'll always have "bare" mode for keeping files-as-hardlinks as root.
But "archive" was my second attempt at a format for non-root file
storage, used by the gnome-ostree buildsystem which runs as non-root.
It was really handy to have a "tar" like mode where I can create
tarballs as a user, that contain files owned by root for example.
The "archive" mode stored content files as two pieces in the
filesystem; ".file" contained metadata, and ".filecontent" was the
actual content, uncompressed. The nice thing about this was that to
check out a tree as non-root, you could just hardlink into the repo.
However, archive was fairly bad for serving via HTTP; it required
*two* HTTP requests per content object, greatly magnifing the already
inefficient fetch process. So "archive-z2" was introduced.
To allow gnome-ostree to still check out trees as a user, the
"uncompressed-object-cache" was introduced, and that's how things have
been working for a while.
So we should just be able to kill this code. Specifically note just
how much better the stage_object() function became.
https://bugzilla.gnome.org/show_bug.cgi?id=706057
We may revive this later, but commits in their current form aren't
very useful for humans to read, so it doesn't make sense to have a
tool to show a history of useless stuff.
More interesting things are diffs between commits, object statistics,
etc.
These corruption tests could be a lot better...like randomly try
single bit flips, range flips. Better, content-aware fuzzing. But
this is useful for now.
See https://wiki.gnome.org/OSTree/DeploymentModel2
This is a major rework of the on-disk filesystem layout, and the boot
process. OSTree now explicitly supports upgrading kernels, and these
upgrades are also atomic.
The core concept of the new model is the "deployment list", which is
an ordered list of bootable operating system trees. The deployment
list is reflected in the bootloader configuration; which has a kernel
argument that tells the initramfs (dracut) which operating system root
to use.
Invidiual notable changes that come along with this:
1) Operating systems should now come with their etc in usr/etc; OSTree
will perform a 3-way merge at deployment time, and place etc in
the actual root. This avoids the need for a bind mount, and is
just a lot cleaner.
2) OSTree no longer bind mounts /root, /home, and /tmp. It is expected
that the the OS/ has these as symbolic links into /var.
At the moment, OSTree only supports managing syslinux; other
bootloader backends will follow.
