IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
This is a lot like `git clone --reference`, but we chose "localcache" as the
term "reference" is already used.
The main use case I'm targeting this for is the Fedora Atomic Host installer
case where we embed the repo content in the installer, but we may want to
kickstart and download newer content. There, while we want to get a newer ref,
we can still use the local repo as an object cache, since we have it sitting
there in memory anyways.
Another case is where one has a host ostree (say e.g. Fedora Atomic
Workstation), and one wants to create a local archive mirror of FAH. Then one
can use `pull --reference /ostree/repo` and pull the common objects (e.g.
contents of `bash.rpm` etc.)
Closes: https://github.com/ostreedev/ostree/issues/975Closes: #982
Approved by: jlebon
Previously, `ostree pull` was silent if not on a tty. I don't
see a reason not to print the final status line at least. This
is prep for more work in the test suite, so I can write assertions
on the output.
But it should also be nicer for people who e.g. do an `ostree pull` in a Jenkins
job or whatever.
Closes: #981
Approved by: jlebon
The `-z2` is annoying now since it's really a legacy; we've long
since supported typing `archive`. Convert the docs fully and
explain that.
Also do some (but not all) of the tests just to encourage newer tests to use
`archive` too.
Closes: #980
Approved by: jlebon
When updating a summary file, parse additional arguments to the `ostree
summary` command as additional metadata to be put into the summary.
Add some tests for this.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
Previously the spurious ‘update’ atom was ignored; now that `ostree
summary --update` accepts extra arguments as additional metadata, it’s
causing an error. Drop it.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
Test various pieces of core and command line utility functionality
relating to the newly-introduced concept of collections. Mostly focussed
around the find-remotes utility, and around handling of collection–refs.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a more complex implementation of OstreeRepoFinder which resolves
ref names to remote URIs by looking for refs advertised by peers on the
local network using DNS-SD records and mDNS (Avahi). The idea is to
allow OS and app updates to be propagated over local networks, without
the internet.
It requires an OSTree server and code to generate the DNS-SD adverts in
order to be fully functional — support for this will be added
separately.
Unit tests are included.
Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This will be used in an upcoming commit. It adds a basic bloom filter
implementation, using the SipHash family of hash functions.
The implementation (including its parameter choices and hash functions)
will become a protocol detail in future, so must not be changed so that
its output is bitwise incompatible between OSTree versions.
Unit tests are included.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.
Unit tests are included.
This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.
Unit tests are included.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #948
Approved by: cgwalters
test-symbols.sh was looking for the DevelBuild string, which is actually
part of the output from `ostree --version`, not $OSTREE_FEATURES.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #948
Approved by: cgwalters
And use it in `test-symbols.sh`, to fix the `distcheck` case;
the previous change stopped distributing `libostree-devel.sym`
in release builds.
Closes: #944
Approved by: jlebon
I only checked the test passed, I didn't read the output closely, and
made it succeed without testing anything.
Fix the absolute/relative `/etc` references.
Closes: #937
Approved by: jlebon
We're hitting this in flathub, where we have a bunch of local builds,
but we also mirror a few refs from the gnome runtime repo into it.
Its fixable by re-doing the summary, but for a short time the
wrong version is visible.
Fixes https://github.com/ostreedev/ostree/issues/846Closes: #935
Approved by: cgwalters
The code here tried to truncate the string to the previous length,
but that doesn't work when recursing, since further calls change the
length.
What actually ended up happening was the string would get corrupted
after the first level of recursion.
Closes: #936
Approved by: jlebon
So far a lot of submitted PR have added symbols into the first
section. Split the file into `-released` and `-devel` to make
this more obvious.
To further enforce things, we hardcode a checksum of the `-released`
file in `test-symbols.sh`. Only release commits should update that
checksum.
Did you notice I like checksums?
Closes: #931
Approved by: pwithnall
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849
It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.
Further, we might as well reject the content as early as possible.
Closes: #930
Approved by: alexlarsson
This imports a function that is used in rpm-ostree, and it's also intended for
use by https://github.com/advancedtelematic/aktualizr to display
what deployment we're going to boot next after the reboot.
Updated-by: Colin Walters <walters@verbum.org>
Closes: #897
Approved by: OYTIS
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.
See: https://github.com/flatpak/flatpak/issues/845
This option suppresses mode bits outside of `0775` for directory
checkouts. I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.
Closes: #927
Approved by: alexlarsson
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845
We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909
This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.
Closes: #926
Approved by: alexlarsson
Thinking about the problem of flatpak converting from `bare-user` to `bare-user-only`
"in place" by creating a new repo and doing a `pull-local`, I realized
that we can optimize this process by doing hardlinks for both metadata
and regular files. The repo formats are *almost* compatible, the
exception being symlinks.
An earlier patch caused us to do hardlinks for metadata, this patch takes things
to the next step and special cases this specific conversion. In this case we
need to parse the source object to determine whether or not it's a symlink.
Closes: #922
Approved by: alexlarsson
Our previous logic for import-via-hardlink only tried if the repo modes match,
but we *can* hardlink metadata between e.g. `archive` and `bare-user` repos, and
that's quite useful thing to do. Our documentation encourages converting to/from
those repo modes locally for build systems.
Closes: #922
Approved by: alexlarsson
See https://github.com/ostreedev/ostree/pull/909 for more information on the
rationale. Basically there's no reason for flatpak (which uses `bare-user-only`)
to have world-writable dirs. Particularly with the presence of the system
helper.
An approach I considered instead was to parse and validate directory metadata
objects at commit time. We still may do that in addition; for file objects we *had*
to do it that way because the actual files would be laid down suid. But directories
live only as inert `.dirmeta` objects until we do a checkout (i.e. `mkdir()`), so
we can solve the problem at checkout time.
Closes: #914
Approved by: alexlarsson
This copies the `ci/` directory from rpm-ostree, with much the same rationale;
among other things we don't want to depend on the Docker hub.
The specific reason I'm doing this is that I want to add a CentOS7 build, but
that means we can't use `projectatomic/ostree-tester`, and at that point we
might as well unwind it all.
Closes: #917
Approved by: jlebon
These exist in the wild for flatpak, and aren't really a problem. The canonical
permissions are still either `0755` or `0644`, we just support the additional
writable bit for the group (i.e. extend the set to include `0775` and `0664`)
now to avoid breaking some flatpak content.
Closes: #913
Approved by: alexlarsson
We didn't have coverage of this before, and adding the test infrastructure will
help ensure we have coverage for more changes here.
Closes: #913
Approved by: alexlarsson
Having every object in a bare-user repo (and checkouts) be executable
is ugly. I can't think of a good reason to do that; they should only
be executable if their input is. This does
for `bare-user` what we did for `bare-user-only` in
https://github.com/ostreedev/ostree/pull/909
It's also a stronger version of what we do with `checkout -U` in suppressing
suid - here we also strip world-writable files and the sticky bit (even though
that's meaningless today, it might not be in the future).
Closes: https://github.com/ostreedev/ostree/issues/907Closes: #908
Approved by: alexlarsson
If there are no deltas to be listed in the summary file, don’t bother
including the key for them in the additional metadata section of the
file. This saves a few bytes in some cases.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
This follows on from commit a946c3d4, which added formatting for
well-known summary metadata keys. This commit adds it for commits.
Currently, the only well-known commit metadata key is
ostree.commit.timestamp. Formatting this correctly is especially
important, since it’s a big-endian uint64, which is completely unusable
for mere mortals when presented as a number rather than a date.
Non-formatted output can still be retrieved using the OSTREE_DUMP_RAW
flag, and the non-formatted key name is always printed for clarity.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
For the flatpak use case where bare-user-only was introduced, we actually
don't want to support s{u,g} id files in particular.
Actually, I can't think of a reason to have anything outside of the
`0755 i.e. (u=rwx,g=rx,o=rx)` mask, so that's what we do here.
This will have the effect of treating existing `bare-user-only` repositories as
corrupted if they have files outside of that mask, but I think we should do this
now; most of the flatpak users will still be on `bare-user`, and we haven't
changed the semantics of that mode yet.
Note that in this patch we will also *reject* file content that doesn't
match this. This is somewhat asymmetric, since we aren't similarly rejecting
e.g. directory metadata. But, this will close off the biggest source
of the problem for flatpak (setuid binaries).
See: https://github.com/ostreedev/ostree/pull/908
See: https://github.com/flatpak/flatpak/pull/837Closes: #909
Approved by: alexlarsson
It's not enough to check that the dir exists, since that's done by
default when we open the repo. We want to actually check that
uncompressed objects were cached (i.e. the opposite of the earlier error
path).
Closes: #903
Approved by: cgwalters
When falling back to copying, we previously would only chmod checked out
files in the non-user-checkout mode. Fix this by always doing chmod.
The file_mode was being prepared but never actually applied.
Add a basic test in the archive-z2 --> usermode checkout case in which
we're guaranteed to always fall back to copy mode.
Closes: #633Closes: #903
Approved by: cgwalters
This reverts commit 1eff3e8343. There
are a few issues with it. It's not a critical thing for now, so
let's ugly up the git history and revisit when we have time to
debug it and add more tests.
Besides the below issue, I noticed that the simple `ostree remote add`
now writes to `/ostree/repo/config` because we *aren't* using the
`--sysroot` argument.
Closes: https://github.com/ostreedev/ostree/issues/901Closes: #902
Approved by: mike-nguyen
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).
In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.
Closes: https://github.com/ostreedev/ostree/issues/892Closes: #896
Approved by: jlebon
The redhat-ci service has been renamed to PAPR. Previous values are
still supported though should be considered deprecated.
Closes: #885
Approved by: cgwalters
A commit can now include a "ostree.endoflife-rebase" metadata key
pointing to a new ref.
When updating, the sysroot upgrader will see this and proceed to
pull and deploy the new ref instead. The origin file in the new
deployment will point to the new ref.
This functionality is planned to be used in Endless OS. We will create
a lesser tested branch for brand new, cutting edge hardware support,
and ship that on hardware platforms that require the latest drivers.
However, once our slower-moving official release is later updated to
support the new hardware, we will use this functionality to migrate
those bleeding-edge users over to the official release.
Closes: #874
Approved by: cgwalters
I want to keep this a "pure copy-able" file into various projects
like rpm-ostree, bwrap, and flatpak. Pull in changes from rpm-ostree
to prep for that.
While we have the patient open, dedup the code for file matching a bit.
Closes: #877
Approved by: jlebon
If somehow a repo has gpg verification on but doesn't have signatures
present for the existing commit, ostree would error out if it needs to
scan the commit object (e.g. if there are no updates available).
An instance of this is currently happening in Fedora AH, in which
signatures are not shipped in the ISO due to filesystem restrictions.
Another possible scenario is if a content provider switches from not
signing commits to signing them; even if older commits are retroactively
signed, clients' local commit objects would error out if they needed
scanning.
This patch adds a check to ensure that we always attempt to fetch the
detached metadata and wait for its result (whether it exists or not)
before moving on to scan their corresponding commit objects.
See also: https://github.com/projectatomic/rpm-ostree/issues/630Closes: #873
Approved by: cgwalters
Special-case when this script is run under RHCI, which will try to fetch
the merge commit if possible. Use RHCI_COMMIT instead to refer to the
actual PR/branch HEAD being evaluated.
Use realpath to workaround the developer's git dir being in a symbolic
link.
Closes: #857
Approved by: cgwalters
This is what caused the merge of
https://github.com/projectatomic/rpm-ostree/pull/652
to blow up, since https://github.com/ostreedev/ostree/pull/848
landed right before we tried to merge it.
When I was writing that PR I remember having an uncertain feeling
since we were doing a `mkdirat` above, but at the time I thought
we'd have test suite coverage...turns out we didn't.
For backwards compatibility, we need to continue to do a `mkdirat` here of the
parent. However...I can't think of a reason anyone would *want* that behavior.
Hence, let's add a special trick - if the destination name is `.`, we skip
`mkdirat()`. That way rpm-ostree for example can open a dfd for `/etc` and avoid
the `mkdir`.
Fold the subpath tests into `test-basic.sh` since it's not worth a separate
file. Add a test case for checking out a file.
Closes: #854
Approved by: jlebon
As $DEITY intended.
I was reading the `prepare-root.c` code and the indentation damage was
distracting. Squash tabs that have leaked into various places in the code. I
didn't yet touch the `src/libostree` bits as that has higher potential for
conflict.
Closes: #852
Approved by: jlebon
Rather than `g_output_stream_splice()`, where the input is a regular
file.
See https://github.com/GNOME/libglnx/pull/44 for some more information.
I didn't try to measure the performance difference, but seeing the
read()/write() to/from userspace mixed in with the pointless `poll()` annoyed me
when reading strace.
As a bonus, we will again start using reflinks (if available) for `/etc`,
which is a regression from the https://github.com/ostreedev/ostree/pull/797
changes (which before used `glnx_file_copy_at()`).
Also, for the first time we'll use reflinks when doing commits from file-backed
content. This happens in `rpm-ostree compose tree` today for example.
Update submodule: libglnx
Closes: #817
Approved by: jlebon
I think tests/ should be just that, ci/ is separate. Also rename
the files to include "travis" since that's what we use them
for right now.
Closes: #843
Approved by: jlebon
These at the moment aren't in a container, and may need space. In the future
overlay2 will help here, we can more easily extend the rootfs.
Closes: #840
Approved by: jlebon
`test-pull-many.sh` is was just too slow to be a unit test. Generating
a bunch of files via shell is slow, the delta generation is slow, etc.
Every developer doesn't need to run it every time.
Somewhat address this by converting it into our installed test framework, which
moves it out of the developer fast paths. Another advantage to this is
that we can simply reuse the FAH tree content rather than synthesizing
new bits each time.
Closes: #840
Approved by: jlebon
There are currently no unstable APIs, but some will be added in
following commits. They will be built and exposed in the libostree
global symbol list iff configured with --enable-experimental-api.
Distributions should not package OSTree with --enable-experimental-api.
This is designed for previewing new APIs on controlled platforms; any of
the APIs hidden behind this option may be changed or removed at any
point.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
The major reason to do this is that running tests *both* installed
and uninstalled in our CI is a mostly pointless waste of time.
Particularly given we have a few expensive tests.
We *do* have tests that only run uninstalled (since they require
the source code) like `test-symbols.sh`.
Hence, add `--enable-installed-tests=exclusive` to mean *only* do installed for
most tests.
We'll still have uninstalled coverage via the Travis/Debian configs, and we
could perhaps do another build with a subset of uninstalled tests, but I'm not
really concerned about it.
I'd like to do a renewed push for the InstalledTests model since
I feel it's just fundamentally better. (`g-d-t-r` kind of sucks,
but then so does the automake runner).
Also while we're here - fix the CI to use the correct context,
which started this mess.
Closes: #837
Approved by: dbnicholson
Since b825aac, trivial-httpd is in $libexecdir/libostree by default and
not available through the ostree runner in PATH. Try to adjust find it
when running the tests installed.
Closes: #837
Approved by: dbnicholson
There were some regex special characters in the pattern strings, which I
think were causing the test to fail on some Travis builds due to using
an invalid regex.
Fix that by matching using fixed strings instead. We don’t need regexes
here. Use a new assert_file_has_content_literal to do that for us.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #838
Approved by: cgwalters
Allow GI bindings to delete refs through ostree_repo_transaction_set_ref
and ostree_repo_transaction_set_refspec by setting the checksum to NULL.
Closes: #834
Approved by: cgwalters
If we're freeing the segment, it's basically always better to use
`autoptr()`. Fewer lines, more reliable, etc.
Noticed an instance of this in the pull code while reviewing a different PR,
decided to do a grep for it and fix it tree wide.
Closes: #836
Approved by: pwithnall
This is a variant of the efforts in https://github.com/ostreedev/ostree/pull/741
Working on `rpm-ostree livefs`, I realized though I needed to just
check out *new* files directly into the live `/etc` (and possibly
delete obsolete files).
The way the current `/etc` merge works is fundamentally different from
that. So my plan currently is to probably do something like:
- Compute diff
- Check out each *new* file individually (as a copy)
- Optionally delete obsolete files
Also, a few other things become more important - in the current deploy code, we
copy all of the files, then relabel them. But we shouldn't expose to *live*
systems the race conditions of doing that, plus we should only relabel files we
checked out.
By converting the deploy's /etc code to use this, we fix the same TODO item
there around atomically having the label set up as we create files. And further,
if we kill the `/var` relabeling which I think is unnecessary since Anaconda
does it, we could delete large chunks of code there.
In the implementation, there are two types of things: regular files, and
symlinks. For regular files, in the `O_TMPFILE` case, we have the ability to
do *everything* atomically (including SELinux labeling) before linking it into
place. So let's just use that. For symlinks, we use `setfscreatecon()`.
Closes: #797
Approved by: jlebon
Our container-driven tests can't e.g. test SELinux sanely, and
have to support being run as root *and* non-root too.
Use redhat-ci to provision a VM and run tests directly there. These are
installed tests too.
Closes: https://github.com/ostreedev/ostree/issues/806Closes: #807
Approved by: jlebon
This could be shared more easily with e.g. rpm-ostree, but what I'm currently
working on is installed, privileged (potentially destructive, i.e. VM) tests
that will source this separately from the current `libtest.sh`. That does work
installed, but in practice is oriented around unit (uninstalled, unprivileged)
tests.
Closes: #807
Approved by: jlebon
We really have an astonishing variety of similar functions which write files and
symlinks. I was working on a different PR and the duplication between the
union-mode and add-mode/none-mode checkout functions bothered me.
I realized that the "handle EEXIST" tri-state maps directly to the
`GLnxLinkTmpfileReplaceMode`, so deduping things makes even more sense.
Closes: #801
Approved by: jlebon
This is intended to be used for copying `/usr/etc` → `/etc` for
deployments.
A TODO here is to use `glnx_file_copy_at()` if the repo mode allows
it - then we'd use reflinks if available.
Closes: #804
Approved by: jlebon
Logic error introduced after refactoring; we hoisted the
`is_bare_user_symlink` variable to the top, but its computation
below. But the `is_bare` symlink depended on it.
Closes: https://github.com/ostreedev/ostree/issues/798Closes: #799
Approved by: jlebon
I think the majority of OSTree usage calls pull with refs, not
explicit commits. We even added special "override syntax" with
`@` (e.g. `ostree pull foo@ab12c34`) as a hybrid.
However, some users may want to still pull explicit commits
for whatever reason. The old static delta logic looked at
the previous commit of the ref. However, in https://github.com/ostreedev/ostree/pull/710
we enhanced the logic to look at all local commits.
It's now a lot more natural to teach the delta logic
to support revisions, e.g. `ostree pull someorigin ab12c34`.
This also fixes the problem that before, `--require-static-deltas`
was completely ignored when processing revisions.
This is a nontrivial refactoring of the logic, but the end
result feels a lot more readable to me.
Closes: https://github.com/ostreedev/ostree/issues/783Closes: #787
Approved by: cgwalters
Previously we'd assert and dump core if one used `checkout -H` without
`-U` on a bare-user repo, because we'd hit the bare-user symlink case.
Rework the code to handle this, and add tests. I hit this when I was going to
suggest to someone to use `-H` to ensure they were getting hardlinks.
Closes: #779
Approved by: jlebon
This is inspired by the [Coccinelle](http://coccinelle.lip6.fr/) usage
in systemd. I also took it a bit further and added infrastructure
to have spatches which should never apply. This acts as a blacklist.
The reason to do the latter is that coccinelle is *way* more powerful than the
regular expresssions we have in `make syntax-check`.
I started with blacklisting `g_error_free()` directly. The reason that's bad is
it leaves a dangling pointer.
Closes: #754
Approved by: jlebon
This is somewhat complicated by such repos only properly supporting
some subset of file metadata (uid/gid 0, etc). We fix this by
always commiting with filters that make it work.
Closes: #750
Approved by: cgwalters
There are a lot of things suboptimal about this approach, but
on the other hand we need to get our CI back up and running.
The basic approach is to - in the test suite, detect if we're on overlayfs. If
so, set a flag in the repo, which gets picked up by a few strategic places in
the core to turn on "ignore xattrs".
I also had to add a variant of this for the sysroot work.
The core problem here is while overlayfs will let us read and
see the SELinux labels, it won't let us write them.
Down the line, we should improve this so that we can selectively ignore e.g.
`security.*` attributes but not `user.*` say.
Closes: https://github.com/ostreedev/ostree/issues/758Closes: #759
Approved by: jlebon
The first options are owner_uid/owner_gid, which makes it possible to use diff
on local files where --owner-uid/gid have been passed to commit.
Closes: #740
Approved by: cgwalters
At least in all Linux kernels up to today, one can never `link()` across
devices, so we might as well verify that up front. This will help for a future
patch to add a new type of union-add checkout, since Linux checks for `EEXIST`
before `EXDEV`.
Closes: #714
Approved by: jlebon
The previous logic for static deltas was to use as a FROM
revision the current branch tip. However, we want
to support deltas between branches in an automatic
fashion.
If a summary file is available, we already have an
enumerated list of deltas - so the logic introduced
here is to search it, and find the newest commit
we have locally that matches the TO revision target.
This builds on some thoughts from
https://github.com/ostreedev/ostree/pull/151#issuecomment-232390232
Closes: https://github.com/ostreedev/ostree/pull/151Closes: #710
Approved by: giuseppe
In https://github.com/ostreedev/ostree/pull/408, we disabled the use of
static deltas when mirroring. Later,
https://github.com/ostreedev/ostree/pull/506 loosened this up again so
that we could use static deltas when mirroring into bare{-user} repos.
However, the issue which originally spurrred #408 is even more generic
than that: we want to avoid static deltas for any archive repo, not just
when doing a mirror pull. This patch tightens this up, and also
relocates the decision code to make it easier to read.
Closes: #715
Approved by: cgwalters
The C API (ostree_repo_static_delta_generate) knows what to do
with it, but this parameter was never exposed via command line
tool.
Closes: https://github.com/ostreedev/ostree/issues/695Closes: #703
Approved by: jlebon
This makes it easier to script downloading updates in the background,
and only do deployments just before rebooting.
Partially addresses https://github.com/ostreedev/ostree/issues/640Closes: #642
Approved by: jlebon
I learned today that `docker version` does this and I really like
the idea. While we have the patient open, also add the gitrev
with code taken from https://github.com/projectatomic/rpm-ostree/pull/584Closes: #691
Approved by: giuseppe
I think I commented this out while debugging something, and forgot to re-enable
it. Reading the log files should be a better again after this.
Closes: #699
Approved by: giuseppe
It's just simpler, and I'm not sure people are going to care
much about the difference by default.
We already folded in the fallback sizes into the download totals, so folding in
the count makes things consistent; previously you could see e.g.
`3/3 parts, 100MB/150MB` and be confused.
Closes: #678
Approved by: giuseppe
For rpm-ostree, we already link to libcurl indirectly via librepo, and
only having one HTTP library in process makes sense.
Further, libcurl is (I think) more popular in the embedded space. It
also supports HTTP/2.0 today, which is a *very* nice to have for OSTree.
This seems to be working fairly well for me in my local testing, but it's
obviously brand new nontrivial code, so it's going to need some soak time.
The ugliest part of this is having to vendor in the soup-url code. With
Oxidation we could follow the path of Firefox and use the
[Servo URL parser](https://github.com/servo/rust-url). Having to redo
cookie parsing also sucked, and that would also be a good oxidation target.
But that's for the future.
Closes: #641
Approved by: jlebon
As OSTree has evolved over time, the tests grew with it. We
didn't start out with static deltas or a summary file, and the
tests reflect this.
What I really want to do is change more of the pull tests, from
corruption/proxying/mirroring etc. to use this more realistic
repo rather than the tiny one the other test creates.
We start by using some of the code from `test-pull-many.sh`, and change that
test to use `--disable-static-deltas` for pull, since the point of that test is
to *not* test deltas.
Still TODO is investigate changing other tests to use this.
Closes: #658
Approved by: jlebon
We weren't running it before. Also I switched it to use GLib. Preparation for
some oxidation work (having an implementation of bupsplit in Rust).
I exported another function to do the raw rollsum operation which is what this
test suite uses.
Closes: #655
Approved by: jlebon
This would be more likely to tickle things like
https://github.com/ostreedev/ostree/issues/601
reliably.
Also, while working on the curl backend, I hit on the fact that curl doesn't
queue (by default, you can enable) and will happily create 20000+ concurrent TCP
connections if you try. Having this test would have made that more likely to
fail.
Closes: #650
Approved by: giuseppe
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.
This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.
Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.
Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.
Closes: https://github.com/ostreedev/ostree/issues/604Closes: #646
Approved by: jlebon
Debian's Lintian packaging consistency check complains that it isn't
executable but has a #! line. In fact it's reasonable to run this
script directly, so make it executable, and put it in a _scripts
variable so it will be installed executable.
Closes: #652
Approved by: cgwalters
They are installed non-executable, which makes Debian's Lintian
packaging consistency check complain that #! is only useful
in executable scripts. But in fact they are not useful to execute
directly (they rely on setup being done in the script that sources
them), so just chmod them -x.
Closes: #652
Approved by: cgwalters
In its initial commit, Alexander Larsson wrote
This works standalone, but unfortunately it breaks in
gnome-desktop-testing-runner as /tmp doesn't support
xattrs, so it is not installed atm.
but we now (a) use /var/tmp, and (b) explicitly skip the test if
xattr support is unavailable. So it should be OK to run now.
Closes: #652
Approved by: cgwalters
libcurl AFAICS doesn't have an API to convert HTTP code ➡️ error
string, so let's make the test regexp operate on both.
Closes: #651
Approved by: giuseppe
It turns out libsoup strips all whitespace even *inside* a URL. We could do that
for libcurl too but...really, people shouldn't do that. In this test we were
adding the trailing newline into the URL. If someone complains who is using the
libcurl code we can deal with it then.
Closes: #651
Approved by: giuseppe
We had a lot of copies of the "echo something 1>&2; exit 1" code even though
`assert_not_reached()` was it. Hence, I think we need a shorter alias for that.
Doing this particularly since I noticed a missing `1` in an `exit 1` call in the
rpm-ostree copy of this.
Closes: #648
Approved by: jlebon
Working on the libcurl backend, I hit the issue that the trivial-httpd program
depends on libsoup. I briefly considered having two versions, but libcurl is
client only, and moreover trivial-httpd is no longer trivial - it has various
features which are used by the test suite extensively.
Hence, what we'll do is build it as a separate binary which links to libsoup,
and use it during the tests. We *also* currently still provide `ostree
trivial-httpd` since some things use it like `rpm-ostree-toolbox` and the
Cockpit tests.
After those are ported to use some other webserver, I plan to add a build-time
option to drop it.
Closes: #636
Approved by: jlebon
This is a migration from the origin version. It's
nicer to have it in the remote, since that's what one
needs to change. Then tools don't need to mess with
the origin file.o
In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from. We're just providing a better error.
Closes: https://github.com/ostreedev/ostree/issues/626Closes: #627
Approved by: jlebon
The spam in stderr was bothering me, and further at some eventual
point in the future we want to annotate the functions with
`__attribute__((nonnull))` which would then cause tests like these to
become undefined behavior.
The coverage of this isn't worth the log spam basically.
Closes: #611
Approved by: jlebon
This reduces the diff when comparing these scripts with similar glue
in dbus or elsewhere.
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters
This is deliberately permissive: a lot of it is generic, and I'm
using similar scripts in dbus.
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters
Otherwise, we'll fail (due to set -u) if this parameter variable isn't
passed.
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters
glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.
Closes: #591
Approved by: cgwalters
The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability. Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.
Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.
Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have. However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.
Closes: https://github.com/ostreedev/ostree/issues/517Closes: #589
Approved by: jlebon
You'd expect
ostree commit --tree=ref=A --tree=ref=B
to produce a commit with the union of the trees given. Instead you'd get
a commit with the contents of just the latter commit. This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead. This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.
This commit removes the optimisation, fixing the bug. Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.
Closes: #581
Approved by: cgwalters
Conceptually we've been moving towards having our GPG verification
paths be per-remote. The code internally supports this, but we
didn't expose an API to use it conveniently.
This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.
Adding an API fixes this nicely.
Closes: #576
Approved by: giuseppe
For Project Atomic, we already have RPM signatures which use files in
`/etc/pki/rpm-gpg`. It's convenient to simply bind the OSTree remote
configuration to those file paths, rather than having duplicate key
data.
This does mean that we need to parse the files for verification, so we
end up importing them into the verifier's temporary keyring, which is
a bit ugly, but it's what other projects do.
Closes: https://github.com/ostreedev/ostree/issues/573Closes: #575
Approved by: giuseppe
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.
Also adds a testcase for this.
Closes: #577
Approved by: cgwalters
Some deployments may want to gate access to content based on things
like OAuth. In this model, the client system would normally compute a
token and pass it to the server via an API.
We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.
Closes: #574
Approved by: giuseppe
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.
I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.
Closes: https://github.com/ostreedev/ostree/issues/528Closes: #566
Approved by: jlebon
Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.
Closes: #560
Approved by: cgwalters
Otherwise several tests fail, for example in this build done in a French
locale by Debian's reproducible builds initiative, to check whether
the resulting binaries are identical to what was produced in an
English locale:
<https://tests.reproducible-builds.org/debian/logs/unstable/amd64/ostree_2016.11-1.build2.log.gz>
(test-basic)
# error: Cannot write to repository: Permission non accordée
...
File 'error-message' doesn't match regexp 'Permission denied'
(test-help)
# Utilisation :
# ostree [OPTION...] COMMAND
...
File 'out' doesn't match regexp '[Uu]sage'
(test-pull-metalink)
# error: Erreur à la ligne 1, caractère 1 : Le document doit commencer avec un élément (par ex. <book>)
...
File 'err.txt' doesn't match regexp 'Document must begin with an element'
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #558
Approved by: cgwalters
I'm doing builds and `make check` inside a Docker container, with
selinux on as a build-time option, but no policy in the container.
This currently aborts. Let's not do that.
(This type of thing is why installed tests are a better model)
Closes: #546
Approved by: jlebon
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.
In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.
Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.
This fixes https://github.com/ostreedev/ostree/issues/541Closes: #542
Approved by: cgwalters
Quoting Dan Nicholson in
<https://github.com/ostreedev/ostree/pull/330#issuecomment-245499099>
mtime of 0 has been the semantics of ostree deployments from basically
the beginning of the project. We (and others, see
flatpak/flatpak@b5204c9) rely on that fact when generating trees.
In particular, this affects caches that use the mtime of the
associated file or directory to determine if the cache is valid. By
arbitrarily changing the mtime of the files to something else, all
the caches we setup in the build are now invalidated. Preseeding
caches is really important to the user experience as it avoids
having the user wait while they're regenerated on first run.
Now, we could change our build infrastructure to preset all the
mtimes to 1 to match this change, but what does that do for our
existing users who are on an ostree that deploys with mtimes of 0?
We could just revert this change at Endless (and the associated one
in Flatpak), and that would be fine for our users. However, if we
point non-Endless users to our apps, they'll have the great
experience of waiting 10 seconds the first time they launch it while
the fontconfig cache is rebuilt unnecessarily.
Closes: #495
Approved by: jlebon
musl libc's implementation of `realpath` works by opening the path and then
doing a lookup in `/proc/self/fd` to find the canonical path. This fails
if `/proc` is not mounted. This causes problems for us if
`ostree-prepare-root` is `init` as `/proc` won't be mounted.
We have to mount `/proc` anyway for `/proc/cmdline` so this fix just
expands the scope over which `/proc` is mounted to include both our
`realpath` calls.
See also:
* http://www.openwall.com/lists/musl/2016/06/08/2 and
* http://git.musl-libc.org/cgit/musl/tree/src/misc/realpath.c?id=e738b8cbe64b6dd3ed9f47b6d4cd7eb2c422b38dCloses: #485
Approved by: cgwalters
This test previously depended on manually building ostree-prepare-root.
Since 42dab85 we've been able to build static binaries with the usual
autotools build-system. This change reflects the fact that
`ostree-prepare-root` is built into $srcdir rather than `src/switchroot`
where I was building manually.
This test now passes with `./configure --with-static-compiler=gcc` (glibc)
but still fails with `./configure --with-static-compiler=musl-gcc` (musl).
Closes: #485
Approved by: cgwalters
This is a proper fix for:
https://bugzilla.gnome.org/show_bug.cgi?id=755787
With this patch, an admin (system builder) can now:
1) Edit /usr/lib/ostree-boot/uEnv.txt
2) Deploy the new tree. OSTree will append system's uEnv.txt
to the OSTree's managed uEnv.txt (loader/uEnv.txt).
It is common for u-boot systems to read in an extra env
from external /uEnv.txt. The same file OSTree uses to pass
in its env. With this patch /uEnv.txt now contains OSTree's
env + custom env added by system builders.
Closes: #466
Approved by: cgwalters
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail. There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.
Closes: https://github.com/ostreedev/ostree/issues/467Closes: #471
Approved by: jlebon
The test suite was actually doing something before, just
not quite what I intended. Without `-U` for bare-user checkouts
we end up doing a copy.
Now, a future commit will change how rofiles work, which would cause
the test suite to permit inplace mutation for non-hardlinked files.
So let's ensure they *are* hardlinked.
Closes: #462
Approved by: jlebon
Pulls from local repos now default to disabling static deltas so that
objects are copied. To check this is the case, see if the object files
are hardlinked after pulling.
Closes: #447Closes: #448
Approved by: cgwalters
When testing pulling of deltas, use the new --require-static-deltas
option to pull-local to ensure that deltas are actually used. To support
the require-static-deltas mode, the summary in the remote repo must be
generated.
Closes: #447Closes: #448
Approved by: cgwalters
Lots and lots of preparation led to this moment - when nothing
apparent changes for users! Woo!
But seriously, having the extra dependency is a minor annoyance, and
in the big picture I think the libgsystem idea was wrong - we need to
land things in GLib, and use git submodules for API-unstable or
Linux-specific sharing. For a lot of OSTree, the libgsystem `GFile*`
orientation was also wrong, we really want fd-relative.
Closes: #444
Approved by: jlebon
I hit an error with [CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel)
where we were doing time-based pruning. `ostree summary -u` started failing,
and it took me a bit to realize it was because we were pruning
even the tip of old branches, which I was not at all expecting,
and I don't think users will too.
Perhaps in the future we could add some sort of --prune-ref-tips or
something if people wanted it, but I doubt it.
Closes: #445
Approved by: jlebon
.travis.yml is obviously still Travis-specific, but tests/ci-* are
designed to be shareable with other CI environments if there is interest
in doing so.
At the moment I'm only testing on Debian and Ubuntu. In principle we
could try a non-Debian-derived Docker container such as Fedora or CentOS
inside travis-ci's Ubuntu environment, similar to what I'm doing
for Debian, but I don't know the correct setup commands to use there.
Closes: #438
Approved by: cgwalters
ostree's naming convention for whiteouts is similar to what is
done in aufs, which means we can't compose the trees to test this
feature under Docker with the aufs storage driver, as used on
travis-ci.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #437
Approved by: cgwalters
