Commit Graph

648 Commits

Author SHA1 Message Date
Simon McVittie
2f22e5b359 test-help.sh: Skip trivial-httpd if enabled
Because it runs a binary in ${libexecdir}, it will only work if
libostree was already installed, which makes the build-time test fail.
It also doesn't produce the output we expect: its usage mechanism
mentions "ostree-trivial-httpd", not "ostree trivial-httpd".

Signed-off-by: Simon McVittie <smcv@collabora.com>

Closes: #1177
Approved by: jlebon
2017-09-15 19:28:25 +00:00
Simon McVittie
1b430a7764 tests: Fix JavaScript tests with gjs 1.50.0
In recent gjs, you can't declare a variable with "let" multiple times.

Signed-off-by: Simon McVittie <smcv@collabora.com>

Closes: #1178
Approved by: cgwalters
2017-09-15 18:51:43 +00:00
Colin Walters
6c0738a000 Release 2017.11
Closes: #1173
Approved by: jlebon
2017-09-14 15:04:42 +00:00
Colin Walters
1c2d344074 tests: Port some bits of C to new style
Where we can; perhaps after updating libglnx we should use the
new test error macro?

Closes: #1169
Approved by: jlebon
2017-09-13 19:32:36 +00:00
Colin Walters
051cdf396c lib/checkout: Rename disjoint union, change to merge identical files
It turns out that librpm automatically merges identical files between
distinct packages, and this occurs in practice with Fedora today between
`chkconfig` and `initscripts` for exmaple.

Since we added this for rpm-ostree, we basically want to do what librpm does,
let's change the semantics to do a merge.  While we're here rename
to `UNION_IDENTICAL`.

Closes: #1156
Approved by: jlebon
2017-09-13 19:19:33 +00:00
Dan Nicholson
3b315e16d8 repo: Ensure new config doesn't set remotes in separate file
If the new configuration passed to ostree_write_config () tries to
update options for a remote defined in a separate config file, return an
error. Without this, the full configuration would contain duplicate
remote specifications, which would raise an error the next time the repo
is opened.

Closes: #1159
Approved by: cgwalters
2017-09-13 16:03:25 +00:00
Jonathan Lebon
077de8ea46 tests/test-help.sh: Rework and strengthen checks
The `sed` expression wasn't actually matching the main output, so we
weren't recursing into the subcommands. Update the syntax to match the
current output and add a check so we don't miss that happening again.

Add a check that the help output is only printed once in all
circumstances. Also add a check for proper handling of non-existent
commands.

Closes: #1126
Approved by: cgwalters
2017-09-13 14:32:20 +00:00
Dan Nicholson
adac42b6ef repo: Add add-remotes-config-dir option
This option allows a repo to explicitly opt out of adding new remotes in
a remotes configuration directory. This currently defaults to true for
system repos and false for non-system repos to maintain legacy behavior
that non-system repos don't add remotes in a configuration directory.
That would be problematic for flatpak, which specifies a remotes config
dir but adds remotes in ways that are incompatible with it.

So, what this really does is allow system repos to control whether they
want to add remotes in the config dir or not. That's important if your
flatpak repo is the system repo like at Endless.

Closes: #1134

Closes: #1155
Approved by: cgwalters
2017-09-11 10:53:20 +00:00
Alexander Larsson
08eaf66827 rofiles-fuse: Fix lchown() and hardlink verification for symlinks
If you lchown("symlink") then we were incorrectly trying to chown the
symlink target, rather than the symlink itself. In particular, this cause
cp -a to fail for a broken symlink. Additionally, it was using the
symlink target when verifying writability, rather than the symlink
itself.

To fix this, we need pass AT_SYMLINK_NOFOLLOW in these cases.

In general, the kernel itself will always resolve any symlinks for us
before calling into the fuse backend, so we should really never do any
symlink following in the fuse fs itself. So, we pro-actively add
NOFOLLOW flags to a few other places:

 truncate:
      In reality this will never be hit, because
      the kernel will resolve symlinks before calling us.
 access:
      It seems the current fuse implementation never calls this
      (faccessat w/AT_SYMLINK_NOFOLLOW never reaches the fuse fs)
      but if this ever is implemented this is the correct behaviour.

We would ideally do `chmod` but this is not implemented on current kernels.
Because we're not multi-threaded, this is OK anyways.

Further, our write verification wasn't correctly handling the case of hardlinked
symlinks, which can occur for `bare` checkouts but *not* `bare-user` which the
tests were using. Change to `bare` mode to verify that.

Closes: #1137
Approved by: alexlarsson
2017-09-08 20:38:39 +00:00
Colin Walters
3594bb2d0f lib: Add a private helper to abort txns, use in sysroot cleanup
Steal some code from flatpak for this, which allows porting a few more things to
new style. I started on a public API version of this but was trying to roll some
other things into it and it snowballed. Let's do this version since it's easy
for now.

While here I changed things so that `generate_deployment_refs()` now just uses
`_set_ref_immediate()` rather than requring a txn.

Also, AFAICS there was no test coverage of `generate_deployment_refs()`; I tried
commenting it out and at least `admin-test.sh` passed. Add some coverage of this
- I verified that with this commenting out bits of that function cause the test
to fail.

Closes: #1132
Approved by: jlebon
2017-09-08 16:25:06 +00:00
Dan Nicholson
43c78c9006 repo: Fix non-system remotes-config-dir usage
Before commit e0346c1, a non-system repo could specify
remotes-config-dir and have remotes read from there. However, adding
remotes would only be done in the config dir for a system repo. Restore
that by respecting remotes-config-dir when no sysroot is found and
adding back the ostree_repo_is_system() check when adding remotes.

Closes: #1133

Closes: #1151
Approved by: cgwalters
2017-09-08 13:54:30 +00:00
Guy Shapiro
2a7fdfdbc5 uboot: add non-default deployments to uEnv.txt
Include non-default deployments in the uEnv.txt file imported by
U-Boot. All the configurations beside the defaults will have
numerical suffix E.G. "kernel_image2" or "bootargs2".
Those U-Boot environment variables may be used from interactive boot
prompt or from "altbootcmd" script.

Closes: #1138
Approved by: cgwalters
2017-09-08 00:58:08 +00:00
Robert McQueen
2d854368a8 lib/gpg: Add _FINGERPRINT_PRIMARY to OstreeGpgVerifyResult
Revert the switch of _FINGERPRINT to giving the primary key ID
rather than the signing key ID, and instead add the primary
key ID as a new attribute which is available if the key is not
missing.

Closes: https://github.com/ostreedev/ostree/issues/608

Closes: #1092
Approved by: cgwalters
2017-09-07 19:56:31 +00:00
Matthew Leeds
1e3f87c34c tests: Check "refs -c PREFIX" behavior
This commit adds tests to check the behavior of "refs -c PREFIX", where
prefix is interpreted as a collection ID.

Closes: #1149
Approved by: cgwalters
2017-09-07 18:49:55 +00:00
Colin Walters
57509e4d50 tests/rofiles-fuse: Add tests for chmod/chown
Prep for https://github.com/ostreedev/ostree/pull/1137 where
we were incorrectly handling `chown()` on symlinks.

Closes: #1141
Approved by: jlebon
2017-09-07 16:32:08 +00:00
Colin Walters
9c4106f166 bin/local-pull: Clarify docs, add more tests for corrupted local pulls
I was reading the pull-local command docs and realized it was somewhat unclear
that `--untrusted` *only* applied to local repo pulls; in other words that we
always treat non-local pulls as untrusted.

Tweak the docstring, and add tests that verify this explicitly.

Closes: #1130
Approved by: jlebon
2017-09-07 15:30:11 +00:00
Colin Walters
88b25dc65c tree-wide: Replace archive-z2 with archive
In almost all places. There are just a few exceptions; one tricky bit for
example is that the repo config must still have `mode=archive-z2`, since
`archive` used to mean something else. (We could very likely just get rid of
that check, but eh, later).

I also added a test that one can still do `ostree repo init --mode=archive-z2`.

Closes: #1125
Approved by: jlebon
2017-09-01 20:54:12 +00:00
Ruixin
f07432d4ce checkout: add an extra checkout_overwrite mode
This is for issue projectatomic/rpm-ostree#365,
an extra option of overwrite mode is added to the checkout command
so that when there is "non-directory" file already exist
during checkout, the error will be handled.

Some tests are added for regression

Closes: #1116
Approved by: cgwalters
2017-09-01 15:42:50 +00:00
Jonathan Lebon
12114ce382 commit: filter out selinux label before commit
The new --selinux-policy added in [0] exposed a subtle issue in the way
we handle labeling during commit. The CI system in rpm-ostree hit this
when trying to make use of it[1].

Basically, because of the way we use a GVariant to represent xattrs, if
a file to be committed already has an SELinux label, the xattr object
ends up with *two* label entries. This of course throws off fsck later
on, since the checksum will have gone over both entries, even though the
on-disk file will only have a single label (in which the second entry
wins).

I confirmed that the `fsck` added in the installed test fails without
the rest of this patch.

[0] https://github.com/ostreedev/ostree/pull/1114
[1] https://github.com/projectatomic/rpm-ostree/pull/953

Closes: #1121
Approved by: cgwalters
2017-08-31 12:07:46 +00:00
Matthew Leeds
93d2590c35 tests: Make the deployment mutable in test-sysroot.js
We attempt to make deployments mutable in the test suite (as opposed to
immutable which is the default) to make it easier to chmod and clean up
the tmp files after each test. This is normally accomplished by setting
OSTREE_SYSROOT_DEBUG=mutable-deployments in libtest.sh, but that only
affects the environment variables for that bash instance, not the
process running gjs. So in test-sysroot.js OSTREE_SYSROOT_DEBUG wasn't
set when sysroot.deploy_tree() was called, which means the deployment
was made immutable which eventually causes the test to fail. This only
occurs when the test is run by the root user because for non-root users
_ostree_linuxfs_fd_alter_immutable_flag() would silently fail and the
deployment would be mutable.

This commit fixes this issue by setting the environment variable in
tests/test-sysroot.js.

Closes: #1122
Approved by: cgwalters
2017-08-31 11:57:46 +00:00
Colin Walters
138c4d7aae libarchive: Add support for translating paths during commit
For rpm-ostree, I want to move RPM files in `/boot` to `/usr/lib/ostree-boot`.
This is currently impossible without forking the libarchive code.  Supporting
this is pretty straightforward; we already had pathname translation in
the libarchive code, we just need to expose it as an option.

On the command line side, I chose to wrap this as a regexp. That should be good
enough for a lot of use cases; sophisticated users should as always be making
use of the API. Note that this required some new `#ifdef LIBARCHIVE` bits to use
the new API. Following previous patterns here, we use the new API only if a
relevant option is enabled, ensuring unit test coverage of both paths.

For the test cases, I ended up changing the accounting to avoid having to
multiply the test count.

Closes: #1105
Approved by: jlebon
2017-08-30 14:30:30 +00:00
Simon McVittie
3fcd81df86 test-basic-user-only: Skip final step if no user xattrs
We don't want to skip the entire test, because the whole point of
bare-user-only is that it works in the absence of xattrs; but we do
need to skip this last stage, which explicitly uses a bare-user
repository.

Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1120
Approved by: cgwalters
2017-08-29 19:08:59 +00:00
Simon McVittie
41e6871e51 basic-test: Skip explicit uses of bare-user if no user xattrs
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1120
Approved by: cgwalters
2017-08-29 19:08:59 +00:00
Simon McVittie
a7686a0c40 libtest: Allow skipping single checks without user xattrs
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1120
Approved by: cgwalters
2017-08-29 19:08:59 +00:00
Simon McVittie
1d755977c8 test-pull-bareuser.sh: This test uses bare-user, hence needs xattrs
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1120
Approved by: cgwalters
2017-08-29 19:08:59 +00:00
Simon McVittie
760ff2f66e test-libarchive-import: Skip if extended attributes are unsupported
This is the case at build-time on some (all?) Debian autobuilders.

Signed-off-by: Simon McVittie <smcv@collabora.com>

Closes: #1120
Approved by: cgwalters
2017-08-29 19:08:59 +00:00
Colin Walters
5e30bbd1b9 bin/commit: Add --selinux-policy option
This was really straightforward to implement, and is useful
for dev/test scenarios mainly like we have in rpm-ostree at least.

Closes: https://github.com/ostreedev/ostree/issues/1113

Closes: #1114
Approved by: jlebon
2017-08-25 16:39:53 +00:00
Jonathan Lebon
9342be6e34 ostree-sysroot: make simple_write_deployment smarter
This is a follow-up to https://github.com/ostreedev/ostree/pull/1097.
We make simple_write_deployment smart enough so that it can be used for
rpm-ostree's purposes. This is mostly an upstreaming of logic that
already existed there.

Notably we correctly append NOT_DEFAULT deployments *after* the booted
deployment and we now support RETAIN_PENDING and RETAIN_ROLLBACK flags
to have more granularity on deployment pruning.

Expose these new flags on the CLI using new options (as well as expose
the previously existing NOT_DEFAULT flag as --not-as-default).

I couldn't add tests for --retain-pending because the merge deployment
is always the topmost one. Though I did check that it worked in a VM.

Closes: #1110
Approved by: cgwalters
2017-08-25 01:02:15 +00:00
Matthew Leeds
7ed881baa7 lib/repo-refs: Include remote refs when using collections
When working with collections it can be useful to see remote refs rather
than just local and mirrored ones. This commit changes the "ostree refs
-c" output to include remote refs, and includes remote refs with
collection IDs in summary file generation as well. The former behavior
is consistent with how "ostree refs" works, and the latter behavior is
useful in facilitating P2P updates even when mirrors haven't been
configured.

To accomplish this, OstreeRepoListRefsExtFlags was extended with an
EXCLUDE_REMOTES flag. This was done rather than an INCLUDE_REMOTES flag
so that existing calls to ostree_repo_list_refs_ext continue to have the
same behavior. This flag was added to ostree_repo_list_collection_refs
(which is an experimental API break).

Also, add unit tests for the "refs -c" and summary file behavior, and
update relevant tests.

Closes: #1069
Approved by: cgwalters
2017-08-24 19:57:33 +00:00
Colin Walters
eb6f7c6db8 lib/commit: Honor commit filter for libarchive --tar-autocreate-parents
This makes `ostree commit --tree=tar` honor `--owner-uid` and `--owner-gid`
for the root directory.

Prep for further commit filtering work, although mostly for the unit test cases;
this ensures we can use `ostree checkout` after autocreating a root directory.

Closes: #1104
Approved by: jlebon
2017-08-23 14:48:12 +00:00
Jonathan Lebon
19429b1a26 tests: fix admin upgrade timestamp file check
In #1055, I tried to be helpful but that didn't work out all too well.
We need to recompute the file path since one is in archive mode and the
other in bare mode.

Closes: #1098
Approved by: cgwalters
2017-08-21 19:30:58 +00:00
Colin Walters
618617d68b lib/pull: Add support for timestamp-check option, use in upgrader
For both flatpak and ostree-as-host, we really want to verify up front during
pulls that we're not being downgraded. Currently both flatpak and
`OstreeSysrootUpgrader` do this before deployments, but at that point we've
already downloaded all the data, which is annoying.

Closes: https://github.com/ostreedev/ostree/issues/687

Closes: #1055
Approved by: jlebon
2017-08-21 18:24:07 +00:00
Colin Walters
1e5b06be5b lib/deploy: Add .img to end of initramfs in /usr/lib/modules
Follow up to <https://github.com/ostreedev/ostree/pull/1079>; I was working on
the rpm-ostree updates for this, and I think it's more consistent if we have
`.img` here, since that's a closer match to the "remove $kver" that results in
`vmlinuz`. Also just best practice to have file suffix types where they make
sense.

The astute reader might notice this sneaks in a change where we'd crash if the
legacy bootdir didn't have an initramfs...yeah, should probably have test
coverage of that.

Closes: #1095
Approved by: jlebon
2017-08-21 16:41:50 +00:00
Colin Walters
3ab0d5e664 lib/sysroot: Support /usr/lib/modules/$kver for kernel/initramfs
This is the new Fedora kernel standard layout; it has the advantage
of being in `/usr` like `/usr/lib/ostree-boot`, but it's not OSTree
specific.

Further, I think in practice forcing tree builders to compute the checksum is an
annoying stumbling block; since we already switched to e.g. computing checksums
always when doing pulls, the cost of doing another checksum for the
kernel/initramfs is tiny. The "bootcsum" becomes more of an internal
implementation detail.

Now, there is a transition; my current thought for this is that rpm-ostree will
change to default to injecting into both `/usr/lib/ostree-boot` and
`/usr/lib/modules`, and stop doing `/boot`, then maybe next year say we drop the
`/usr/lib/ostree-boot` by default.

A twist here is that the default Fedora kernel RPM layout (and what's in
rpm-ostree today) includes a kernel but *not* an initramfs in
`/usr/lib/modules`. If we looked only there, we'd just find the kernel. So we
need to look in both, and then special case this - pick the legacy layout if we
have `/usr/lib/modules` but not an initramfs.

While here, rework the code to have an `OstreeKernelLayout` struct which makes
dealing with all of the variables nicer.

Closes: #1079
Approved by: jlebon
2017-08-18 17:34:36 +00:00
Colin Walters
395676917b Release 2017.10
Closes: #1089
Approved by: jlebon
2017-08-17 18:18:21 +00:00
Colin Walters
fd98bda3c7 repo: Introduce ostree_repo_open_at() and ostree_repo_create_at()
This essentially completes our fd-relative conversion.

While here, I cleaned up the semantics of `ostree_repo_create()` and
`ostree_repo_create_at()` to be more atomic - basically various scripts were
testing for the `objects` subdirectory, so let's formalize that.

Closes: #820
Approved by: jlebon
2017-08-15 12:35:10 +00:00
Jonathan Lebon
64b7c42025 pull: mark commits from local cache as partial
If one of the localcache repos has the exact same commit we resolved
from the remote, then we need to make sure to mark it as partial so that
we download the full tree.

Closes: #1074

Closes: #1076
Approved by: cgwalters
2017-08-14 12:04:42 +00:00
Colin Walters
f1102763df lib/sysroot-deploy: Refactor kernel layout parsing
I'd like to move the new canonical kernel directory to `/usr/lib/modules/$kver`,
as Fedora has done. The `get_kernel_from_tree()` function now abstracts over
parsing the data (src vs destination filenames, as well as checksum) in
preparation for adding the new case.

In preparation for this, let's change the current test suite to use the
*current* directory of `/usr/lib/ostree-boot`, and also add coverage of `/boot`.

Closes: #1053
Approved by: jlebon
2017-08-09 13:46:17 +00:00
Colin Walters
8642ef5ab3 lib/deploy: Use a FIFREEZE/FITHAW cycle for /boot
See: http://marc.info/?l=linux-fsdevel&m=149520244919284&w=2

XFS doesn't flush the journal on `syncfs()`. GRUB doesn't know how to follow the
XFS journal, so if the filesystem is in a dirty state (possible with xfs
`/boot`, extremely likely with `/`, if the journaled data includes content for
`/boot`, the system may be unbootable if a system crash occurs.

Fix this by doing a `FIFREEZE`+`FITHAW` cycle.  Now, most people
probably would have replaced the `syncfs()` invocation with those two
ioctls.  But this would have become (I believe) the *only* place in
libostree where we weren't safe against interruption.  The failure
mode would be ugly; nothing else would be able to write to the filesystem
until manual intervention.

The real fix here I think is to land an atomic `FIFREEZETHAW` ioctl
in the kernel.  I might try a patch.

In the meantime though, let's jump through some hoops and set up
a "watchdog" child process that acts as a fallback unfreezer.

Closes: https://github.com/ostreedev/ostree/issues/876

Closes: #1049
Approved by: jlebon
2017-08-08 16:09:04 +00:00
Philip Withnall
be4832242d lib/repo-pull: Fix counting of latest commits when finding repos
The intended behaviour of ostree_repo_find_remotes() is to return
results which have the latest version of at least one of the requested
refs. Results which have some of the requested refs, but don’t have the
latest version of any of them, should be ignored. The logic to do this
was broken in the case that a result contained a positive number of the
requested refs, but none of them were the latest version. (It previously
worked when the result contained none of the requested refs.)

Fix the counting to work correctly in both cases.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Colin Walters
48f8f32921 bin/pull: Fix @ override syntax when pulling multiple refs
Coverity spotted an infloop here since we were incrementing `i++`
instead of `j++`.  But adding a test revealed other bugs - we need
to keep the arrays in sync.

Coverity CID: 1452204

Closes: #1041
Approved by: pwithnall
2017-08-07 16:10:08 +00:00
Colin Walters
7f33d94dde tests/libtest-core.sh: Add a comment that this copy is canonical
Ref: https://github.com/projectatomic/bubblewrap/pull/203

Closes: #1047
Approved by: jlebon
2017-08-02 17:55:50 +00:00
Colin Walters
d5273b34d0 lib/repo: Add API to create and list ref aliases
There are multiple use cases where we'd like to alias refs.

First, having a "stable" alias which gets swapped across major
versions: https://pagure.io/atomic-wg/issue/228

Another case is when a ref is obsoleted;
<https://pagure.io/atomic-wg/issue/303>
This second one could be done with endoflife rebase, but I think
this case is better on the server side, as we might later change
our minds and do actual releases there.

I initially just added some test cases for symlinks in the `refs/heads` dir to
ensure this actually works (and it did), but I think it's worth having APIs.

Closes: #1033
Approved by: jlebon
2017-08-02 17:33:10 +00:00
Colin Walters
40b41d2cc9 tree-wide: Use GLib autocleanups for libarchive
Define typedefs for read/write archives, and use the GLib
autocleanups for them.   Prep for updating libglnx to drop its
custom autocleanup macros.

Closes: #1042
Approved by: jlebon
2017-08-02 16:55:38 +00:00
Colin Walters
fe8938227e tests/lzma: Fix off-by-one in buffer size
Coverity spotted that we had an off-by-one here since we were using
`i+1`.  Fix this by adding a `-1` to the bounds check.  Also use
`sizeof()` to ensure the data and size are coupled.

Coverity CID: 1452207

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters
5daae8629e tests/lzma: Use GRand over random()
It's designed for test suites and non-critical random uses like this. This
silences a Coverity warning about weak randomness.

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters
36ab8cd0e1 tests: Add some return value checking to pacify Coverity
No real problems here, but Coverity likes to see consistent checking of return
values, and I agree with it.

Coverity CID: 1452213
Coverity CID: 1452211

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters
fa3e07e5d7 Release 2017.9
Closes: #1029
Approved by: jlebon
2017-07-26 21:04:08 +00:00
Colin Walters
ed99b4169e tests: More fixes for gjs tests
Previous to this commit, the gjs tests were installed-only; and our
logic for handling the "--enable-installed-tests=exclusive" logic
actually also meant they weren't installed.

It did work for me locally with `--enable-installed-tests`.

However, to make things fully symmetric, let's enable the js tests to also be
run under `make check`.

Also remove `corrupt-repo-ref.js` from the PAPR invocation since it's not
actually a unit test, it's a utility helper.

Closes: #1022
Approved by: jlebon
2017-07-21 15:45:27 +00:00
Colin Walters
779f125cbe lib/repo: Auto-recreate repo/tmp if it's deleted
We can accumulate a lot of space there; let's be nice to people who delete the
whole directory.

Closes: https://github.com/ostreedev/ostree/issues/1018

Closes: #1020
Approved by: jlebon
2017-07-19 15:01:19 +00:00