Commit Graph

3523 Commits

Author SHA1 Message Date
Anton Gerasimov
db10bf464f Add support for pkcs11 URIs in TLS client key/certificate
Closes: #1183
Approved by: cgwalters
2017-09-19 20:30:22 +00:00
Colin Walters
5c4f26bd65 lib/pull: Wait for pending ops to complete on error
I saw in a stack trace that the main thread was calling `exit()` even while
worker threads were alive and doing sha256/write/fsync etc. for objects.

The stack trace was a SEGV as the main thread was calling into library
`atexit()` handlers and we were a liblz4 destructor:

```
 #0  0x00007f2db790f8d4 _fini (liblz4.so.1)
 #1  0x00007f2dbbae1c68 __run_exit_handlers (libc.so.6)
```

(Why that library has a destructor I don't know offhand, can't find
 it in the source in a quick look)

Anyways, global library destructors and worker threads continuing simply don't
mix. Let's wait for our outstanding operations before we exit. This is also a
good idea for projects using libostree as a shared library, as we don't want
worker threads outliving operations.

Our existing pull corruption tests exercise coverage here.

I added a new `caught-error` status boolean to the progress API, and use it the
commandline to tell the user that we're waiting for outstanding ops.

Closes: #1185
Approved by: jlebon
2017-09-19 19:05:26 +00:00
Colin Walters
3e564116b2 lib/repo: Minor cleanup to object import function
We have a lot of layers of abstraction here; let's fold in the `trusted`
conditional into the call, since that's all the public API we're using does
anyways.

Prep for a future patch around object copying during imports.

Closes: #1187
Approved by: jlebon
2017-09-19 18:51:03 +00:00
Simon McVittie
223c940b46 tests: Explicitly unset LANGUAGE after setting LC_ALL
As a GNU extension, LANGUAGE takes precedence over LC_ALL for
gettext(3) whenever the locale is not C, causing tests that grep for
specific English strings to fail when run in non-English locales.
The upstream glibc proposal for C.UTF-8 would give C.UTF-8 the same
special case as C here, but the implementation in Debian does not
currently have this, so we have to unset LANGUAGE too.

Signed-off-by: Simon McVittie <smcv@collabora.com>

Closes: #1188
Approved by: jlebon
2017-09-19 15:16:09 +00:00
Colin Walters
13c3898cc2 tree-wide: Some glnx_fstatat_allow_noent() porting
The new API is definitely nicer.

Closes: #1180
Approved by: jlebon
2017-09-19 15:03:05 +00:00
Philip Withnall
c62b36046b lib/repo: Drop outdated FIXME comment from ostree-repo.h
This was some incomplete planning from while the find_remotes() API was
being designed; now totally outdated.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Philip Withnall
15247641d9 lib/repo-finder-mount: Change the schema for finding repos on volumes
See issue #1174 for the rationale behind this. In summary:
 • It required two lists of collection–refs to be maintained: one in the
   repository, and one pointing to the repository.
 • It didn’t automatically work for live USBs of OSs based on OSTree
   (where there’s always a repository at /ostree/repo).
 • It was unnecessarily complex.

The new scheme allows a list of repositories to be searched, but without
needing a layer of indirection through their collection–refs. It adds
/ostree/repo and /.ostree/repo as well-known repository locations which
are always checked on a mounted volume (if they exist).

Update the unit tests accordingly.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

https://github.com/ostreedev/ostree/issues/1174

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Philip Withnall
981eb6c226 lib/repo: Add ostree_repo_equal() for comparing repos
This will compare their root directory inodes to see if they are the
same repository on disk. A convenience method for the users of the
public API who can’t access OstreeRepo.inode.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Colin Walters
1036cc8084 ci: Use distro-sync instead of upgrade
This should help ensure that if installing `glibc-devel` downgrades `glibc`
we'll run through our hackround for
<https://bugzilla.redhat.com/show_bug.cgi?id=1394862>.

Closes: #1190
Approved by: jlebon
2017-09-19 14:29:47 +00:00
Colin Walters
ab200495a8 bin/prune: Add --only-branch
In 5c940987e7 / #646 we
added `--retain-branch-depth`; this adds a symmetric
`--only-branch` for the case where a repo owner just
wants to prune a specific branch.

The implementation here is pretty straightforward; we
just walk all refs and inject the equivalent of
`--retain-branch-depth=$ref=-1` if they're *not* in
`--only-branch`.

Closes: https://github.com/ostreedev/ostree/issues/1115

Closes: #1127
Approved by: jlebon
2017-09-18 17:20:38 +00:00
Colin Walters
58e4abe4bc tests/prune: Factor out a helper for counting commits in the repo
We had lots of duplicates; prep for adding more tests.

Closes: #1127
Approved by: jlebon
2017-09-18 17:20:38 +00:00
Colin Walters
d0b0578cc1 Update libglnx
Update libglnx, which is mostly port the repo stagedir code
to the new tmpdir API.  This turned out to require some
libglnx changes to support de-allocating the tmpdir ref while
still maintaining the on-disk dir.

Update submodule: libglnx

Closes: #1172
Approved by: jlebon
2017-09-18 17:09:34 +00:00
yceh
b39a61b493 bin/commit: Fix build failure when libarchive is not used
A `"}"` at line 641 is missing when `HAVE_LIBARCHIVE` is not defined
(even though probably few will use ostree without libarchive).

Closes: #1181
Approved by: jlebon
2017-09-18 13:15:20 +00:00
Simon McVittie
2f22e5b359 test-help.sh: Skip trivial-httpd if enabled
Because it runs a binary in ${libexecdir}, it will only work if
libostree was already installed, which makes the build-time test fail.
It also doesn't produce the output we expect: its usage mechanism
mentions "ostree-trivial-httpd", not "ostree trivial-httpd".

Signed-off-by: Simon McVittie <smcv@collabora.com>

Closes: #1177
Approved by: jlebon
2017-09-15 19:28:25 +00:00
Simon McVittie
1b430a7764 tests: Fix JavaScript tests with gjs 1.50.0
In recent gjs, you can't declare a variable with "let" multiple times.

Signed-off-by: Simon McVittie <smcv@collabora.com>

Closes: #1178
Approved by: cgwalters
2017-09-15 18:51:43 +00:00
Colin Walters
0488b4870e lib/pull: Drop partial fetch code from libsoup backend
Doing this in prep for libglnx tmpdir porting, but I think we should also do
this because the partial fetch code IMO was never fully baked; among other
things it was never integrated into the scheme we came up with for "boot id
sync" that we use for complete/staged objects.

There's a lot of complexity here that while we have some coverage for, I think
we need to refocus on the core functionality. The libcurl backend doesn't have
an equivalent to this today.

In particular for small objects, this is simply overly complex. The downside is
clearly for large objects like FAH's 61MB initramfs; not being able to resume
fetches of those is unfortunate.

In practice though, I think most people should be using deltas, and we need to
make sure deltas work for large objects anyways.

Further ultimately the peer-to-peer work should help a lot for people
with truly unreliable connections.

Closes: #1176
Approved by: jlebon
2017-09-15 17:01:51 +00:00
Colin Walters
c32f234e9a lib/checkout: Do UNION_FILES via atomic renameat()
I was looking at fixing an `rpm-ostree livefs` bug where we need to replace
`/usr/lib/passwd`. It's obviously bad if that temporarily disappears 😉. My plan
is to do a subpath checkout of just `/usr/lib/{passwd,group}`.

Make this atomic (i.e. file always exists) by changing the logic to create a
temporary link in repo/tmp, then rename() it into place.

A bonus here is we kill one of the very few (only?) non-error-cleanup i.e.
"non-linear" `goto`s in the ostree codebase.

Closes: #1171
Approved by: jlebon
2017-09-15 16:44:00 +00:00
Colin Walters
7499620254 lib/repo: Port gpg signing function to new code style
We already had all of the autocleanups ready for this.

Closes: #1164
Approved by: jlebon
2017-09-15 01:43:16 +00:00
Jonathan Lebon
73fa43abc7 build-sys: Post-release version bump
Closes: #1175
Approved by: cgwalters
2017-09-14 20:29:33 +00:00
Colin Walters
6c0738a000 Release 2017.11
Closes: #1173
Approved by: jlebon
2017-09-14 15:04:42 +00:00
Colin Walters
1c2d344074 tests: Port some bits of C to new style
Where we can; perhaps after updating libglnx we should use the
new test error macro?

Closes: #1169
Approved by: jlebon
2017-09-13 19:32:36 +00:00
Colin Walters
051cdf396c lib/checkout: Rename disjoint union, change to merge identical files
It turns out that librpm automatically merges identical files between
distinct packages, and this occurs in practice with Fedora today between
`chkconfig` and `initscripts` for exmaple.

Since we added this for rpm-ostree, we basically want to do what librpm does,
let's change the semantics to do a merge.  While we're here rename
to `UNION_IDENTICAL`.

Closes: #1156
Approved by: jlebon
2017-09-13 19:19:33 +00:00
Colin Walters
8d3752a0d6 lib/repo: Port tmpdir locking func to new style
Prep for future work.

Closes: #1168
Approved by: jlebon
2017-09-13 19:02:31 +00:00
Colin Walters
93038bcf71 build-sys: Add -Werror=switch
We use the "exhaustive enum" pattern (i.e. no `default:`) in some places so
we're forced to touch all users when adding cases.

Closes: #1167
Approved by: peterbaouoft
2017-09-13 17:06:45 +00:00
Dan Nicholson
3b315e16d8 repo: Ensure new config doesn't set remotes in separate file
If the new configuration passed to ostree_write_config () tries to
update options for a remote defined in a separate config file, return an
error. Without this, the full configuration would contain duplicate
remote specifications, which would raise an error the next time the repo
is opened.

Closes: #1159
Approved by: cgwalters
2017-09-13 16:03:25 +00:00
Jonathan Lebon
4cc813133c bin/remote: don't load repo on root command
Subcommands will demand a repo argument themselves. This allows one to
call `ostree remote` and get the "No subcommand" error rather than the
"Missing --repo" error.

Closes: #1126
Approved by: cgwalters
2017-09-13 14:32:20 +00:00
Jonathan Lebon
4c02fc2daa bin/admin: Don't load sysroot for root commands
There's no need to load the sysroot for root commands which have
subcommands, like `ostree admin` and `ostree admin instutil`. Otherwise,
even just calling them without arguments will cause a failure. The
subcommands will have the appropriate flags set as needed.

Closes: #1126
Approved by: cgwalters
2017-09-13 14:32:20 +00:00
Jonathan Lebon
225bbdf002 bin/static-delta: Convert to new style and tweak output
Convert the whole file to new style. Also tweak the help outputs to make
it similar enough to the other commands for tests to pass. Of course, we
should just centralize all subcommand handling the same way it was done
in rpm-ostree, though let's punt on that for now.

Closes: #1126
Approved by: cgwalters
2017-09-13 14:32:20 +00:00
Jonathan Lebon
4efab3feb0 bin/admin: Don't require root for instutil
Otherwise, we can't even do `--help` on it. The subcommands all already
have the root flag set.

Closes: #1126
Approved by: cgwalters
2017-09-13 14:32:20 +00:00
Jonathan Lebon
077de8ea46 tests/test-help.sh: Rework and strengthen checks
The `sed` expression wasn't actually matching the main output, so we
weren't recursing into the subcommands. Update the syntax to match the
current output and add a check so we don't miss that happening again.

Add a check that the help output is only printed once in all
circumstances. Also add a check for proper handling of non-existent
commands.

Closes: #1126
Approved by: cgwalters
2017-09-13 14:32:20 +00:00
Jonathan Lebon
ec9a58f247 bin/config: Tweak parameter string
Make the parameter string more detailed.

Closes: #1126
Approved by: cgwalters
2017-09-13 14:32:19 +00:00
Jonathan Lebon
48364459b8 bin/main: Print usage when no command given
Minor regression from https://github.com/ostreedev/ostree/pull/1106. We
want to print the usage text both when unknown commands are passed, as
well as when no commands are passed at all.

Closes: #1126
Approved by: cgwalters
2017-09-13 14:32:19 +00:00
Colin Walters
662ad5b171 lib/sysroot: Use direct g_mkdtemp() for overlay tmpdir
The new libglnx `glnx_mkdtempat()` uses autocleanups, which
is inconvenient for this use case where we *don't* want autocleanups.
Since we don't need it to be fd-relative, just directly invoke
`g_mkdtemp_full()` which is fine for this use case.

Prep for updating libglnx.

Closes: #1161
Approved by: jlebon
2017-09-12 20:09:12 +00:00
Colin Walters
9689fb720a ci: Fixate CentOS container image until rpm-md repos sync
Copy of https://github.com/projectatomic/rpm-ostree/pull/985

Closes: #1160
Approved by: jlebon
2017-09-12 18:03:58 +00:00
Dan Nicholson
adac42b6ef repo: Add add-remotes-config-dir option
This option allows a repo to explicitly opt out of adding new remotes in
a remotes configuration directory. This currently defaults to true for
system repos and false for non-system repos to maintain legacy behavior
that non-system repos don't add remotes in a configuration directory.
That would be problematic for flatpak, which specifies a remotes config
dir but adds remotes in ways that are incompatible with it.

So, what this really does is allow system repos to control whether they
want to add remotes in the config dir or not. That's important if your
flatpak repo is the system repo like at Endless.

Closes: #1134

Closes: #1155
Approved by: cgwalters
2017-09-11 10:53:20 +00:00
Dan Nicholson
ccbbf77c71 sysroot: Reload config after setting sysroot kind
This allows any repo configuration defaults to be set based on whether
it's a system repo or not.

Closes: #1155
Approved by: cgwalters
2017-09-11 10:53:20 +00:00
Matthew Leeds
9f78386819 lib/repo: Update outdated comment
Closes: #1157
Approved by: cgwalters
2017-09-09 10:47:07 +00:00
Alexander Larsson
08eaf66827 rofiles-fuse: Fix lchown() and hardlink verification for symlinks
If you lchown("symlink") then we were incorrectly trying to chown the
symlink target, rather than the symlink itself. In particular, this cause
cp -a to fail for a broken symlink. Additionally, it was using the
symlink target when verifying writability, rather than the symlink
itself.

To fix this, we need pass AT_SYMLINK_NOFOLLOW in these cases.

In general, the kernel itself will always resolve any symlinks for us
before calling into the fuse backend, so we should really never do any
symlink following in the fuse fs itself. So, we pro-actively add
NOFOLLOW flags to a few other places:

 truncate:
      In reality this will never be hit, because
      the kernel will resolve symlinks before calling us.
 access:
      It seems the current fuse implementation never calls this
      (faccessat w/AT_SYMLINK_NOFOLLOW never reaches the fuse fs)
      but if this ever is implemented this is the correct behaviour.

We would ideally do `chmod` but this is not implemented on current kernels.
Because we're not multi-threaded, this is OK anyways.

Further, our write verification wasn't correctly handling the case of hardlinked
symlinks, which can occur for `bare` checkouts but *not* `bare-user` which the
tests were using. Change to `bare` mode to verify that.

Closes: #1137
Approved by: alexlarsson
2017-09-08 20:38:39 +00:00
Colin Walters
067da211cd lib/syslinux: Port to new code style
There was only one tricky bit here around the ownership of the lines; I made use
of `g_steal_pointer()` to consistently track ownership, and converted to a `for`
loop while still preserving the loop logic around the last entry.

Closes: #1154
Approved by: jlebon
2017-09-08 18:00:19 +00:00
Colin Walters
3594bb2d0f lib: Add a private helper to abort txns, use in sysroot cleanup
Steal some code from flatpak for this, which allows porting a few more things to
new style. I started on a public API version of this but was trying to roll some
other things into it and it snowballed. Let's do this version since it's easy
for now.

While here I changed things so that `generate_deployment_refs()` now just uses
`_set_ref_immediate()` rather than requring a txn.

Also, AFAICS there was no test coverage of `generate_deployment_refs()`; I tried
commenting it out and at least `admin-test.sh` passed. Add some coverage of this
- I verified that with this commenting out bits of that function cause the test
to fail.

Closes: #1132
Approved by: jlebon
2017-09-08 16:25:06 +00:00
Colin Walters
6be4dfe66e lib/grub2: Port some to new code style
I resisted trying to do anything invasive here like fd-relative porting as our
coverage is weak. But this was all straightforward porting to decl-after-stmt
style.

Closes: #1153
Approved by: jlebon
2017-09-08 16:07:18 +00:00
Dan Nicholson
43c78c9006 repo: Fix non-system remotes-config-dir usage
Before commit e0346c1, a non-system repo could specify
remotes-config-dir and have remotes read from there. However, adding
remotes would only be done in the config dir for a system repo. Restore
that by respecting remotes-config-dir when no sysroot is found and
adding back the ostree_repo_is_system() check when adding remotes.

Closes: #1133

Closes: #1151
Approved by: cgwalters
2017-09-08 13:54:30 +00:00
Guy Shapiro
2a7fdfdbc5 uboot: add non-default deployments to uEnv.txt
Include non-default deployments in the uEnv.txt file imported by
U-Boot. All the configurations beside the defaults will have
numerical suffix E.G. "kernel_image2" or "bootargs2".
Those U-Boot environment variables may be used from interactive boot
prompt or from "altbootcmd" script.

Closes: #1138
Approved by: cgwalters
2017-09-08 00:58:08 +00:00
Guy Shapiro
a567b5b47b uboot: move system uEnv merge to new function, clean up
Split the code that merge the system uEnv to new function. While we're here,
clean up the logic to e.g. use `ot_openat_ignore_enoent()`.

Closes: #1138
Approved by: cgwalters
2017-09-08 00:58:08 +00:00
Colin Walters
c7d0be4fba tree-wide: Add error prefixing for most remaining syscalls
There were some important ones there like a random `syncfs()`. The remaining
users are mostly blocked on the "fstatat enoent" case, I'll wait to port those.

Closes: #1150
Approved by: jlebon
2017-09-07 22:31:16 +00:00
Colin Walters
7afa966198 lib/sysroot: Use fd-relative acccess for bootversion cleanup
I noticed this was an easy change.

Closes: #1148
Approved by: peterbaouoft
2017-09-07 20:13:24 +00:00
Robert McQueen
59dff7175e lib/gpg: Provide the public key to the duplicate check
Add keys from the signing homedir to the GpgVerifier used to look
for duplicate signatures. This will allow signatures from subkeys
to be canonicalised and recognised as already signed despite the
differing key ID, avoiding duplicate signatures.

Closes: https://github.com/ostreedev/ostree/issues/608

Closes: #1092
Approved by: cgwalters
2017-09-07 19:56:31 +00:00
Robert McQueen
6b6408a7d0 lib/gpg: Correct missing line prefix with bad signatures
In the case the signature time was bad, a line prefix was missing from the
result of `ostree_gpg_verify_result_describe_variant()`.

Closes: #1092
Approved by: cgwalters
2017-09-07 19:56:31 +00:00
Robert McQueen
2d854368a8 lib/gpg: Add _FINGERPRINT_PRIMARY to OstreeGpgVerifyResult
Revert the switch of _FINGERPRINT to giving the primary key ID
rather than the signing key ID, and instead add the primary
key ID as a new attribute which is available if the key is not
missing.

Closes: https://github.com/ostreedev/ostree/issues/608

Closes: #1092
Approved by: cgwalters
2017-09-07 19:56:31 +00:00
Matthew Leeds
1e3f87c34c tests: Check "refs -c PREFIX" behavior
This commit adds tests to check the behavior of "refs -c PREFIX", where
prefix is interpreted as a collection ID.

Closes: #1149
Approved by: cgwalters
2017-09-07 18:49:55 +00:00