Commit Graph

3231 Commits

Author SHA1 Message Date
Colin Walters
d2a92df155 repo/commit: Dedup content writing API implementation
Similar to metadata, for `write_content_trusted()` we can just
call `_write_content()` with a `NULL` output checksum.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters
22b1234f52 repo/commit: Dedup metadata writing API implementations
First, the streaming metadata API is pretty dumb, since metadata
should be small.  Really we should have supported a `GBytes`
version.  Currently, this API *is* used when we do local pulls,
so this commit has test coverage.  However, I plan to change
the object import to avoid using this.  But that's fine, since
I can't think of why someone would use this API.

Next, the only difference between `ostree_repo_write_metadata()` and
`ostree_repo_write_metadata_trusted()` is whether or not we pass
an output checksum; so just dedup the implementations.

Also while I'm here break out the input length validation and do
it early in the streaming case.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters
1eff3e8343 Add a notion of "physical" sysroot, use for remote writing
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).

In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.

Closes: https://github.com/ostreedev/ostree/issues/892

Closes: #896
Approved by: jlebon
2017-06-01 18:15:56 +00:00
Colin Walters
2414704609 lib/util: Some style conversion
I saw a few instances of `glnx_set_error_from_errno() + return FALSE`,
and fixed them and did a bit of style conversion.

Closes: #895
Approved by: jlebon
2017-05-30 20:33:28 +00:00
Colin Walters
9a3555a74b cmd: Use autoptr for GKeyFile
Prep for code style conversion.

Closes: #891
Approved by: jlebon
2017-05-29 14:09:08 +00:00
Colin Walters
88a1fc92a9 tree-wide: Add+run spatch to use glnx_throw()
I had to run a sed job to add whitespace after, but otherwise this was easy.

Closes: #890
Approved by: jlebon
2017-05-26 19:27:11 +00:00
Colin Walters
9bf8a8503a lib/sysroot: Add non-failable ostree_sysroot_repo()
Having a failable accessor is annoying, since it's really common
to reference both.  Instead, open the repo once when we load
the sysroot, and provide a non-failable accessor.

This is also prep for `ostree_repo_open_at()`, which collapses the separation
between `ostree_repo_new()` and `ostree_repo_open()`.

Closes: #886
Approved by: jlebon
2017-05-26 19:17:59 +00:00
Colin Walters
ed430b45de lib: Add an "is_system" member to OstreeRepo
This is prep for introducing a fd-relative `ostree_repo_new_at()`.
Previously, `ostree_repo_is_system()` compared `GFile` paths, but
there's a much simpler check we can do first - if this repository
was created via `OstreeSysroot`, it must be a system repo.

Closes: #886
Approved by: jlebon
2017-05-26 19:17:59 +00:00
Colin Walters
2f834968c6 lib/fsutil: Port to new code style
Pretty trivial.

Closes: #889
Approved by: jlebon
2017-05-26 19:05:53 +00:00
Colin Walters
ff2b881275 lib/fsutil: Delete unused GFile ioctl method
All the deployment code uses fds.

Closes: #889
Approved by: jlebon
2017-05-26 19:05:53 +00:00
Colin Walters
ad119aece9 pull-test: Add some 404 tests
See: https://github.com/flatpak/flatpak/issues/816

Closes: #887
Approved by: jlebon
2017-05-26 14:33:59 +00:00
Jonathan Lebon
07acb5b82c PAPR: migrate to the new name
The redhat-ci service has been renamed to PAPR. Previous values are
still supported though should be considered deprecated.

Closes: #885
Approved by: cgwalters
2017-05-25 16:31:08 +00:00
Colin Walters
822ade62c6 tests: Add some C tests for object writing
Prep for https://github.com/ostreedev/ostree/pull/881

Closes: #884
Approved by: jlebon
2017-05-25 14:24:05 +00:00
Colin Walters
c6960e63b2 lib/deploy: Port config merge logic to new code style
This is a de-scoping of work I did in preparation for
rpm-ostree [live updates](https://github.com/projectatomic/rpm-ostree/pull/652).
Originally I was going to expose this as a public API.

However, I decided to do things differently, but the cleanup here for new code
style and fd-relative is nice to have anyways.

We rework things to use `OstreeDeployment*`, which the caller is expected to
already have, rather than `GFile*`s pointing to the config directories.

Closes: #741
Approved by: jlebon
2017-05-24 16:31:55 +00:00
Alexander Larsson
be0c02d4f9 fetcher: Send Accept-Encoding: gzip when downloading summary
The summary file can get large, but it compresses well (something
which is not true of other files in the ostree repo which are
already compressed). By sending Accept-Encoding: gzip (and
handling the compressed results) we send a lot less data.

I set up the flathub repo (http://flathub.org/repo) to enable
gzip for the summary file (only), and the result is that the
331514 byte large summary was transferred in 122889 bytes.
On my (fast) network this decreased the time i took to do
"flatpak remote-ls flathub" by about 100msec.

This fixes https://github.com/ostreedev/ostree/issues/802

Closes: #882
Approved by: cgwalters
2017-05-24 13:52:17 +00:00
Philip Withnall
56188808b4 build: Use AM_TESTS_ENVIRONMENT rather than TESTS_ENVIRONMENT
TESTS_ENVIRONMENT is reserved for the user to be able to set when
running the tests. AM_TESTS_ENVIRONMENT is for the tests’ Makefile to
set itself.

https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #880
Approved by: cgwalters
2017-05-23 17:51:27 +00:00
Colin Walters
e99777e8d2 Add stub for new libglnx tmpfile API, port simpler callers to it
It's hard right now to do a full port to the new libglnx tmpfile
API since there are complex cases in the commit path which deal
with symlinks as well.

Let's make things more gradual by introducing the important part (struct with
autocleanup) here in libotutil, port what we can. This will make a future
complete port easier.

Closes: #871
Approved by: jlebon
2017-05-23 14:06:24 +00:00
Colin Walters
db00c9591f bin/cookies: Delete dead tmpfile code in cookie list command
This was a copy-paste-o.

Closes: #871
Approved by: jlebon
2017-05-23 14:06:24 +00:00
Daniel Drake
c1ed9a15c1 Allow commits to mark refs as EOL, replaced by others
A commit can now include a "ostree.endoflife-rebase" metadata key
pointing to a new ref.

When updating, the sysroot upgrader will see this and proceed to
pull and deploy the new ref instead. The origin file in the new
deployment will point to the new ref.

This functionality is planned to be used in Endless OS. We will create
a lesser tested branch for brand new, cutting edge hardware support,
and ship that on hardware platforms that require the latest drivers.
However, once our slower-moving official release is later updated to
support the new hardware, we will use this functionality to migrate
those bleeding-edge users over to the official release.

Closes: #874
Approved by: cgwalters
2017-05-22 20:18:07 +00:00
Daniel Drake
cd65f85dcb libtest: allow committing to alternative branches
This will be used by the upcoming test-admin-upgrade-endoflife.sh

Closes: #874
Approved by: cgwalters
2017-05-22 20:18:07 +00:00
Colin Walters
a2be46114a tests/libtest-core: Copy rpm-ostree changes, clean up
I want to keep this a "pure copy-able" file into various projects
like rpm-ostree, bwrap, and flatpak.  Pull in changes from rpm-ostree
to prep for that.

While we have the patient open, dedup the code for file matching a bit.

Closes: #877
Approved by: jlebon
2017-05-19 19:13:39 +00:00
Colin Walters
add88c3a23 ci: Add unit case for --enable-experimental-api
We're starting to get a lot of contexts, and this is likely going to drive some
requirements for consolidation and improvements like not testing *every* context
on every PR, etc.

But for now since experimental is new, and under development, let's test it.

Closes: #875
Approved by: cgwalters
2017-05-19 15:02:00 +00:00
Colin Walters
84d6267b61 tests/test-symbols.sh: Fix with --enable-experimental-api
We missed a `--no-filename` for grep with the documentation.

Closes: #875
Approved by: cgwalters
2017-05-19 15:02:00 +00:00
Philip Withnall
1feda846ef lib/remote: Fix compilation with --enable-experimental-api
The whole ostree-remote.h file is only included in the public ostree.h
header if OSTREE_ENABLE_EXPERIMENTAL_API is defined, so there’s no need
to change the set of methods defined in it according to whether we’re
compiling with experimental API.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:02:00 +00:00
Philip Withnall
242a0fd779 lib/repo: Make ost_repo_remove_remote() available internally
Make it an internal, not static, API; like _ostree_repo_add_remote(). It
will be used in many the same situations.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Philip Withnall
b6ac28b0da lib/repo: Add return value to _ostree_repo_add_remote()
Return whether the remote already existed. This is an internal API, so
it’s not an API break. The return value will be useful in upcoming
commits for working out whether to later remove a remote again.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Philip Withnall
ed7905d000 lib/remote: Add arguments to internal OstreeRemote constructor
Add a name argument to the internal OstreeRemote constructor,
since this member (and several derived from it) is non-nullable,
and hence must always be set at construction time.

This changes the only call sites of the constructor to use the new API,
which is internal.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Philip Withnall
2910b88081 lib/remote: Add internal annotations to OstreeRemote
Just for internal documentation; g-ir-scanner doesn’t read or understand
them.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Philip Withnall
d2eaded90d lib/remote: Add a getter for OstreeRemote.name
Now that we’ve got a public, sealed OstreeRemote structure, we can start
carefully exposing members of it as API.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Jonathan Lebon
a8fd37b6a0 pull: complete detached meta fetch before scanning
If somehow a repo has gpg verification on but doesn't have signatures
present for the existing commit, ostree would error out if it needs to
scan the commit object (e.g. if there are no updates available).

An instance of this is currently happening in Fedora AH, in which
signatures are not shipped in the ISO due to filesystem restrictions.
Another possible scenario is if a content provider switches from not
signing commits to signing them; even if older commits are retroactively
signed, clients' local commit objects would error out if they needed
scanning.

This patch adds a check to ensure that we always attempt to fetch the
detached metadata and wait for its result (whether it exists or not)
before moving on to scan their corresponding commit objects.

See also: https://github.com/projectatomic/rpm-ostree/issues/630

Closes: #873
Approved by: cgwalters
2017-05-18 01:14:15 +00:00
Colin Walters
88792f0f22 Release 2017.6
There's already a lot queued.  In particular this brings some API
additions that rpm-ostree depends on.

Closes: #865
Approved by: jlebon
2017-05-17 21:08:37 +00:00
Colin Walters
1470ff58b0 lib/pull: Port some functions to new code style
Porting a lot of this file would be hard since in many cases we do processing in
the `out:` section, so let's do what we can.

Closes: #870
Approved by: jlebon
2017-05-17 18:31:25 +00:00
Colin Walters
1a8f2f0769 switchroot/generator: Add var.mount to local-fs.target.requires
Unbreaks mounting in CentOS. Newer systemd in Fedora pulls didn't need this, I
think due to `RequiresMountsFor=`. Anyways, this is what the fstab generator
does, and it's clearly right ✓.

Closes: https://github.com/ostreedev/ostree/issues/867

Closes: #869
Approved by: jlebon
2017-05-17 13:53:54 +00:00
Colin Walters
28e3071256 build: Use cd $(srcdir) instead of git -C
Since the version in CentOS is too old, and we get a spam of warnings, plus
things like detecting the git repo break.

Fixes: 50f73cbac3

Closes: #868
Approved by: jlebon
2017-05-17 13:45:58 +00:00
Colin Walters
3ef2870700 lib/upgrader: Port to new code style
No surprises here.  Prep for future work.

Closes: #864
Approved by: jlebon
2017-05-17 13:37:18 +00:00
Colin Walters
9380dbb14d lib: Add "open dfd iter handling noent" helper, port tree-wide
Follow up to a previous patch that addressed a double-close; I
realized we already had a helper for doing "open dfd iter, do nothing
if we get ENOENT".  Raise it to libotuil, and port all consumers.

Closes: #863
Approved by: jlebon
2017-05-16 18:39:19 +00:00
Colin Walters
90cd7f7234 tree-wide: Add a few missing O_CLOEXEC
I noticed an instance of this while working on https://github.com/ostreedev/ostree/pull/861
Which apparently I cargo-culted into the new system generator bits.
Let's break this out as a small concise change.

Closes: #866
Approved by: jlebon
2017-05-16 17:34:46 +00:00
Colin Walters
30705889cb Switch to using a systemd generator for /var
If one wants to set up a mount for `/var` in `/etc/fstab`, it
won't be mounted since `ostree-prepare-root` set up a bind mount for
`/var` to `/sysroot/ostree/$stateroot/var`, and systemd will take
the already extant mount over what's in `/etc/fstab`.

There are a few options to fix this, but what I settled on is parsing
`/etc/fstab` in a generator (exactly like `systemd-fstab-generator` does),
except here we look for an explicit mount for `/var`, and if one *isn't* found,
synthesize the default ostree mount to the stateroot. Another nice property is
that if an admin creates a `var.mount` unit in `/etc` for example, that will
also override our mount.

Note that today ostree doesn't hard depend on systemd, so this behavior only
kicks in if we're built with systemd *and* libmount support (for parsing
`/etc/fstab`).  I didn't really test that case though.

Initially I started writing this as a "pure libc" program, but at one point
decided to use `libostree.so` to find the booted deployment. That didn't work
out because `/boot` wasn't necessarily mounted and hence we couldn't find the
bootloader config. A leftover artifact from this is that the generator code
calls into libostree via the "cmd private" infrastructure. But it's an easy way
to share code, and doesn't hurt.

Closes: #859
Approved by: jlebon
2017-05-16 16:13:05 +00:00
Colin Walters
d815ba2a81 switchroot/remount: Check mount status before remounting, be verbose
By checking the mount status, we avoid remounting things if we don't
need to.  And printing a single line per mount helps debugging when
things go wrong.

Closes: #859
Approved by: jlebon
2017-05-16 16:13:05 +00:00
Colin Walters
f21f500e40 switchroot/remount: Trim set of remounted filesystems
I really have no idea what I was thinking with that list of mount points. It
seems arbitrary. Sadly `git log` doesn't help, and there's no comments.

Basically, the only mounts we should care about are those that libostree
creates. Which are just `/sysroot` and `/var`. Systemd will handle the other
things like `/tmp`, it's not our job, and we shouldn't touch them.

Closes: #859
Approved by: jlebon
2017-05-16 16:13:05 +00:00
Colin Walters
19827a9965 lib/repo: Fix double close()
Should probably change `_take_fd()` to take a pointer and set to `-1`
at some point.

Regression from 8d58ab1002

Closes: #862
Approved by: jlebon
2017-05-16 14:01:24 +00:00
Colin Walters
a5eef45deb lib/remote: Box OstreeRemote if experimental-api
To avoid an introspection warning.  Otherwise, don't box it.

Closes: #858
Approved by: pwithnall
2017-05-16 08:28:47 +00:00
Colin Walters
05d0ee5cbe remount: Drop support for auto-tmpfs-on-var; use systemd.volatile=state
In current systemd, there is:
[systemd-volatile-root](https://www.freedesktop.org/software/systemd/man/systemd-volatile-root.service.html)
which was introduced by [this commit](91214a37ef).

I'd like to make further changes to how we handle `/var`, and I don't
want to reason about the interaction of our "tmpfs var" with too many
other things.

The comment about having "all /var handling in one place" was always inaccurate
given that we rely on systemd for mounting. And in general, I don't want to
duplicate too many things systemd does - it does them well, documents them, etc.

As far as I know, it was basically just Owen who was using this for the GNOME
hardware testing effort, and I'm sure he could easily switch over to
`systemd.volatile=state`.

Closes: #856
Approved by: owtaylor
2017-05-15 18:09:21 +00:00
Jonathan Lebon
23c60cda22 libglnx: bump and use new helper methods
Update submodule: libglnx

Closes: #857
Approved by: cgwalters
2017-05-12 21:02:16 +00:00
Jonathan Lebon
5811d4e8a3 tests/ci-commitmessage-submodules.sh: fix for RHCI
Special-case when this script is run under RHCI, which will try to fetch
the merge commit if possible. Use RHCI_COMMIT instead to refer to the
actual PR/branch HEAD being evaluated.

Use realpath to workaround the developer's git dir being in a symbolic
link.

Closes: #857
Approved by: cgwalters
2017-05-12 21:02:16 +00:00
Colin Walters
a195888b0f lib/checkout: Fix regression in subpath for regular files
This is what caused the merge of
https://github.com/projectatomic/rpm-ostree/pull/652
to blow up, since https://github.com/ostreedev/ostree/pull/848
landed right before we tried to merge it.

When I was writing that PR I remember having an uncertain feeling
since we were doing a `mkdirat` above, but at the time I thought
we'd have test suite coverage...turns out we didn't.

For backwards compatibility, we need to continue to do a `mkdirat` here of the
parent. However...I can't think of a reason anyone would *want* that behavior.
Hence, let's add a special trick - if the destination name is `.`, we skip
`mkdirat()`. That way rpm-ostree for example can open a dfd for `/etc` and avoid
the `mkdir`.

Fold the subpath tests into `test-basic.sh` since it's not worth a separate
file. Add a test case for checking out a file.

Closes: #854
Approved by: jlebon
2017-05-12 14:00:20 +00:00
Colin Walters
b83d509e78 tree-wide: Switch tabs ⭾ in various files over to spaces ␠
As $DEITY intended.

I was reading the `prepare-root.c` code and the indentation damage was
distracting. Squash tabs that have leaked into various places in the code. I
didn't yet touch the `src/libostree` bits as that has higher potential for
conflict.

Closes: #852
Approved by: jlebon
2017-05-11 18:17:26 +00:00
Colin Walters
ce4d21bc17 checkout: Plug a memleak of the state stringbuf
A struct without a cleanup macro is a struct likely to leak.

Closes: #850
Approved by: jlebon
2017-05-11 15:34:51 +00:00
Colin Walters
0177214982 lib/repo: Port more of GPG and summary functions to new code style
These ones were pretty easy, not sure why I didn't do them in an earlier pass.

Closes: #849
Approved by: jlebon
2017-05-11 15:26:49 +00:00
Colin Walters
964ca9d434 repo: Fix double close() in summary generation
Happened to notice this while doing a style port.

Closes: #849
Approved by: jlebon
2017-05-11 15:26:49 +00:00