Commit Graph

103 Commits

Author SHA1 Message Date
Colin Walters
a9a81f3a29 signing: Change API to create instances directly
This cleans up the verification code; it was weird how
we'd get the list of known names and then try to create
an instance from it (and throw an error if that failed, which
couldn't happen).
2020-05-10 14:18:55 +00:00
Denis Pynkin
ee5d0f350f signapi: expose metadata format and key
Explicitly expose functions for querying the metadata format
and key name used by OstreeSign object:
 - ostree_sign_metadata_format
 - ostree_sign_metadata_key

This allows to use the same metadata format and key name
by 3-rd party applications using signapi.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-04-24 16:34:50 +00:00
Denis Pynkin
b4050b4a34 lib/sign: make ed25519 engine non-public
Remove unneeded public declaration for ed25519 signing engine.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:55 +03:00
Denis Pynkin
4d0e3a66c5 lib/sign: make dummy engine non-public
Remove unneeded public declaration for dummy signing engine.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:55 +03:00
Denis Pynkin
0bdcf14d56 lib/sign: new function for summary file signing
Add function `ostree_sign_summary()` allowing to sign the summary file.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:55 +03:00
Denis Pynkin
908a2cd760 apidoc: add API documentation for signing interface
Add the documentation for all public functions.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
ceaf6d7f54 lib/sign: add ostree_seign_clear_keys function
Add the function for implicit cleanup of all loaded keys.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
bc4488692c lib/sign: public API optimisation
Removed from public `ostree_sign_detached_metadata_append` function.
Renamed `metadata_verify` into `data_verify` to fit to real
functionality.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
95ab57c17e lib/sign-ed25519: cleanup unneeded code
Removed unused code.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
2303202c86 sign: API changes for public keys and CLI keys format
API changes:
- added function `ostree_sign_add_pk()` for multiple public keys using.
- `ostree_sign_set_pk()` now substitutes all previously added keys.
- added function `ostree_sign_load_pk()` allowed to load keys from file.
- `ostree_sign_ed25519_load_pk()` able to load the raw keys list from file.
- use base64 encoded public and private ed25519 keys for CLI and keys file.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
edbbe1c4f2 lib/sign: initial implementation
Added the initial version of signing interface allowing to allowing to
sign and verify commits.
Implemented initial signing modules:
 - dummy -- simple module allowing to sign/verify with ASCII string
 - ed25519 -- module allowing to sign/verify commit with ed25519
   (EdDSA) signature scheme provided by libsodium library.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Colin Walters
b3bbbd1542 repo/commit: Add support for --selinux-policy-from-base
The [dev-overlay](332c6ab3b9/src/cmd-dev-overlay)
script shipped in coreos-assembler mostly exists to deal
with the nontrivial logic around SELinux policy.  Let's make
the use case of "commit some binaries overlaying a base tree, using
the base's selinux policy" just require a magical
`--selinux-policy-from-base` argument to `ostree commit`.

A new C API was added to implement this in the case of `--tree=ref`;
when the base directory is already checked out, we can just reuse
the existing logic that `--selinux-policy` was using.

Requires: https://github.com/ostreedev/ostree/pull/2039
2020-03-24 16:34:26 +00:00
Dan Nicholson
260bcd1193 core: Add ostree_commit_get_object_sizes API
This function parses the object listing in the `ostree.sizes` metadata
and returns an array of `OstreeCommitSizesEntry` structures.

Unfortunately, for reasons I don't understand, the linker wants to
resolve `_ostree_read_varuint64` from `ostree-core.c` even though it's
not used by `test-checksum.c` at all.
2020-01-20 20:46:29 -07:00
Dan Nicholson
fcbb453443 core: Add OstreeCommitSizesEntry type
This will be used when reading out entries in the `ostree.sizes`
metadata. Each entry corresponds to an object in the metadata array.
2020-01-20 20:44:12 -07:00
Colin Walters
5af403be0c Support mounting /sysroot (and /boot) read-only
We want to support extending the read-only state to cover `/sysroot`
and `/boot`, since conceptually all of the data there should only
be written via libostree.  Or at least for `/boot` should *mostly*
just be written by ostree.

This change needs to be opt-in though to avoid breaking anyone.

Add a `sysroot/readonly` key to the repository config which instructs
`ostree-remount.service` to ensure `/sysroot` is read-only.  This
requires a bit of a dance because `/sysroot` is actually the same
filesystem as `/`; so we make `/etc` a writable bind mount in this case.

We also need to handle `/var` in the "OSTree default" case of a bind
mount; the systemd generator now looks at the writability state of
`/sysroot` and uses that to determine whether it should have the
`var.mount` unit happen before or after `ostree-remount.service.`

Also add an API to instruct the libostree shared library
that the caller has created a new mount namespace.  This way
we can freely remount read-write.

This approach extends upon in a much better way previous work
we did to support remounting `/boot` read-write.

Closes: https://github.com/ostreedev/ostree/issues/1265
2019-12-11 15:33:57 +00:00
Philip Chimento
54639c03ca libostree: Add ostree_async_progress_copy_state()
This allows copying the state from one OstreeAsyncProgress object to
another, atomically, without invoking the callback. This is needed in
libflatpak, in order to chain OstreeAsyncProgress objects so that you
can still receive progress updates when iterating a different
GMainContext than the one that the OstreeAsyncProgress object was
created under.

See https://github.com/flatpak/flatpak/pull/3211 for the application of
this API.
2019-11-20 12:24:26 -08:00
Jason Wessel
b709c3c67b fsck: Implement a partial commit reason bitmask
After the corruption has been fixed with "ostree fsck -a --delete", a
second run of the "ostree fsck" command will print X partial commits
not verified and exit with a zero.

The zero exit code makes it hard to detect if a repair operation needs
to be run.  When ever fsck creates a partial commit it should add a
reason for the partial commit to the state file found in
state/<hash>.commitpartial.  This will allow a future execution of the
fsck to still return an error indicating that the repository is still
in the damaged state, awaiting repair.

Additional reason codes could be added in the future for why a partial
commit exists.

Text from: https://github.com/ostreedev/ostree/pull/1880
====
cgwalters commented:

To restate, the core issue is that it's valid to have partial commits
for reasons other than fsck pruned bad objects, and libostree doesn't
have a way to distinguish.

Another option perhaps is to write e.g. fsck-partial into the
statefile state/<hash>.commitpartial which would mean "partial, and
expected to exist but was pruned by fsck" and fsck would continue to
error out until the commit was re-pulled. Right now the partial stamp
file is empty, so it'd be fully compatible to write a rationale into
it.
====

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>

Closes: #1910
Approved by: cgwalters
2019-09-09 13:40:36 +00:00
Rafael Fonseca
a904f434cd lib/kargs: add missing function to symbol-versioning table
Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>

Closes: #1883
Approved by: jlebon
2019-07-02 13:58:26 +00:00
Allen Bai
be2572bf68 lib/kargs: Make API public and upstream new rpm-ostree APIs
This change makes public the current kargs API in src/libostree/ostree-kernel-args.c
and adds documentations.

Upstreams the new kargs API from rpm-ostree/src/libpriv/rpmostree-kargs-process.c

Merges libostree_kernel_args_la_SOURCES to libostree_1_la_SOURCES in Makefile-libostree.am

Upstreams tests/check/test-kargs.c from rpm-ostree.

Closes: #1833

Closes: #1869
Approved by: jlebon
2019-06-18 14:29:14 +00:00
Tristan Van Berkom
da4d0245af lib/commit: Added new ostree_repo_write_archive_to_mtree_from_fd()
Similar to ostree_repo_write_archive_to_mtree(), but takes
a file descriptor to read the archive from instead of mandating
a file path.

Usefull for importing archives into an OSTree repo over a socket
or from standard input in command line tools.

Closes: #1862
Approved by: jlebon
2019-05-28 17:03:22 +00:00
Jonathan Lebon
80435e12a7 Release 2019.2
Also add `ostree_repo_get_bootloader` to the public API, which was
missed when it was initially merged.

Closes: #1844
Approved by: rfairley
2019-04-25 16:47:40 +00:00
Matthew Leeds
ed41822b45 Rename core.repo-finders to core.default-repo-finders
This renames a config key to make its semantics more obvious. Despite
what the commit message says, it only applies when a set of repo finders
is not specified (either on the command line or in a library API call).
This also renames the corresponding ostree_repo_get function. We can do
this since it hasn't been released yet.

Closes: #1763
Approved by: pwithnall
2018-10-23 14:26:50 +00:00
Matthew Leeds
3956fc885b Allow disabling pulling from LAN/USB/Internet
Currently libostree essentially has two modes when it's pulling refs:
the "legacy" code paths pull only from the Internet, and the code paths
that are aware of collection IDs try to pull from the Internet, the
local network, and mounted filesystems (such as USB drives). The problem
is that while we eventually want to migrate everyone to using collection
IDs, we don't want to force checking LAN and USB sources if the user
just wants to pull from the Internet, since the LAN/USB code paths can
have privacy[1], security[2], and performance[3] implications.

So this commit implements a new repo config option called "repo-finders"
which can be configured to, for example, "config;lan;mount;" to check
all three sources or "config;mount;" to disable searching the LAN. The
set of values mirror those used for the --finders option of the
find-remotes command. This configuration affects pulls in three places:
1. the ostree_repo_find_remotes_async() API, regardless of whether or
not the user of the API provided a list of OstreeRepoFinders
2. the ostree_repo_finder_resolve_async() /
ostree_repo_finder_resolve_all_async() API
3. the find-remotes command

This feature is especially important right now since we soon want to
have Flathub publish a metadata key which will have Flatpak clients
update the remote config to add a collection ID.[4]

This effectively fixes https://github.com/flatpak/flatpak/issues/1863
but I'll patch Flatpak too, so it doesn't pass finders to libostree only
to then have them be removed.

[1] https://github.com/flatpak/flatpak/issues/1863#issuecomment-404128824
[2] https://github.com/ostreedev/ostree/issues/1527
[3] Based on how long the "ostree find-remotes" command takes to
  complete, having the LAN finder enabled slows down that step of the
  pull process by about 40%. See also
  https://github.com/flatpak/flatpak/issues/1862
[4] https://github.com/flathub/flathub/issues/676

Closes: #1758
Approved by: cgwalters
2018-10-21 19:11:43 +00:00
Matthew Leeds
fc84fb402c lib/repo: Define a metadata key, ostree.deploy-collection-id
This commit defines a metadata key that tells clients to update their
remote config to add a collection ID. This functionality is currently
implemented in Flatpak for the key "xa.collection-id", but there are two
good reasons for moving the key to OSTree:

1) Servers such as Flathub shouldn't set xa.collection-id in their
metadata now or in the medium term future, because many users are still
using old versions of Flatpak and OSTree[1] which would hit various
bugs[2][3][4] on the P2P code paths that are enabled by collection IDs.
Defining a new key means that only clients running recent
(as-yet-unreleased) versions of Flatpak and OSTree will pay attention to
it and deploy the collection ID, leaving the users on old versions
unaffected.

2) OSTree is as "invested" in collection IDs as Flatpak, so there's no
reason the key should be defined in Flatpak rather than here. According
to Philip Withnall, the reason the key was put in Flatpak originally was
that at the time there was uncertainty about tying OSTree to collection
IDs.

[1] https://ahayzen.com/direct/flathub.html#downloadsbyflatpakstacked
[2] https://github.com/ostreedev/ostree/commit/e4e6d85ea
[3] https://github.com/flatpak/flatpak/commit/5813639f
[4] https://github.com/flatpak/flatpak/commit/5b21a5b7

Closes: #1726
Approved by: pwithnall
2018-09-21 13:04:51 +00:00
Robert McQueen
b32c9e0df9 OstreeMutableTree: add _remove method
There is no API method to remove a file or subdirectory from a MutableTree
besides directly manipulating the GHashTable returned by _get_files or
_get_subdirs. This isn't possible from an introspection binding that transforms
the returned GHashTable, and may also leave the tree checksum in an invalid
state. Introduce a new method so that removing files or subdirectories is
safe, and possible from bindings.

Closes: #1724
Approved by: jlebon
2018-09-20 17:49:55 +00:00
Umang Jain
68420f70bb lib/repo: Add an API to get min-free-space-* reserved bytes
https://phabricator.endlessm.com/T23694

Closes: #1715
Approved by: cgwalters
2018-09-04 21:31:34 +00:00
William Manley
c7b12a8730 ostree repo commit: Speed up composing trees with --tree=ref
Running `ostree commit --tree=ref=a --tree=dir=b` involves reading the
whole of a into an `OstreeMutableTree` before composing `b` on top.  This
is inefficient if a is a complete rootfs and b is just touching one file.
We process O(size of a + size of b) directories rather than
O(number of touched dirs).

This commit makes `ostree commit` more efficient at composing multiple
directories together.  With `ostree_mutable_tree_fill_empty_from_dirtree`
we create a lazy `OstreeMutableTree` which only reads the underlying
information from disk when needed.  We don't need to read all the
subdirectories just to get the checksum of a tree already checked into the
repo.

This provides great speedups when composing a rootfs out of multiple other
rootfs as we do in our build system.  We compose multiple containers
together with:

    ostree commit --tree=ref=base-rootfs --tree=ref=container1 --tree=ref=container2

and it is much faster now.

As a test I ran

    time ostree --repo=... commit --orphan --tree=ref=big-rootfs --tree=dir=modified_etc

Where modified_etc contained a modified sudoers file under /etc.  I used
`strace` to count syscalls and I seperatly took timing measurements.  To
test with a cold cache I ran

    sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Results:

|                      | Before | After |
| -------------------- | ------ | ----- |
| Time (cold cache)    |   8.1s | 0.12s |
| Time (warm cache)    |   3.7s | 0.08s |
| `openat` calls       |  53589 |   246 |
| `fgetxattr` calls    |  78916 |     0 |

I'm not sure if this will have some negative interaction with the
`_ostree_repo_commit_modifier_apply` which is short-circuited here.  I
think it was disabled for `--tree=ref=` anyway, but I'm not certain.  All
the tests pass anyway.

I originally implemented this in terms of the `OstreeRepoFile` APIs, but
it was *way* less efficient, opening and reading many files unnecessarily.

Closes: #1643
Approved by: cgwalters
2018-07-09 13:10:51 +00:00
Matthew Leeds
8fbf19c9f5 Make P2P API public (no longer experimental)
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.

This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak.  However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.

In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:10 +00:00
Colin Walters
9131d8a4cc lib/sysroot: Add wrapper API to prune system repository
The initial motivation for this is that the "staging" code currently
didn't rewrite the deployment refs, meaning that the staged commit
could be pruned.

Hence first, this new API ensures that deployments also
hold a strong ref to their commit, without relying on the magical
"deployment refs" that we inject.  That has always been a weird
artifact of the strict layering separation between OstreeSysroot
and OstreeRepo.

I also plan to change rpm-ostree to start using this API to
hold references to base layers for client-side layering; it also
today generates various refs.

That said, if we still want to support multiple processes
writing to a single repo (as happens on EndlessOS today) we
still need to write refs; perhaps later we could add a concept
of "generators" or something that create refs based on whatever
logic?

Another minor thing this fixes is that we had a printf inside
the library; this propagates the pruned data to the higher level
which can log however it likes.

Closes: #1566
Approved by: jlebon
2018-05-24 12:56:11 +00:00
Colin Walters
371081d123 lib: Add a public helper method for pruning to find all ref'd commits
Prep for reworking how we do sysroot cleanup.  We're going to
start doing more lowlevel pruning work there, and I wanted to avoid
duplicating the ref enumeration.

Closes: #1566
Approved by: jlebon
2018-05-24 12:56:11 +00:00
Colin Walters
8c1542134c lib/repo: Enable locking by default, but drop external API
The code has been sitting around for a while but since I disabled
it by default, I doubt anyone is really using it or relying on it.

This patch and turns on locking by default, and also drops the
API which was only public in the experimental API builds.
Conceptually these are two distinct things, and we
may actually want to split up the patches.

I don't think this will break anyone, but it's hard to say for sure.
It's also going to be hard to find out until we actually release
I suspect...

But anyone who is broken should be able to add `locking=false` into
their repo config.  On the flip side Endless has been shipping with
this enabled and it is reported to help.

The reason to drop the APIs: I'm a bit concerned about the interactions over time
between libostree's use of the API and any apps that start using it.
For example, if an app specifies a SHARED lock in their code, then
later internally we decide to temporarily grab an `EXCLUSIVE`, but the
app had a second thread/process that was `EXCLUSIVE` already, and
that process was waiting on the first bit of code, then we could
deadlock. I can't think of a real world situation where this would happen
yet though.

We are likely to in the future have say `fsck` take an external lock,
`checkout` grab a shared one, etc.

Closes: #1555
Approved by: jlebon
2018-04-30 17:24:51 +00:00
Alexander Larsson
f258e9e5ff lib/repo: Add ostree_repo_traverse_commit_union_with_parents
This is a version of ostree_repo_traverse_commit_union that also
remembers where the objects came from, by recording the parent
relationships in a hashtable. This can be used to later find which
commits each object was from, which we want to use in fsck.

Closes: #1533
Approved by: cgwalters
2018-04-14 15:36:21 +00:00
Colin Walters
eb506c759c Add concept of "staged" deployment
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.

This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem.  It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297

We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.

Closes: https://github.com/ostreedev/ostree/issues/545

Closes: #1503
Approved by: jlebon
2018-04-12 14:55:12 +00:00
Colin Walters
7f88fddcd4 sysroot: Add concept of deployment "pinning" 📌
Example user story: Jane rebases her OS to a new major version N, and wants to
keep around N-1 even after a few upgrades for a while so she can easily roll
back. I plan to add `rpm-ostree rebase --pin` to opt-in to this for example.

Builds on the new `libostree-transient` group to store pinning state there.

Closes: https://github.com/ostreedev/ostree/issues/1460

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Colin Walters
c40a47e965 sysroot: Add API to clean up transient keys in origin files
The `origin/unlocked` and `origin/override-commit` keys are examples of state
that's really transient; we don't want to maintain them across upgrades. Right
now there are bits for this in both `ostree admin upgrade` as well as in
rpm-ostree.

This new API will slightly clean up both cases, but it's really prep for adding
a concept of deployment "pinning" that will live in the new
`libostree-transient` group.

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Colin Walters
0041a7a1ed core: Add API (and standard concept for) content checksum
There are a few cases for knowing whether a commit has identical
content to another commit.  Some people want to do a "promotion workflow",
where the content of a commit on a tesitng branch is then "promoted"
to a production branch with `ostree commit --tree=ref`.

Another use case I just hit in rpm-ostree deals with
[jigdo](https://github.com/projectatomic/rpm-ostree/issues/1081) where we're
importing RPMs on both the client and server, and will be using the
content checksum, since the client/server cases inject different metadata
into the commit object.

Closes: https://github.com/ostreedev/ostree/issues/1315

Closes: #1449
Approved by: jlebon
2018-02-12 19:03:18 +00:00
Marcus Folkesson
6bf4b3e1d8 Add SPDX-License-Identifier to source files
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.

Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
  text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
  environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
  license for their code is respected

See http://spdx.org for further reading.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>

Closes: #1439
Approved by: cgwalters
2018-01-30 20:03:42 +00:00
Colin Walters
4a2e08148d lib/core: Add a "break hardlink" API
This imports the code from rpm-ostree:
9ff9f6c997/src/libpriv/rpmostree-util.c (L742)

I plan to use this for rofiles-fuse to implement
copyup: https://github.com/ostreedev/ostree/issues/1377

But it's just obviously generally useful for projects using
libostree I think.

Closes: #1378
Approved by: jlebon
2017-12-14 21:56:26 +00:00
Colin Walters
7935b881bf lib/repo: Add an API to mark a commit as partial
For the [rpm-ostree jigdo ♲📦](https://github.com/projectatomic/rpm-ostree/issues/1081) work.
We're basically doing "pull" via a non-libostree mechanism, and this
should be fully supported.  As I mentioned earlier we should try to
have `ostree-repo-pull.c` only use public APIs; this gets us closer
to that.

Closes: #1376
Approved by: jlebon
2017-12-14 15:51:07 +00:00
Philip Withnall
1b7d83114e lib/pull: Split verify_bindings() out into a cmdprivate method
It will be used by the fsck utility in future. We could expose it
publicly in future too, if needed.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Colin Walters
73d910e82e Add public API for fsck, use it before loading metadata
A while ago I did `truncate -s 0 /path/to/repo/00/123.commit`, and expected a
checksum error, but I actually got a validation error due to us loading the
commit into a variant and trying to parse out the parent checksum, etc.

I first started by changing the `load_and_fsck_one_object()` function to
checksum before loading, but the problem is that we do a traverse of all objects
first. Fixing this is going to require an `OSTREE_REPO_COMMIT_TRAVER_FLAG_FSCK`
or something.

In the meantime at least though, let's add a public API to fsck a single object
which *does* checksum cleanly before parsing the object, and change the `fsck`
command to use it.

We then change the fsck binary to do this while iterating over the refs
and finding the commit object.  This way we'll at least get a checksum
first for commit objects, even if not dirtree/dirmeta.

Closes: #1364
Approved by: jlebon
2017-12-12 14:03:09 +00:00
Dan Nicholson
7d863ed9e4 lib/repo: Add locking auto cleanup handler
Define an auto cleanup handler for use with repo locking. This is based
on the existing auto transaction cleanup. A wrapper for
ostree_repo_lock_push() is added with it. The intended usage is like so:

  g_autoptr(OstreeRepoAutoLock) lock = NULL;
  lock = ostree_repo_auto_lock_push (repo, lock_type, cancellable, error);
  if (!lock)
    return FALSE;

The functions and type are marked to be skipped by introspection since I
can't see them being usable from bindings.

Closes: #1343
Approved by: cgwalters
2017-12-05 02:32:47 +00:00
Dan Nicholson
4e78ddd2da lib/repo: Add repo locking mechanism
Currently ostree has no method of guarding against concurrent pruning.
When there are multiple repo writers, it's possible to have a pull or
commit race against a prune and end up with missing objects.

This adds a file based repo locking mechanism. The intention is to take
a shared lock when writing objects and an exclusive lock when deleting
them. In order to make use of the locking throughout the library in a
fine grained fashion, the lock acts recursively with a stack of lock
states. If the lock becomes exclusive, it will stay in that state until
the stack is unwound past the initial exclusive push. The file locking
is similar to GLnxLockFile in that it uses open file descriptor locks
but falls back to flock when needed.

The lock also attempts to be thread safe by storing the lock state in
thread local storage with GPrivate. This means that each thread will
have an independent lock for each repository it opens. There are some
drawbacks to that, but it seemed impossible to manage the lock state
coherently in the face of multithreaded access.

The API is a push/pop interface in accordance with the recursive nature
of the locking. The push interface uses an enum that's translated to
LOCK_SH or LOCK_EX as needed. Both interfaces use an internal timeout
field to decide whether to manage the lock in a blocking or non-blocking
fashion. The intention is to allow ostree applications as well as
administrators to control this timeout. For now, the default is a 30
second timeout.

Note that the timeout is handled synchronously in thread since the lock
is maintained in thread local storage. I.e., the thread that acquires
the lock needs to be the same thread that runs the operation. There may
be a way to offer an asynchronous version, but it's not clear exactly
how that would work since it would likely involve a separate thread that
invokes a callback when the locking operation completes.

https://bugzilla.gnome.org/show_bug.cgi?id=759442

Closes: #1343
Approved by: cgwalters
2017-12-05 02:32:47 +00:00
Joaquim Rocha
a1745e1a79 lib/remote: Add a method to return the URL
When using dynamic remotes (LAN and USB), we cannot use their name with
the common remote related ops (ostree_repo_remote_...) because ostree
doesn't keep this type of remotes in its internal hash table.
Unfortunately this means that we cannot access the URL of those remotes
either (in order to e.g. set the right URL for those remotes in
Flatpak).

Since the URL is actually stored in a key file that belongs to the
OstreeRemote, then we can simply allow users access to it through a
getter.

So this patch adds a method that allows to return the URL directly from
the OstreeRemote without having to go through the OstreeRepo.

The test-repo-finder-config is also updated by this patch to check if
the URL is correct.

Closes: #1353
Approved by: cgwalters
2017-11-28 18:53:25 +00:00
Dan Nicholson
a256b2d1a3 lib/remote: Export ostree_remote_get_type symbol
Without this, you can't really use OstreeRemote as a GObject, which is a
requirement for bindings.

This was found when attempting to include OstreeRemote in the GIR, and
g-ir-scanner wasn't able to link it's temporary object due to an
"undefined reference to `ostree_remote_get_type'" error.

Closes: #1337
Approved by: pwithnall
2017-11-10 10:03:44 +00:00
Colin Walters
1222c2271b repo: Add wrapper function for setting devino cache on checkout opts
I was trying to use this with pygobject for an OCI+ostree project, and pygobject
rejected simply assigning to the field (understandably, since it can't bind the
lifetime together).

Add a wrapper function, which is still unsafe, but hides that unsafety
where most people shouldn't find it.  And if they do...well, sorry,
Rust wasn't invented when ostree was started.

Closes: #1295
Approved by: pwithnall
2017-10-20 18:20:19 +00:00
Philip Withnall
2531d8fe63 lib/repo-finder: Add OstreeRepoFinderOverride
This is another OstreeRepoFinder implementation; it returns results from
a given set of URIs. It’s designed to be used for implementing user
overrides to other repo-finders, or for implementing unit tests.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1281
Approved by: mwleeds
2017-10-19 19:11:58 +00:00
Jonathan Lebon
077d2718ad lib/core: add ostree_checksum_file_at API
This is like `ostree_checksum_file` but fd-relative. This will be used
by https://github.com/ostreedev/ostree/pull/1258.

AFAICT, we actually didn't have any tests that check the `checksum` CLI.
Add a basic one here to test the old code as well as the new code.

Closes: #1263
Approved by: cgwalters
2017-10-12 12:53:01 +00:00
Philip Withnall
149aec1099 lib/repo-refs: Add first version of ostree_repo_resolve_collection_ref()
This is a parallel for ostree_repo_resolve_rev_ext() which works on
collection–refs. At the moment, the implementation is simple and uses
ostree_repo_list_collection_refs(). In future, it could be rewritten to
check the checksum directly rather than enumerating all
potentially-relevant checksums.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1182
Approved by: cgwalters
2017-09-27 14:44:00 +00:00
Philip Withnall
64b23fd089 lib/repo: Add ostree_repo_hash() and tests
Add a hash function for OstreeRepo instances, which relies on the repo
being open, and hence being able to hash the device and inode of its
root directory.

Add unit tests for this and ostree_repo_equal().

Signed-off-by: Philip Withnall <withnall@endlessm.com>

https://github.com/ostreedev/ostree/issues/1191

Closes: #1205
Approved by: cgwalters
2017-09-21 21:25:58 +00:00