Commit Graph

171 Commits

Author SHA1 Message Date
Colin Walters
ce36e05d38
Merge pull request #2434 from cgwalters/custom-remote
Add support for "custom remotes"
2021-09-08 09:27:26 -04:00
Colin Walters
511c7a13ee Add support for "custom remotes"
This will be helpful for the "ostree native container" work in
https://github.com/ostreedev/ostree-rs-ext/

Basically in order to reuse GPG/signapi verification, we need
to support adding a remote, even though it can't be used via
`ostree pull`.  (At least, not until we merge ostree-rs-ext into ostree, but
 even then I think the principle stands)
2021-09-08 07:42:25 -04:00
Colin Walters
8821ec6e56 upgrade: Stabilize deployment staging
We're waaay overdue for this, it's been the default
in rpm-ostree for years, and solves several important bugs
around not capturing `/etc` while things are running.

Also, `ostree admin upgrade --stage` (should) become idempotent.

Closes: https://github.com/ostreedev/ostree/issues/2389
2021-09-07 16:12:43 -04:00
Dan Nicholson
58a683f8f0 bin/remote: Rename list-gpg-keys to gpg-list-keys
As pointed out in the original review, `gpg-list-keys` fits better
alongside the existing `gpg-import`.

Changes were done with:

```
git grep -l list-gpg-keys | xargs sed -i 's/list-gpg-keys/gpg-list-keys/'
for src in $(git ls-files '*list-gpg-keys*'); do
  dst=${src/list-gpg-keys/gpg-list-keys}
  git mv "$src" "$dst"
done
```
2021-08-23 15:30:31 -06:00
Colin Walters
98f3fe3d8e
Merge pull request #2401 from dbnicholson/gpg-key-info
Remote GPG key info
2021-08-20 15:11:11 -04:00
Dan Nicholson
74fb0c5f78 bin/remote: Add list-gpg-keys subcommand
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating that GPG key updates have been
applied. Tests are added, which checks the
`ostree_repo_remote_get_gpg_keys` API by extension.
2021-07-15 15:50:04 -06:00
Benjamin Gilbert
5b5d3b12fb man: improve statoverride description
A statoverride file written in the obvious way will produce incorrect
results for two independent reasons.  Document them.
2021-07-14 16:34:35 -04:00
Timothée Ravier
02527f115e *: rename master to main in tests & examples 2021-05-07 16:55:03 +02:00
Colin Walters
857587615d Add an API+CLI to inject metadata for bootable OSTree commits
I was doing some rpm-ostree work and I wanted to compare two
OSTree commits to see if the kernel has changed.  I think
this should be a lot more natural.

Add `ostree commit --bootable` which calls into a new generic
library API `ostree_commit_metadata_for_bootable()` that
discovers the kernel version and injects it as an `ostree.linux`
metadata key.  And for extra clarity, add an `ostree.bootable`
key.

It's interesting because the "core" OSTree layer is all about
generic files, but this is adding special APIs around bootable
OSTree commits (as opposed to e.g. flatpak as well as
things like rpm-ostree's pkgcache refs).

Eventually, I'd like to ensure everyone is using this and
hard require this metadata key for the `ostree admin deploy`
flow - mainly to prevent accidents.
2021-03-12 19:01:42 +00:00
Phaedrus Leeds
d3e40ca7f6 man: Add missing repo mode in config docs 2021-03-08 13:18:31 -08:00
Dan Nicholson
4db2ba0eb1 pull: Allow disabling commit binding verification
In some cases such as backups or mirroring you may want to pull commits
from one repo to another even if there commits that have incorrect
bindings. Fixing the commits in the source repository to have correct
bindings may not be feasible, so provide a pull option to disable
verification.

For Endless we have several repositories that predate collection IDs and
ref bindings. Later these repositories gained collection IDs to support
the features they provide and ref bindings as the ostree tooling was
upgraded. These repositories contain released commits that were valid to
the clients they were targeting at the time. Correcting the bindings is
not really an option as it would mean invalidating the repository
history.
2020-12-17 14:07:08 -07:00
William Manley
31acd2ef99 Add support for explicitly requesting any specific bootloader type
...with the `sysroot.bootloader` configuration option.  This can be useful
when converting a system to use `ostree` which doesn't currently have a
bootloader configuration that `ostree` can automatically detect, and is
also useful in combination with the `--sysroot` option when provisioning a
rootfs for systems other than the one you're running `ostree admin deploy`
on.
2020-10-26 23:51:11 +00:00
Alexander Larsson
024ef1d756 deltas: Add and document no-deltas-in-summary config option
By default this is FALSE to keep existing clients working.
2020-10-23 12:30:08 +02:00
OpenShift Merge Robot
9f98aa9223
Merge pull request #2198 from cgwalters/no-hardlink-zerosize
checkout: Don't hardlink zero sized files
2020-10-05 22:06:38 +02:00
Colin Walters
558720e7aa checkout: Don't hardlink zero sized files
Alternative to https://github.com/ostreedev/ostree/pull/2197

Python's (usually) zero-sized `__init__.py` files can provoke
us hitting the hardlink limits on some filesystems (`EMLINK`).
At least one Fedora rpm-ostree user hit this.

The benefits of hardlinking here are quite marginal; lots
of hardlinks can behave suboptimally in particular filesystems
like BTRFS too.

This builds on prior code which made this an option, introduced
in 673cacd633
Now we just do it uncondtionally.

Also this provoked a different bug in a very obscure user mode checkout
case; when the "real" permissions were different from the "physical"
permissions, we would still hardlink.  Fix the test case for this.
2020-10-01 16:47:07 -04:00
Frédéric Danis
2e97f5659f bin/static-delta: Add signature parameters to apply-offline
This allows to check the delta signature before applying it.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
512db0435c bin/static-delta: Add command to verify delta signature
Add new "static-delta verify" sub-command.
This supports multiple keys to verify the static-delta file.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Frédéric Danis
92efbc00d8 bin/static-delta: Add support to sign superblock
Add signing ability to "static-delta generate" builtin.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
2020-09-14 09:27:19 +02:00
Stefan Agner
b94c3ae79f man: add missing options to the ostree-commit man page
Add missing parameter to the ostree-commit man page.

Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
2020-07-31 11:54:19 +02:00
Stefan Agner
999f9a2b2d man: add glossary to main man page
Add glossary to define some commonly used literals throughout the ostree
man pages.

Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
2020-07-31 11:49:38 +02:00
Colin Walters
a615d35762 pull: Add --per-object-fsync
This is the opposite of
https://github.com/ostreedev/ostree/issues/1184

Motivated by OpenShift seeing etcd performance issues during
OS updates: https://github.com/openshift/machine-config-operator/issues/1897

Basically, if we switch to invoking `fsync()` as we go, it makes
ostree performance worse (in my tests, 31s to write 2G versus 7s if we
delay sync) but it avoids *huge* outliers in `fsync()` time for etcd.
2020-07-18 14:59:01 +00:00
Stefan Agner
38aa912a0e man/checkout: document missing options
Document missing options in the ostree checkout man page.

Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
2020-06-08 18:23:10 +02:00
Stefan Agner
843482e589 man/checkout: fix short name option of --user-mode
The short name option of --user-mode is -U.

Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
2020-06-08 15:43:26 +02:00
Colin Walters
329a82c57e commit: Add --base argument
I was trying to followup the `--selinux-policy-from-base` work
to add a `cosa build --fast=overlay` for coreos-assembler,
but hit on the fact that using e.g. `--owner-uid` disables
commit optimizations.

A while ago, https://github.com/ostreedev/ostree/pull/1643 landed
which optimized this for the case where no modifications are provided.
But, we really need the SELinux policy bits, and it's super convenient
to run `ostree commit` as non-root.

It's fairly surprising actually that it's taken us so long to
iterate on a good interface for this "commit changes on top of a base"
model.  In practice, many nontrivial cases really end up needing
to do a (hardlink) checkout, and that case is optimized.

But for this coreos-assembler work I want to directly overlay onto
a commit object another commit object.

That previous PR above added exactly the API we need, so let's
expose it in the CLI.

What you can see happening in the test is that we provide
`--owner-uid 42`, but that only applies to directories/files
that were added in the commit.

And now that I look at this, I think what we really want here
is to avoid changing directories that exist in the base, but
eh; in practice the main use here is for `--owner-uid 0` while
committing as non-root; and that works fine with this since
the baseline uid will be zero as well.
2020-04-06 19:28:51 +00:00
OpenShift Merge Robot
a16fe86b36
Merge pull request #1878 from d4s/wip/d4s/no_gpg
Alternative signing system
2020-04-04 19:46:12 +02:00
Matthew Leeds
cd37293b5a find-remotes: Add a --mirror option
This will be useful in the unit test added by the next commit. It just
passes OSTREE_REPO_PULL_FLAGS_MIRROR to the call to
ostree_repo_pull_from_remotes_async().
2020-03-28 10:35:19 -07:00
Denis Pynkin
65c16a8318 man: add signature options for ostree summary
Add a description of new options `--sign-type` and `--sign` for
`ostree summary` command.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:55 +03:00
Denis Pynkin
e799186658 man: document commit signing
Added options descriptions for `ostree-commit` allowing
to sign the commit.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Denis Pynkin
0c89055b19 man: document ostree sign
Add man page for `ostree sign`.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
2020-03-25 15:23:54 +03:00
Jonathan Lebon
f7a8617c01 bin/diff: Clarify documentation around REV and DIR syntax
Related: #2032
2020-03-18 10:32:52 -04:00
clime
8e65366ffd Update ostree-pull.xml with info about pulled refs location and access 2020-02-19 20:12:23 +01:00
Dan Nicholson
97c831dd5f bin/show: Add --print-sizes option to show sizes metadata
Use the new `ostree_commit_get_object_sizes()` API to read the
`ostree.sizes` commit metadata and print a summary.
2020-01-20 20:46:31 -07:00
Dan Nicholson
d916383953 bin/refs: Allow overwriting existing ref
Currently if you want to update a non-alias ref, you need to first check
if it exists and use either `ostree refs --create` or `ostree reset` as
appropriate. That's unnecessarily complicated and is much less
convenient than the old `write-refs` builtin that simply called
`ostree_repo_set_ref_immediate()` without any checks.

Add a `--force` option to be used with `--create` that does not raise an
error when the destination ref already exists.

Closes: #1870
Approved by: jlebon
2019-06-12 15:00:19 +00:00
Tristan Van Berkom
1c7bddd57a man: Document ability to commit archives from stdin
Closes: #1862
Approved by: jlebon
2019-05-28 17:03:22 +00:00
Robert Fairley
21ebc7d21e Add sysroot.bootloader repo config key
The sysroot.bootloader key configures the bootloader
that OSTree uses when deploying a sysroot. Having this key
allows specifying behavior not to use the default bootloader
backend code, which is preferable when creating a first
deployment from the sysroot (#1774).

As of now, the key can take the values "auto" or "none". If
the key is not given, the value defaults to "auto".

"auto" causes _ostree_sysroot_query_bootloader() to be used
when writing a new deployment, which is the original behavior
that dynamically detects which bootloader to use.

"none" avoids querying the bootloader dynamically. The BLS
config fragments are still written to
sysroot/boot/loader/entries for use by higher-level software.

More values can be supported in future to specify a single
bootloader, different behavior for the bootloader code, or
a list of bootloaders to try.

Resolves: #1774

Closes: #1814
Approved by: jlebon
2019-03-01 21:20:35 +00:00
Matthew Leeds
656853a98a man/ostree-config: Consistently use GROUPNAME placeholder
It doesn't make much sense to use SECTIONNAME in some places and
GROUPNAME in others when they mean the same thing.

Closes: #1743
Approved by: cgwalters
2019-03-01 16:29:44 +00:00
Matthew Leeds
eecd989d46 ostree/config: Add an "unset" operation
Currently there's a way to set a key to the empty string but there's no
way to unset it completely (remove the key from the group). This might
be helpful for instance if you want to temporarily set
"core.lock-timeout-secs" to a specific value for the duration of one
operation and then return it to the default after that operation
completes.

This commit implements an "unset" operation for the config command, adds
a unit test, and updates the man page.

Closes: #1743
Approved by: cgwalters
2019-03-01 16:29:44 +00:00
Matthew Leeds
f5c86c834b man: Document ostree --version
Closes: #1820
Approved by: rfairley
2019-02-14 19:52:13 +00:00
Dan Nicholson
b33a4e9b1e remote-add: Add --force option to add or replace remote
This uses the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to add a
remote or replace an existing one. This is roughly the opposite of
--if-not-exists and will raise an error if both options are passed.

Closes: #1166
Approved by: cgwalters
2019-02-08 14:36:41 +00:00
Dan Nicholson
8431bb5406 man: Document remote add --if-not-exists option
Closes: #1166
Approved by: cgwalters
2019-02-08 14:36:41 +00:00
Umang Jain
c25b1c7063 man/ostree.repo-config: Update min-free-space-* docs
Docs should reflect the changes done in PR:
https://github.com/ostreedev/ostree/pull/1779

Closes: #1786
Approved by: jlebon
2018-12-12 14:25:16 +00:00
rfairley
05e8c7ef6a lib/repo: Search a list of paths in gpgkeypath for gpg keys
This allows specifying gpgpath as list of
paths that can point to a file or a directory. If a directory path
is given, paths to all regular files in the directory are added
to the remote as gpg ascii keys. If the path is not a directory,
the file is directly added (whether regular file, empty - errors
will be reported later when verifying gpg keys e.g. when pulling).

Adding the gpgkeypath property looks like:

ostree --repo=repo remote add --set=gpgpath="/path/key1.asc,/path/keys.d" R1 https://example.com/some/remote/ostree/repo

Closes #773

Closes: #1773
Approved by: cgwalters
2018-11-21 17:03:10 +00:00
Matthew Leeds
ed41822b45 Rename core.repo-finders to core.default-repo-finders
This renames a config key to make its semantics more obvious. Despite
what the commit message says, it only applies when a set of repo finders
is not specified (either on the command line or in a library API call).
This also renames the corresponding ostree_repo_get function. We can do
this since it hasn't been released yet.

Closes: #1763
Approved by: pwithnall
2018-10-23 14:26:50 +00:00
Matthew Leeds
1d6347fe97 lib/repo-pull: Disable LAN updates by default
This commit disables searching on the local network for refs, unless
explicitly requested by the user either by changing the value of the
"core.repo-finders" config option, or by passing an OstreeRepoFinderAvahi to
ostree_repo_find_remotes_async() / ostree_repo_finder_resolve_async(),
or by specifying "lan" in the --finders option of the find-remotes
command.

The primary reason for this is that ostree_repo_find_remotes_async()
takes about 40% longer to complete with the LAN finder enabled, and that
API is used widely (e.g. in every flatpak operation). It's also probable
that some users don't want ostree doing potentially unexpected traffic
on the local network, even though everything pulled from a peer is GPG
verified.

Flathub will soon deploy collection IDs to everyone[1] so these code
paths will soon see a lot more use and that's why this change is being
made now.

Endless is the only potential user of the LAN updates feature, and we
can revert this patch on our fork of ostree. For it to be used outside
Endless OS we will need to upstream eos-updater-avahi and
eos-update-server into ostree.

[1] https://github.com/flathub/flathub/issues/676

Closes: #1758
Approved by: cgwalters
2018-10-21 19:11:43 +00:00
Matthew Leeds
3956fc885b Allow disabling pulling from LAN/USB/Internet
Currently libostree essentially has two modes when it's pulling refs:
the "legacy" code paths pull only from the Internet, and the code paths
that are aware of collection IDs try to pull from the Internet, the
local network, and mounted filesystems (such as USB drives). The problem
is that while we eventually want to migrate everyone to using collection
IDs, we don't want to force checking LAN and USB sources if the user
just wants to pull from the Internet, since the LAN/USB code paths can
have privacy[1], security[2], and performance[3] implications.

So this commit implements a new repo config option called "repo-finders"
which can be configured to, for example, "config;lan;mount;" to check
all three sources or "config;mount;" to disable searching the LAN. The
set of values mirror those used for the --finders option of the
find-remotes command. This configuration affects pulls in three places:
1. the ostree_repo_find_remotes_async() API, regardless of whether or
not the user of the API provided a list of OstreeRepoFinders
2. the ostree_repo_finder_resolve_async() /
ostree_repo_finder_resolve_all_async() API
3. the find-remotes command

This feature is especially important right now since we soon want to
have Flathub publish a metadata key which will have Flatpak clients
update the remote config to add a collection ID.[4]

This effectively fixes https://github.com/flatpak/flatpak/issues/1863
but I'll patch Flatpak too, so it doesn't pass finders to libostree only
to then have them be removed.

[1] https://github.com/flatpak/flatpak/issues/1863#issuecomment-404128824
[2] https://github.com/ostreedev/ostree/issues/1527
[3] Based on how long the "ostree find-remotes" command takes to
  complete, having the LAN finder enabled slows down that step of the
  pull process by about 40%. See also
  https://github.com/flatpak/flatpak/issues/1862
[4] https://github.com/flathub/flathub/issues/676

Closes: #1758
Approved by: cgwalters
2018-10-21 19:11:43 +00:00
Matthew Leeds
039bbe5682 man/create-usb: Don't recommend summary updates
This commit removes the recommendation in the create-usb man page for
the user to update the summary in the source repo before using the
create-usb command. I'm not sure where I got the idea that create-usb
depends on a summary in the source repo. I went back to the first commit
that introduced the create-usb command and even using that a summary
isn't required, so it seems unlikely that this changed recently.

This is good news because the exclusive lock that's taken for summary
updates has been causing problems on Endless (due to other processes
having a lock for the duration of the 30 second acquire time out
period).

Closes: #1746
Approved by: cgwalters
2018-10-02 15:49:42 +00:00
Matthew Leeds
1a7536c0f9 man/ostree.repo-config: Document locking options
This commit documents the "locking" and "lock-timeout-secs" options
which have been around for a few releases.

Closes: #1737
Approved by: jlebon
2018-09-28 15:49:11 +00:00
Matthew Leeds
9a06c5409e create-usb: Add a --commit option
Currently on Endless OS, the OSTree ref for the operating system is
something like os/eos/amd64/eos3, so that's what gets passed to `ostree
create-usb` when copying the OS to a USB drive (for offline updates).
However, when eos-updater checks for updates it pulls the metadata for a
candidate commit and in so doing updates that eos3 ref to point to the
partial commit being examined as a potential update rather than the
deployed commit. This causes `ostree create-usb` to fail with an error
like "No such metadata object
7fb045cb2d1f1f3a81bfc157c6128ff443eb56350315b9536bdb56aee0659863.dirtree".

OSTree creates deployment refs that look like "ostree/1/1/0" to maintain
a pointer to the deployed commit, but create-usb can't use these because
it shows up in the summary as just a ref, not a collection-ref.

So this commit adds a --commit option to the create-usb command, so we
can use the appropriate ref but copy the deployed commit rather than a
(potentially partial) update commit.

Closes: #1735
Approved by: cgwalters
2018-09-25 15:47:25 +00:00
Jonathan Lebon
d786b9fd5f man/config: Keep cmdsynopsis for GROUP.KEY version
It's nice to still show the previous usage since it's easier to type in
trivial cases like `core.mode`.

Closes: #1710
Approved by: sinnykumari
2018-08-21 10:36:12 +00:00
Sinny Kumari
dde3f1c0fb src/ostree: Add --group option to ostree config
Fetching value from a repo config using 'ostree config
get SECTIONNAME.KEYNAME' didn't work in some cases like
when having dots in Group Name entry.
As per Desktop entry file specification, Group Name
may contain all ASCII characters except for [ and ]
and control characters.
Link - https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-1.1.html

Having --group option will help user to clearly specify
Group Name and get desired result.

It also adds test for ostree config get|set and bash
completion for --group option

Fixes https://github.com/ostreedev/ostree/issues/1565

Closes: #1696
Approved by: cgwalters
2018-08-20 14:31:15 +00:00
Will Thompson
ada5716fb2 man/ostree.repo-config: improve min-free-space-* docs
0 is not a valid value (the units are required), and 1MB is taken to
mean 1048576 bytes, not 1000000 bytes (unlike other human-readable sizes
in libostree, which are formatted by g_format_size(), and hence use
power-of-10 units). It's probably a bit late to change this latter
point, but the documentation should mention it.

Define 'MB' etc. more precisely; include the byte counts in the
examples; and improve the formatting of the min-free-space-* paragraphs.

Signed-off-by: Will Thompson <wjt@endlessm.com>

Closes: #1706
Approved by: mwleeds
2018-08-19 01:43:05 +00:00
Jonathan Lebon
72a54fa877 lib/config: Deprecate commit-update-summary option
Now that we have `auto-update-summary`, there is no point in having
`commit-update-summary`. The latter also only had an effect through
the `commit` CLI command, whereas the former is embedded directly in
libostree.

There is one corner case that slips through: `commit` would update the
summary file even if orphan commits were created, which we no longer do
here. I can't imagine anyone relying on this, so it seems safe to drop.

Closes: #1689

Closes: #1693
Approved by: mwleeds
2018-08-01 19:59:07 +00:00
Jonathan Lebon
786ee6bdec lib/config: Rename change-update-summary to auto-...
Mildly bikeshed, though I find the name `auto-update-summary` to be
easier to grok than `change-update-summary`. I think it's because it can
be read as "verb-verb-noun" rather than "noun-verb-noun".

Closes: #1693
Approved by: mwleeds
2018-08-01 19:59:07 +00:00
Matthew Leeds
6869bada49 config: Add a core/change-update-summary option
This commits adds and implements a boolean repo config option called
"change-update-summary" which updates the summary file every time a ref
changes (additions, updates, and deletions).

The main impetus for this feature is that the `ostree create-usb` and
`flatpak create-usb` commands depend on the repo summary being up to
date. On the command line you can work around this by asking the user to
run `ostree summary --update` but in the case of GNOME Software calling
out to `flatpak create-usb` this wouldn't work because it's running as a
user and the repo is owned by root. That strategy also means flatpak
can't update the repo metadata refs for fear of invalidating the
summary.

Another use case for this relates to LAN updates. Specifically, the
component of eos-updater that generates DNS-SD records advertising ostree
refs depends on the repo summary being up to date.

Since ostree_repo_regenerate_summary() now takes an exclusive lock, this
should be safe to enable. However it's not enabled by default because of
the performance cost, and because it's more useful on clients than
servers (which likely have another mechanism for updating the summary).

Fixes https://github.com/ostreedev/ostree/issues/1664

Closes: #1681
Approved by: jlebon
2018-07-30 17:19:12 +00:00
Matthew Leeds
799cfcb5b3 man/ostree.repo-config: Update min-free-space-* docs
Now that it's possible to have both min-free-space-size and
min-free-space-percent set in a repo config, update the docs to make
the behavior clear in that case.

Closes: #1687
Approved by: jlebon
2018-07-25 13:21:30 +00:00
Umang Jain
66079c7b65 lib/repo: Allow min-free-space-size and -percent to co-exist
Previously, we would error out if both of the options were mentioned
in the config file (even if one of them is disabled with 0). There
were few suggestions that this behavior was not quite right.

Therefore, instead of throwing error and exiting, it's preferred to
warn the user. Hence, the solution that worked out is:
* Allow both options to exist simulateneously
* Check each config's value and decide:
  * If both are present and are non-zero, warn the user. Also, prefer
    to use min-free-space-size over the another.
  * If both are absent, then use -percent=3% as fallback
  * Every other case is valid hence, no warning

https://phabricator.endlessm.com/T13698
(cherry picked from commit be68991cf80f0aa1da7d36ab6e1d2c4d6c7cd3fb)
Signed-off-by: Robert McQueen <rob@endlessm.com>

Closes: #1685
Approved by: cgwalters
2018-07-20 14:58:40 +00:00
Robert Fairley
7baf167881 ostree/pull: Add network-retries command line option
This exposes a way to specify from the command line the number
of times to retry each download after a network error. If a negative
value is given, then the default number of retries (5) is used. If 0
is given, then errors are returned without retrying.

closes #1659

Closes: #1669
Approved by: jlebon
2018-07-05 17:59:09 +00:00
Umang Jain
0c8b86ea09 lib/repo: Minor fixes around min-free-space
Summary:
* Remove a useless if condition in prepare_transaction()
* Fix glnx_throw error propagation
* Integer overflow check while parsing min-free-space-size config
* Documentation fixes

Closes: #1663
Approved by: jlebon
2018-07-03 12:59:26 +00:00
Matthew Leeds
f1133a2a0a man/ostree.repo-config: Document collection-id
The collection-id option in the core section was recently made public
but not documented.

Closes: #1646
Approved by: cgwalters
2018-06-26 12:52:49 +00:00
Umang Jain
31809d32f2 lib/repo: Add min-free-space-size option
Similar to min-free-space-percent but it supports specific sizes
(in MB, GB or TB). Also, making min-free-space-percent and -size
mutually exclusive.

min-free-space-percent does not give a fine tuning of the free disk
space that a user might decide to keep. It can translate to very large
size (e.g. 1% = ~10GB on 1TB HDD) or very small (e.g. 1% = ~330MB on 32GB
system like Endless devices). Hence, it makes sense to introduce a config
option to honor specific size as per the user.

Closes: #1616
Approved by: jlebon
2018-06-13 18:57:37 +00:00
Matthew Leeds
c767f7b739 admin: Fix list of subcommands in help and manpage
This adds subcommands that were missing from the ostree-admin man page,
and makes cosmetic fixes there and in the --help output to ensure
alphabetical order and remove trailing whitespace.

Closes: #1621
Approved by: jlebon
2018-06-12 14:36:24 +00:00
Matthew Leeds
8fbf19c9f5 Make P2P API public (no longer experimental)
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.

This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak.  However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.

In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:10 +00:00
Jonathan Lebon
a85f06952d man/ostree-init: Briefly describe various modes
Closes: #1561

Closes: #1560
Approved by: cgwalters
2018-04-30 17:58:38 +00:00
Jonathan Lebon
54c5ab76a9 man/ostree: Document --repo option a bit more
This new information is already mostly part of `ostree.repo(5)`, though
let's put it in `ostree(1)` as well since that's where the switch is
officially documented.

Closes: #1560
Approved by: cgwalters
2018-04-30 17:58:38 +00:00
Matthew Leeds
972a1921f5 man: Add man page for create-usb
Closes: #1543
Approved by: cgwalters
2018-04-19 13:35:15 +00:00
Giuseppe Scrivano
127d8bb846 commit: add logic for .payload-link
When a new object is added to the repository, create a
$PAYLOAD-SHA256.payload-link symlink file as well.  The target of the
symlink is the checksum of the object that was added the repository.

Whenever we add a new object file, in addition to lookup if the file is
already present with the same checksum we also check if an object with
the same payload is in the repository.

If a file with the same payload is already present in the repository, we
copy it with `glnx_regfile_copy_bytes` that internally attempts to
create a reflink (ioctl (..., FICLONE, ..)) to the target file if the
file system supports it.  This enables to have objects that share the
payload but have a different inode and xattrs.

By default the payload-link-threshold value is G_MAXUINT64 that disables
the feature.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1443
Approved by: cgwalters
2018-03-07 18:28:59 +00:00
Colin Walters
1c9baad8a8 docs/prune: Document that --static-deltas-only isn't that useful
This is the documentation followup to: https://github.com/ostreedev/ostree/pull/1482

See also https://github.com/ostreedev/ostree/issues/1481

Closes: #1484
Approved by: jlebon
2018-03-06 14:02:32 +00:00
Colin Walters
7f88fddcd4 sysroot: Add concept of deployment "pinning" 📌
Example user story: Jane rebases her OS to a new major version N, and wants to
keep around N-1 even after a few upgrades for a while so she can easily roll
back. I plan to add `rpm-ostree rebase --pin` to opt-in to this for example.

Builds on the new `libostree-transient` group to store pinning state there.

Closes: https://github.com/ostreedev/ostree/issues/1460

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Marcus Folkesson
6bf4b3e1d8 Add SPDX-License-Identifier to source files
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.

Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
  text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
  environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
  license for their code is respected

See http://spdx.org for further reading.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>

Closes: #1439
Approved by: cgwalters
2018-01-30 20:03:42 +00:00
Matthew Leeds
4c2205276c man: Add man page for find-remotes
Closes: #1410
Approved by: pwithnall
2018-01-15 19:26:43 +00:00
Matthew Leeds
792b32eb96 man: Update ostree-summary.xml
Update the man page for the summary command to add the undocumented
options, make the syntax clear, and add examples.

Closes: #1416
Approved by: pwithnall
2018-01-15 17:33:06 +00:00
Colin Walters
d340fe4060 bin/fsck: Make ref binding verification optional
Today the rpm-ostree test suite uses `refs --create` to save
commits.  I think this is a legitimate use case, and other
people may be doing something similar.

On the other hand, I think we should probably be changing the rpm-ostree test
suite to create "unbound" commits. But let's be maximially compatible here since
we hit a real-world case where something needed to change.

Closes: #1379
Approved by: pwithnall
2017-12-14 18:41:00 +00:00
Philip Withnall
fb7692bd44 ostree/fsck: Add --verify-back-refs option
This new option verifies that the refs listed in the ref-bindings for
each commit all point to that commit (i.e. there aren’t multiple commits
listing the same ref in their ref-bindings, and there aren’t any commits
with non-empty ref-bindings which aren’t pointed at by a ref).

This is useful when generating a new repository from scratch, but not
useful when adding new commits to an existing repository (since the old
commits will still, correctly, have ref-bindings from when the refs
pointed at them). That’s why it has to be enabled explicitly using
--verify-back-refs, rather than being on by default.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Colin Walters
7173ac76bc pull: Add http2=false remote config option
This seems to work around
https://github.com/ostreedev/ostree/issues/1362

Though I'm not entirely sure why yet. But at least with this it'll be easier for
people to work around things locally.

Closes: #1368
Approved by: jlebon
2017-12-07 19:08:01 +00:00
Jonathan Lebon
077d2718ad lib/core: add ostree_checksum_file_at API
This is like `ostree_checksum_file` but fd-relative. This will be used
by https://github.com/ostreedev/ostree/pull/1258.

AFAICT, we actually didn't have any tests that check the `checksum` CLI.
Add a basic one here to test the old code as well as the new code.

Closes: #1263
Approved by: cgwalters
2017-10-12 12:53:01 +00:00
Colin Walters
bba7eb8069 commit: Add _CONSUME modifier flag
For many cases of commit, we can actually optimize things by simply "adopting"
the object rather than writing a new copy. For example, in rpm-ostree package
layering.

We can only make that optimization though if we take ownership of the file. This
commit hence adds an API where a caller tells us to do so. For now, that just
means we `unlink()` the files/dirs as we go, but we can now later add the
"adopt" optimization.

Closes: #1255
Approved by: jlebon
2017-10-10 13:02:08 +00:00
Philip Withnall
feeb3548f4 ostree/summary: Generate an ostree-metadata ref when updating summary
This is the new way of publishing repository metadata, rather than as
additional-metadata in the summary file. The use of an ostree-metadata
ref means that the metadata from multiple upstream collections is not
conflated when doing P2P mirroring of many repositories.

The new ref is only generated if the repository has a collection ID set.
The old summary file continues to be generated for backwards
compatibility (and because it continues to be the canonical ref →
checksum map for the repository).

The new code is only used if configured with --enable-experimental-api.

Includes unit tests.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1158
Approved by: cgwalters
2017-10-02 13:39:41 +00:00
Philip Withnall
d1a5652bdd man: Fix a copypasta error in ostree-summary.xml
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1158
Approved by: cgwalters
2017-10-02 13:39:41 +00:00
Colin Walters
051cdf396c lib/checkout: Rename disjoint union, change to merge identical files
It turns out that librpm automatically merges identical files between
distinct packages, and this occurs in practice with Fedora today between
`chkconfig` and `initscripts` for exmaple.

Since we added this for rpm-ostree, we basically want to do what librpm does,
let's change the semantics to do a merge.  While we're here rename
to `UNION_IDENTICAL`.

Closes: #1156
Approved by: jlebon
2017-09-13 19:19:33 +00:00
Dan Nicholson
adac42b6ef repo: Add add-remotes-config-dir option
This option allows a repo to explicitly opt out of adding new remotes in
a remotes configuration directory. This currently defaults to true for
system repos and false for non-system repos to maintain legacy behavior
that non-system repos don't add remotes in a configuration directory.
That would be problematic for flatpak, which specifies a remotes config
dir but adds remotes in ways that are incompatible with it.

So, what this really does is allow system repos to control whether they
want to add remotes in the config dir or not. That's important if your
flatpak repo is the system repo like at Endless.

Closes: #1134

Closes: #1155
Approved by: cgwalters
2017-09-11 10:53:20 +00:00
Colin Walters
88b25dc65c tree-wide: Replace archive-z2 with archive
In almost all places. There are just a few exceptions; one tricky bit for
example is that the repo config must still have `mode=archive-z2`, since
`archive` used to mean something else. (We could very likely just get rid of
that check, but eh, later).

I also added a test that one can still do `ostree repo init --mode=archive-z2`.

Closes: #1125
Approved by: jlebon
2017-09-01 20:54:12 +00:00
Ruixin
f07432d4ce checkout: add an extra checkout_overwrite mode
This is for issue projectatomic/rpm-ostree#365,
an extra option of overwrite mode is added to the checkout command
so that when there is "non-directory" file already exist
during checkout, the error will be handled.

Some tests are added for regression

Closes: #1116
Approved by: cgwalters
2017-09-01 15:42:50 +00:00
Jonathan Lebon
9342be6e34 ostree-sysroot: make simple_write_deployment smarter
This is a follow-up to https://github.com/ostreedev/ostree/pull/1097.
We make simple_write_deployment smart enough so that it can be used for
rpm-ostree's purposes. This is mostly an upstreaming of logic that
already existed there.

Notably we correctly append NOT_DEFAULT deployments *after* the booted
deployment and we now support RETAIN_PENDING and RETAIN_ROLLBACK flags
to have more granularity on deployment pruning.

Expose these new flags on the CLI using new options (as well as expose
the previously existing NOT_DEFAULT flag as --not-as-default).

I couldn't add tests for --retain-pending because the merge deployment
is always the topmost one. Though I did check that it worked in a VM.

Closes: #1110
Approved by: cgwalters
2017-08-25 01:02:15 +00:00
Jonathan Lebon
11fdca2919 pull: better description for --mirror
Describe the behaviour of --mirror a bit better.

Closes: #1100

Closes: #1099
Approved by: dustymabe
2017-08-23 01:23:38 +00:00
Jonathan Lebon
64b7c42025 pull: mark commits from local cache as partial
If one of the localcache repos has the exact same commit we resolved
from the remote, then we need to make sure to mark it as partial so that
we download the full tree.

Closes: #1074

Closes: #1076
Approved by: cgwalters
2017-08-14 12:04:42 +00:00
Jonathan Lebon
d3a5d26176 man: document configured branches
Closes: #1072

Closes: #1073
Approved by: guyshapiro
2017-08-10 14:54:03 +00:00
Matthew Leeds
6c7a6519dd man: Update ostree-refs manpage
Update the ostree-refs manpage to document recently added options, and
fix some minor mistakes.

Closes: #1068
Approved by: cgwalters
2017-08-10 14:31:49 +00:00
Stef Walter
39b3c8e90a man: The min-free-space-percent item goes in [core] section
The documentation incorrectly indicates that min-free-space-percent
goes in the [remote "name"] section. It should go in [core] instead.

Closes: #1062
Approved by: cgwalters
2017-08-08 12:56:47 +00:00
Guy Shapiro
b08a198f00 Documentation: static delta default from
Document that the default behavior of `ostree static-delta generate` if to generate the delta from the parent.
Closes: #1057
Approved by: cgwalters
2017-08-07 15:06:55 +00:00
Colin Walters
fbefe875c7 docs/pull: Finish incomplete docs for pull@hash syntax
Closes: https://github.com/ostreedev/ostree/issues/1043

Closes: #1045
Approved by: guyshapiro
2017-08-01 21:51:17 +00:00
Colin Walters
1f5ce1a9f7 lib/repo: Add min-free-space-percent option, default 3%
For ostree-as-host, we're the superuser, so we'll blow past
any reserved free space by default.  While deltas have size
metadata, if one happens to do a loose fetch, we can fill
up the disk.

Another case is flatpak: the system helper has similar concerns
here as ostree-as-host, and for `flatpak --user`, we also
want to be nice and avoid filling up the user's quota.

Closes: https://github.com/ostreedev/ostree/issues/962

Closes: #987
Approved by: jlebon
2017-07-04 16:15:11 +00:00
Colin Walters
4273e670ea Add "pull --localcache-repo"
This is a lot like `git clone --reference`, but we chose "localcache" as the
term "reference" is already used.

The main use case I'm targeting this for is the Fedora Atomic Host installer
case where we embed the repo content in the installer, but we may want to
kickstart and download newer content. There, while we want to get a newer ref,
we can still use the local repo as an object cache, since we have it sitting
there in memory anyways.

Another case is where one has a host ostree (say e.g. Fedora Atomic
Workstation), and one wants to create a local archive mirror of FAH. Then one
can use `pull --reference /ostree/repo` and pull the common objects (e.g.
contents of `bash.rpm` etc.)

Closes: https://github.com/ostreedev/ostree/issues/975

Closes: #982
Approved by: jlebon
2017-06-30 19:37:23 +00:00
Philip Withnall
e0ad9b2266 ostree/summary: Add support for adding additional metadata
When updating a summary file, parse additional arguments to the `ostree
summary` command as additional metadata to be put into the summary.

Add some tests for this.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall
b7b79fa78d refs: Add a --collections argument to the built-in refs command
Rather than change the output format used by the existing refs command
to output collection IDs in addition to ref names, this functionality has
been hidden behind an --collections argument. If it’s not specified `ostree
refs` will output the same content as before for a given repository. If
it is specified, the collection ID for each ref will be included in the
output as (collection ID, ref name).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
205a84b36e remote-add: Add a --collection-id argument to the built-in add command
This allows newly configured remotes to have their collection ID specified,
so that refs from them can be downloaded from peers as well as the
upstream collection, using the remote’s configuration.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall
144e325579 init: Add a --collection-id argument to the built-in init command
This allows new repositories to be configured with a collection ID which
can be used to uniquely identify refs which originated from this
repository.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Colin Walters
55603a0c52 Rename "osname" → "stateroot"
I never really liked the term "osname". I feel "stateroot" is a *lot* clearer,
since the osname/stateroot mostly just holds `/var`. Further it avoids the `os`
prefix which is already overloaded.

Some of the existing docs already talked about "operating system state", which
further reinforces this.

There's *lot* more things than this which reference the term "osname", but I
don't want to change *everything* yet in this patch in case we decide to do
something different - this just gets the highlights.

Closes: #794
Approved by: jlebon
2017-04-24 16:09:51 +00:00
Erik Larsson
e665e51408 diff: Add ostree_diff_dirs_with_options(), expose via cmdline
The first options are owner_uid/owner_gid, which makes it possible to use diff
on local files where --owner-uid/gid have been passed to commit.

Closes: #740
Approved by: cgwalters
2017-03-21 13:38:04 +00:00
Colin Walters
94948e3522 checkout: Support a "pure addition" mode
I plan to use this for `rpm-ostree livefs`.
https://github.com/projectatomic/rpm-ostree/issues/639

Closes: #714
Approved by: jlebon
2017-03-06 20:58:04 +00:00
Colin Walters
36b28cb4d2 upgrade: Add support for --pull-only and --deploy-only
This makes it easier to script downloading updates in the background,
and only do deployments just before rebooting.

Partially addresses https://github.com/ostreedev/ostree/issues/640

Closes: #642
Approved by: jlebon
2017-02-27 20:57:23 +00:00