Commit Graph

38 Commits

Author SHA1 Message Date
Ivan A. Melnikov
47d5f6fa92 repos: Add candidates_to_build
This is a helper function to find out which packages
from one repo can probably be immediately build for
the other one.

The implementation is suboptimal, but works just fine
for now.
2024-12-20 13:08:12 +04:00
Ivan A. Melnikov
e2f1051f7d repos: Rewrite missing_rating in terms of BuildReporter 2024-12-20 13:06:41 +04:00
Ivan A. Melnikov
a7df65c0e4 repos: Fix BuildReporter caching 2024-12-20 13:05:30 +04:00
Ivan A. Melnikov
573c9ae458 repos.BuildReporter: Option to ignore file dependencies
This turns out to be useful when may reports are
analized automatically.
2023-12-13 14:09:04 +04:00
Ivan A. Melnikov
333310dc9b repos: Reduce the logging from build report 2023-12-06 09:45:40 +04:00
Ivan A. Melnikov
94458d44ce repos.recursive_build_report: Refactor
Use BuildReporter.recursive_closure for the first stage
of report generation. This leads to nice code cleanup.
2023-12-05 15:07:15 +04:00
Ivan A. Melnikov
1c0903b9dc recursive_build_report: Return earlier if summary was not requested 2023-12-05 14:08:18 +04:00
Ivan A. Melnikov
09b4700064 repos: Introduce BuildReporter
Instead of global cache, we now have an object
which encapuslates it, together with parameters
that affect reports. This seems to be simpler
and more robust model.

missing_rating and recursive_build_report
now use it, also the refactoring is not yet
complete.
2023-12-05 14:03:36 +04:00
Ivan A. Melnikov
51aa8967d0 repos: Don't fail if addons are not preset in config 2023-12-05 13:55:08 +04:00
Ivan A. Melnikov
8388c0627b lists: Parallelize read_pkglist_heders_for_repo
Use multiprocessing module to read the headers for each
repo in parallel. Usualy reading data for repository means
loading 6 lists, so the gain is considerable.
2023-12-05 13:27:43 +04:00
Ivan A. Melnikov
bbf24803fa repos: Add RecursiveBuildReport missing_rating
The plan is to split the recursive_build_report
into more structured and easy-to-operate pieces.
The first part of those pieces is introduction
of a separate class that walks the dependency
trees to build a closure of build requirements
for a given package.

To show what's already possible with it,
we add new report, missing_rating_report,
that shows the most wanted of packages that
are missing from a given repository.
2023-10-24 18:32:25 +04:00
Ivan A. Melnikov
ad19973427 repos: Cache for build report 2023-10-24 18:29:10 +04:00
Ivan A. Melnikov
66ce9a190c repos: BREAKING: A class for triplet report
Reports that consist of triplets (kind, dep, pkg)
are surprisingly common. This commit introduces
a class to store such report, in hope to be able
to manipulate them more intuitively.

The change is breaking, as some reports (like
build_report and unmet_report) are now returning
the new class instance instead of string.
2023-10-23 15:21:22 +04:00
Ivan A. Melnikov
1ad8c35682 repos: Save load timestamp in the repository 2023-10-23 14:22:20 +04:00
Ivan A. Melnikov
816ff27123 BREAKING: Refactor repos loading
To avoid hardcoding repository names and make
the code more easily extendable, we now
use the keys from the config file as
the repository names instead of hardcoding
the fixed set of names.

This means most config will require updating.
2023-10-05 12:46:28 +04:00
Ivan A. Melnikov
d15a93c39a repos: Avoid missing build requirements
Fixes the previous commit series.
2023-08-17 17:26:35 +04:00
Ivan A. Melnikov
36010b8fc7 repos: Refactor loading the global repos
Instead of ignoring all the exceptions silently,
we check if the repository is present in the config
file, and the loading fail loudly for any other reason
if it happens to fail.
2023-08-17 17:02:32 +04:00
Ivan A. Melnikov
84f5e191ba repos: Form the basic chroots and cache them
This way, we'll take into account the dependencies
on the packages from the base environment when
sorting the package set topologically.
2023-08-17 17:01:05 +04:00
Ivan A. Melnikov
2b779f5113 Introduce a separate entity for chroots
Holding together a set of binaries turns out
to be a nice abstraction.
2023-08-17 16:20:20 +04:00
Ivan A. Melnikov
ec92670e48 repos: Lower 'Cycle detected' message to debug
The summary contains all the cycles anyway, and many
such messages are not useful and only slow things down.
2023-08-17 15:42:34 +04:00
Ivan A. Melnikov
29059c3ae4 Repos: support repo addons and repos with unmets
In the early bootstrap stage (like loongarch64 now)
repository may have some unmets (mostly build unmets)
that are worked around by adding additional (e.g. noarch)
source into the build environment.

This commit tries to model this by adding a set of binaries
to repo (we call this 'addon') and using it to satisfy
build requiremets if none are provided by the repo itself.

This commit also improves build report accuracy in presence
of unments: packages that are required to resolve the unmets
we find are added to build requirements of the package
with special marker.
2023-08-14 14:09:26 +04:00
Ivan A. Melnikov
d040c3f6a4 repos: Option to generate recursive report w/o summary
On certain graphs, repeative topolgical sorting takes
too much time. So, we introduce an option to disable
the summary and produce all the required build
reports only.
2023-08-14 11:53:58 +04:00
Ivan A. Melnikov
8141ceb1ab repos: Add colorizing reports
These reports can be used to compare the sets of source
packages of two repositories, e.g. to find out which
packages should be updated.
2023-08-04 13:58:18 +04:00
Ivan A. Melnikov
fe17158f69 repos: Prefer requested sources for recursive report 2023-08-03 18:21:09 +04:00
Ivan A. Melnikov
79a92c2358 repos: Add repo source_for_dependency method
... to conviniently learn what sources you need to build
to get something that provides this dependency, by name.
2023-08-03 16:38:39 +04:00
Ivan A. Melnikov
5216695457 repos: Update summary formatting for recursive report
- distingush new packages ('+') from updated ('u');
- wider field for SRPM.
2023-08-02 15:01:15 +04:00
Ivan A. Melnikov
47bac38aa8 repos: Consider checkinstall, too
Some checkinstall packages (like rpminstall-tests-checkinstall)
have large dependencies that affect ability to build package
when install checks are enabled.
2023-08-02 14:59:08 +04:00
Ivan A. Melnikov
ea0db14e8b scripts: Use public resources in run.sh
This should serve as example. For internal uses,
one should add their own script.
2023-08-01 18:24:02 +04:00
Ivan A. Melnikov
d0197d8475 repos: Load more repositories, uniformely
We are interested in x86_64 (as base), mipsel, riscv64 and
loongarch64 (named la64 in the interactive prompt
for brievity).
2023-08-01 18:09:14 +04:00
Ivan A. Melnikov
14a501684e repos: Fix 'target has NONE' in recursive build report
Due to typo, recursive build reports sometimes
contained 'has NONE' lines for sources that
were, in fact, present in the target repository.

This commit fixes that.
2023-08-01 18:07:23 +04:00
Ivan A. Melnikov
8660198cbd repos: Rework unmet-based reports; add diff_deps
Refactor unmet-based repos for cleaner code.

Add one more unmet-based repo, to check what
packages in the target repo still have
a dependency but should not.
2023-07-24 17:47:27 +04:00
Ivan A. Melnikov
e39256876e repos: Better heuristics for repo 64-bitness 2023-07-24 17:46:24 +04:00
Ivan A. Melnikov
4469729bc1 lists: Ignore *.bz2 pkglists
They are often outdated.
2023-07-24 17:44:38 +04:00
Ivan A. Melnikov
90d99e3882 lists: Read only the first of compressed pkglists
By default, genbasedir saves several copies of every
package list, compressing them with different methods
(xz, bz2, uncompressed). With this change, we read
only the first one, thus avoiding extra work
and (false) warnings about duplicate binaries.
2023-06-02 15:52:46 +04:00
Ivan A. Melnikov
e2dc5736a2 run.sh: Exclude contents_index
It's too big and not useful for us.
2023-05-23 10:02:01 +04:00
Ivan A. Melnikov
acf51ca89d Add some dependencies info to README.md 2023-05-19 14:25:43 +04:00
Ivan A. Melnikov
b446b5f37c Add some code
Python code copyed from port-compare commit
153130a18f2ce80d8f9646ab94b0f9eeb0e994de
with minor modifications.

Runner script added
2023-05-19 13:47:29 +04:00
Ivan A. Melnikov
2631b10264 Initial commit 2023-05-19 13:46:31 +04:00