Commit Graph

31 Commits

Author SHA1 Message Date
Ivan A. Melnikov
09b4700064 repos: Introduce BuildReporter
Instead of global cache, we now have an object
which encapuslates it, together with parameters
that affect reports. This seems to be simpler
and more robust model.

missing_rating and recursive_build_report
now use it, also the refactoring is not yet
complete.
2023-12-05 14:03:36 +04:00
Ivan A. Melnikov
51aa8967d0 repos: Don't fail if addons are not preset in config 2023-12-05 13:55:08 +04:00
Ivan A. Melnikov
8388c0627b lists: Parallelize read_pkglist_heders_for_repo
Use multiprocessing module to read the headers for each
repo in parallel. Usualy reading data for repository means
loading 6 lists, so the gain is considerable.
2023-12-05 13:27:43 +04:00
Ivan A. Melnikov
bbf24803fa repos: Add RecursiveBuildReport missing_rating
The plan is to split the recursive_build_report
into more structured and easy-to-operate pieces.
The first part of those pieces is introduction
of a separate class that walks the dependency
trees to build a closure of build requirements
for a given package.

To show what's already possible with it,
we add new report, missing_rating_report,
that shows the most wanted of packages that
are missing from a given repository.
2023-10-24 18:32:25 +04:00
Ivan A. Melnikov
ad19973427 repos: Cache for build report 2023-10-24 18:29:10 +04:00
Ivan A. Melnikov
66ce9a190c repos: BREAKING: A class for triplet report
Reports that consist of triplets (kind, dep, pkg)
are surprisingly common. This commit introduces
a class to store such report, in hope to be able
to manipulate them more intuitively.

The change is breaking, as some reports (like
build_report and unmet_report) are now returning
the new class instance instead of string.
2023-10-23 15:21:22 +04:00
Ivan A. Melnikov
1ad8c35682 repos: Save load timestamp in the repository 2023-10-23 14:22:20 +04:00
Ivan A. Melnikov
816ff27123 BREAKING: Refactor repos loading
To avoid hardcoding repository names and make
the code more easily extendable, we now
use the keys from the config file as
the repository names instead of hardcoding
the fixed set of names.

This means most config will require updating.
2023-10-05 12:46:28 +04:00
Ivan A. Melnikov
d15a93c39a repos: Avoid missing build requirements
Fixes the previous commit series.
2023-08-17 17:26:35 +04:00
Ivan A. Melnikov
36010b8fc7 repos: Refactor loading the global repos
Instead of ignoring all the exceptions silently,
we check if the repository is present in the config
file, and the loading fail loudly for any other reason
if it happens to fail.
2023-08-17 17:02:32 +04:00
Ivan A. Melnikov
84f5e191ba repos: Form the basic chroots and cache them
This way, we'll take into account the dependencies
on the packages from the base environment when
sorting the package set topologically.
2023-08-17 17:01:05 +04:00
Ivan A. Melnikov
2b779f5113 Introduce a separate entity for chroots
Holding together a set of binaries turns out
to be a nice abstraction.
2023-08-17 16:20:20 +04:00
Ivan A. Melnikov
ec92670e48 repos: Lower 'Cycle detected' message to debug
The summary contains all the cycles anyway, and many
such messages are not useful and only slow things down.
2023-08-17 15:42:34 +04:00
Ivan A. Melnikov
29059c3ae4 Repos: support repo addons and repos with unmets
In the early bootstrap stage (like loongarch64 now)
repository may have some unmets (mostly build unmets)
that are worked around by adding additional (e.g. noarch)
source into the build environment.

This commit tries to model this by adding a set of binaries
to repo (we call this 'addon') and using it to satisfy
build requiremets if none are provided by the repo itself.

This commit also improves build report accuracy in presence
of unments: packages that are required to resolve the unmets
we find are added to build requirements of the package
with special marker.
2023-08-14 14:09:26 +04:00
Ivan A. Melnikov
d040c3f6a4 repos: Option to generate recursive report w/o summary
On certain graphs, repeative topolgical sorting takes
too much time. So, we introduce an option to disable
the summary and produce all the required build
reports only.
2023-08-14 11:53:58 +04:00
Ivan A. Melnikov
8141ceb1ab repos: Add colorizing reports
These reports can be used to compare the sets of source
packages of two repositories, e.g. to find out which
packages should be updated.
2023-08-04 13:58:18 +04:00
Ivan A. Melnikov
fe17158f69 repos: Prefer requested sources for recursive report 2023-08-03 18:21:09 +04:00
Ivan A. Melnikov
79a92c2358 repos: Add repo source_for_dependency method
... to conviniently learn what sources you need to build
to get something that provides this dependency, by name.
2023-08-03 16:38:39 +04:00
Ivan A. Melnikov
5216695457 repos: Update summary formatting for recursive report
- distingush new packages ('+') from updated ('u');
- wider field for SRPM.
2023-08-02 15:01:15 +04:00
Ivan A. Melnikov
47bac38aa8 repos: Consider checkinstall, too
Some checkinstall packages (like rpminstall-tests-checkinstall)
have large dependencies that affect ability to build package
when install checks are enabled.
2023-08-02 14:59:08 +04:00
Ivan A. Melnikov
ea0db14e8b scripts: Use public resources in run.sh
This should serve as example. For internal uses,
one should add their own script.
2023-08-01 18:24:02 +04:00
Ivan A. Melnikov
d0197d8475 repos: Load more repositories, uniformely
We are interested in x86_64 (as base), mipsel, riscv64 and
loongarch64 (named la64 in the interactive prompt
for brievity).
2023-08-01 18:09:14 +04:00
Ivan A. Melnikov
14a501684e repos: Fix 'target has NONE' in recursive build report
Due to typo, recursive build reports sometimes
contained 'has NONE' lines for sources that
were, in fact, present in the target repository.

This commit fixes that.
2023-08-01 18:07:23 +04:00
Ivan A. Melnikov
8660198cbd repos: Rework unmet-based reports; add diff_deps
Refactor unmet-based repos for cleaner code.

Add one more unmet-based repo, to check what
packages in the target repo still have
a dependency but should not.
2023-07-24 17:47:27 +04:00
Ivan A. Melnikov
e39256876e repos: Better heuristics for repo 64-bitness 2023-07-24 17:46:24 +04:00
Ivan A. Melnikov
4469729bc1 lists: Ignore *.bz2 pkglists
They are often outdated.
2023-07-24 17:44:38 +04:00
Ivan A. Melnikov
90d99e3882 lists: Read only the first of compressed pkglists
By default, genbasedir saves several copies of every
package list, compressing them with different methods
(xz, bz2, uncompressed). With this change, we read
only the first one, thus avoiding extra work
and (false) warnings about duplicate binaries.
2023-06-02 15:52:46 +04:00
Ivan A. Melnikov
e2dc5736a2 run.sh: Exclude contents_index
It's too big and not useful for us.
2023-05-23 10:02:01 +04:00
Ivan A. Melnikov
acf51ca89d Add some dependencies info to README.md 2023-05-19 14:25:43 +04:00
Ivan A. Melnikov
b446b5f37c Add some code
Python code copyed from port-compare commit
153130a18f2ce80d8f9646ab94b0f9eeb0e994de
with minor modifications.

Runner script added
2023-05-19 13:47:29 +04:00
Ivan A. Melnikov
2631b10264 Initial commit 2023-05-19 13:46:31 +04:00