IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
When there is number after "T" (suggested number of threads or "0" for
getncpus), lzopen_internal() mode parser would skip one byte, and when
it's at the end of the string it would then parse undesired garbage from
the memory, making intermittent compression failures.
Fixes: c23872d71 ("Add support for multithreaded xz compression")
Fixes-upstream: 7740d1098.
Upstream-PR: https://github.com/rpm-software-management/rpm/pull/1478
Mode flags could change from what is specified in defines and errors
could happen before any other descriptive messages would appear, but
it's useful to know some additional info about xz compression to debug
multi-threading memory failures.
Reviewed-by: Arseny Maslennikov <arseny@altlinux.org>
Liblzma is non-robust on memory allocation problems - while it works
well on 64-bit architectures with plenty of RAM, it's problematic on
32-bit architectures when multi-threading is used.
Try another workaround: instead of just guessing the best delta, move
half of the current (512MiB) delta into 'guard malloc' allocation test
of what LZMA encoder memusage would return.
This is definitely hackish (and vulnerable to toctou), but it's that
much we could do without significantly reworking stream_encoder_mt.
Error message:
Wrote: /usr/src/RPM/RPMS/armh/libmozjs78-tools-78.0.1-alt2.armh.rpm (w5.lzdio)
error: lzwrite: lzma error 5
error: lzclose: lzma error 11
error: create archive failed on file /usr/src/tmp/libmozjs78-buildroot/usr/lib/debug/usr/lib/libmozjs-78.so.debug: cpio: write failed - Bad file descriptor
Amends: f5fcb8f43 ("Lower memory limit on 32-bit arches for xz compression, again")
Amends: afe660558 ("Lower memory limit on 32-bit systems for xz compression")
Amends: 0b18c0498 ("prevent exceeding 32 bit memory limitations with multithreaded xz compression")
Reviewed-by: Dmitry V. Levin <ldv@altlinux.org>
rpmio.c: In function 'get_compression_threads':
rpmio.c:2114:12: warning: implicit declaration of function 'rpmExpandNumeric' [-Wimplicit-function-declaration]
2114 | threads = rpmExpandNumeric("%{getncpus}");
| ^~~~~~~~~~~~~~~~
Reported-by: Dmitry V. Levin <ldv@altlinux.org>
For some systems 128 MiB delta is not enough for `w7T.xzdio`. Found by
experiment that 256 MiB is enough.
Fixes:
error: create archive failed on file /usr/src/in/source/FlightGear-data-2020.1.2.tar: cpio: write failed - Bad file descriptor
Based on commit a60f36a55 by Per Øyvind Karlsen. Original commit message:
As 32 bit build suffers under the limitation of 32 bit address space,
regardless of it's environment would be ie. 64 bit and not have this
constration, rpm must make sure not to exceed this memory limitation.
When using multithreaded xz compression, the number of threads used will
increase the memory usage, making it necessary to check the memory
required with the number of threads to be used.
Number of compression threads will therefore be kept reduced untill
amount of memory required won't exceed this limitation.
For 32 bit binaries running under 64 bit host environment, where less
available memory will be reserved for kernel, easing memory constraints,
determination of this will be done by a combination of checking host
arch as well as whether 32 bit personality flag is set, thereby still
allow a sligthly greater memory usage for such cases to avoid
imposing unnecessatry limitations under such environments.
Changes from the original commit:
- Do not call uname(2).
- Limits are increased.
- Do not call lzma_memlimit_set().
- No message about thread limitation.
Reviewed-by: Dmitry V. Levin <ldv@altlinux.org>
Mechanically substitute _IO_off64_t with off64_t using
$ git grep -Fwl _IO_off64_t |xargs -r sed -i s/_IO_off64_t/off64_t/g
This follows glibc-2.28~574 and fixes build with glibc >= 2.28.
- Implemented limited support for large files: a 2GB+ file can now be packaged,
but the total size of uncompressed cpio payload is capped at 4 GB.
- Automatically downgrade LZMA compression levels 7-9 -> 6 on small payloads.
This is the reason why files >= 2G could not be added to cpio, even on
64-bit architectures. Fwrite was called with a buffer >= 2G, but the
return value was truncated to int (this narrowing conversion is probably
undefined behavior already; what happens next, though, is that after
casting to size_t it assumes a very big numeric value of >= 2^64-2^32).
So rpmbuild was failing with an unrelated errno, "cpio: write failed -
Bad file descriptor".
There are other reasons why this still won't work on 32-bit architectures.
For one, rpmbuild mmaps a file before feeding it to cpio, and 32-bit
architectures have a very limited ability, if any, to mmap 2G+ files.
In file included from /usr/include/string.h:648:0,
from ../system.h:76,
from rpmio.c:6:
In function 'strncat',
inlined from 'Fdopen' at rpmio.c:3473:5:
/usr/include/bits/string3.h:158:10: warning: call to __builtin___strncat_chk might overflow destination buffer
return __builtin___strncat_chk (__dest, __src, __len, __bos (__dest));
In file included from /usr/include/string.h:648:0,
from ../system.h:76,
from rpmio.c:6:
In function 'strncat',
inlined from 'Fdopen' at rpmio.c:3473:5:
/usr/include/bits/string3.h:158:10: warning: call to __builtin___strncat_chk might overflow destination buffer
return __builtin___strncat_chk (__dest, __src, __len, __bos (__dest));
Grab the uncompressed payload size from header and compare number of
bytes copied to that for exit code. Previously, truncated payloads could
have returned with success. This also fixes the exit code for large
payloads.
