prevent exceeding 32 bit memory limitations with multithreaded xz compression

Based on commit a60f36a55 by Per Øyvind Karlsen. Original commit message:

  As 32 bit build suffers under the limitation of 32 bit address space,
  regardless of it's environment would be ie. 64 bit and not have this
  constration, rpm must make sure not to exceed this memory limitation.
  When using multithreaded xz compression, the number of threads used will
  increase the memory usage, making it necessary to check the memory
  required with the number of threads to be used.

  Number of compression threads will therefore be kept reduced untill
  amount of memory required won't exceed this limitation.

  For 32 bit binaries running under 64 bit host environment, where less
  available memory will be reserved for kernel, easing memory constraints,
  determination of this will be done by a combination of checking host
  arch as well as whether 32 bit personality flag is set, thereby still
  allow a sligthly greater memory usage for such cases to avoid
  imposing unnecessatry limitations under such environments.

Changes from the original commit:

- Do not call uname(2).
- Limits are increased.
- Do not call lzma_memlimit_set().
- No message about thread limitation.

Reviewed-by: Dmitry V. Levin <ldv@altlinux.org>

rpmio/rpmio.c

@@ -7,6 +7,7 @@
 #include <stdarg.h>
 #include <errno.h>
 #include <ctype.h>
+#include <sys/personality.h>
 
 #if HAVE_MACHINE_TYPES_H
 # include <machine/types.h>
@@ -2876,6 +2877,29 @@ static LZFILE *lzopen_internal(const char *path, const char *mode, int fd, int x
 		    .filters = lzfile->filters,
 		    .check = LZMA_CHECK_CRC64 };
 
+#if __SIZEOF_LONG__ == 4
+		/* Reduce number of threads for 32-bit systems to limit memory usage. */
+		if (threads > 1) {
+		    uint64_t memory_usage;
+		    uint32_t memlimit = (SIZE_MAX >> 1) + (SIZE_MAX >> 2); /* 3 GiB */
+
+		    if ((personality(0xffffffff) & PER_MASK) == PER_LINUX32)
+			memlimit = SIZE_MAX; /* 4 GiB */
+
+		    memlimit -= SIZE_MAX >> 5; /* 128 MiB */
+
+		    /* keep reducing the number of threads until memory usage gets below limit */
+		    while ((memory_usage = lzma_stream_encoder_mt_memusage(&mt_options)) > memlimit) {
+			/* number of threads shouldn't be able to hit zero with
+			 * compression settings available to set through rpm... */
+			if (--mt_options.threads == 0) {
+			    mt_options.threads = 1;
+			    break;
+			}
+		    }
+		}
+#endif
+
 		ret = lzma_stream_encoder_mt(&lzfile->strm, &mt_options);
 	    }
 #endif