Resolve (some of) parallelism bottlenecks in test suites

[Xi Ruoyao]

There are some parallelism bottlenecks in test suites:

• Autoconf, Libtool, Tar, and maybe others
  • Solution: set TESTSUITEFLAGS=-j$N globally (like MAKEFLAGS)
• Automake (with more than 4 cores)
  • Solution: set -j $(($N > 4 ? $N : 4)) instead of just -j4
• OpenSSL
  • Solution: HARNESS_JOBS=$N make test
• Perl
  • Solution: TEST_JOBS=$N make test_harness
• Binutils, Libffi, and Findutils
  • Not solvable, limitation of DejaGNU. GCC has managed to resolve the issue but the solution is very complex.
• Make
  • Not solvable, test suite just does not support parallelism.
• Vim
  • Not solvable, of course we cannot test an editor in a parallelized way.

[pierre]

WIP: see banch xry111/parallelism

[pierre]

bison and kbd, too, honor TESTSUITEFLAGS.

[Xi Ruoyao]

Rebased and added two commits.

Log: https://wiki.linuxfromscratch.org/lfs/log/?rev=xry111/parallelism&stop_rev=trunk

[Xi Ruoyao]

Replying to Xi Ruoyao:

Rebased and added two commits.

Log: https://wiki.linuxfromscratch.org/lfs/log/?rev=xry111/parallelism&stop_rev=trunk

To editors: please review the change. If there is no issue we'll adapt jhalfs to handle this.

[pierre]

I am all for it. On IRC, Bruce has proposed an alternative to $(nproc || echo 1), using lscpu. But lscpu, too is not guaranteed to be present on the host, I think. We could parse /proc/cpuinfo, which is almost guaranteed to be on the host (but we have to test with small distros like alpine). Something like:

awk '/siblings/{print $3}' /proc/cpuinfo | tail -n 1

Note that if we need to test with a given number of processors, we can limit them by writing to /sys/devices/system/cpu/cpu<N>/online. Or we can use systemd-run -p AllowedCPUs= (needs systemd obviously) or use cgroup cpuset controller (need to be enabled in kernel). This could be implemented in jhalfs, but users are free to use them too (we can give some explanations).

[Xi Ruoyao]

Replying to pierre:

I am all for it. On IRC, Bruce has proposed an alternative to $(nproc || echo 1), using lscpu. But lscpu, too is not guaranteed to be present on the host, I think. We could parse /proc/cpuinfo, which is almost guaranteed to be on the host (but we have to test with small distros like alpine). Something like:

awk '/siblings/{print $3}' /proc/cpuinfo | tail -n 1

This command does not work well with a multi-CPU configuration. For example, if there are two Xeon CPUs both has 32 logical cores, the command will just say "32".

[Xi Ruoyao]

grep -c '^processor' /proc/cpuinfo

But I'm not sure the first grep version supporting "-c".

[Xi Ruoyao]

Replying to Xi Ruoyao:

grep -c '^processor' /proc/cpuinfo

But I'm not sure the first grep version supporting "-c".

It's available in grep-2.5.1: ​https://git.savannah.gnu.org/cgit/grep.git/tree/src/grep.c?h=v2.5.1

[pierre]

Not sure why we require grep-2.5.1a or above instead of grep-2.5.1. According to the changelog, the only changes between those versions were packaging and some tests...

[Xi Ruoyao]

Hmm, maybe it will be easier to just bump Coreutils minimal version to 8.1 and check if nproc works in host system requirements.

[Xi Ruoyao]

Fixed:

• r12.0-87-g7a72a22f9 Merge tag 'xry111/update-20231113' into trunk
• r12.0-79-g1780e72d8 udev: Fix a small typo in command
• r12.0-85-ge07c41946 Package updates
• r12.0-84-gd490056b6 perl: Use TEST_JOBS=$(nproc) make test_harness for test suite
• r12.0-83-g3a8d1825c openssl: Add HARNESS_JOBS=$(nproc)
• r12.0-82-gc17a77452 automake: Do not use only 4 cores for testing if we have more
• r12.0-81-g95ebbb42b chroot: Set MAKEFLAGS and TESTSUITEFLAGS for parallelism
• r12.0-80-ga0a803c0b settingenviron: Set MAKEFLAGS for parallelism in ~lfs/.bashrc
• r12.0-79-ge70bf8f4f hostreqs: Bump Coreutils minimal version to 8.1 and check for nproc