Broken lapack/blas in Rocky Linux 9.3

To preface this post, this is all that changed with the RPM: import lapack-3.9.0-10.el9 (d6d9cc1a) · Commits · staging / rpms / lapack · GitLab - Not much changed here, except for the addition of explicit requires for some packages.

This is how pungi (our initial composing tool) determined what was needed to be in the repositories. The packages have always been built (you can actually verify that in the vault devel 9.2 repo, but pungi is what ultimately determines what will be in the repos by looking at dependencies of each package. Pungi is what decided it needed to be there.

If you look at the rpm description, they both say they are for 64bit integars, but lapack64_ has a symbol name suffix. see dnf repoquery -q lapack64 -i and dnf repoquery -q lapack64_ -i.

Below are the dependencies used to build lapack for 9.3.

================================================================================
 Package               Arch      Version                   Repository      Size
================================================================================
Installing:
 gcc-gfortran          x86_64    11.4.1-2.1.el9            yumrepofs_0     13 M
Installing dependencies:
 annobin               x86_64    12.12-1.el9               yumrepofs_0    977 k
 cpp                   x86_64    11.4.1-2.1.el9            yumrepofs_0     11 M
 gcc                   x86_64    11.4.1-2.1.el9            yumrepofs_0     32 M
 gcc-plugin-annobin    x86_64    11.4.1-2.1.el9            yumrepofs_0     48 k
 glibc-devel           x86_64    2.34-83.el9.7             yumrepofs_0     50 k
 glibc-headers         x86_64    2.34-83.el9.7             yumrepofs_0    450 k
 kernel-headers        x86_64    5.14.0-362.el9            yumrepofs_0    6.4 M
 libedit               x86_64    3.1-38.20210216cvs.el9    yumrepofs_0    103 k
 libgfortran           x86_64    11.4.1-2.1.el9            yumrepofs_0    807 k
 libmpc                x86_64    1.2.1-4.el9               yumrepofs_0     61 k
 libquadmath           x86_64    11.4.1-2.1.el9            yumrepofs_0    197 k
 libquadmath-devel     x86_64    11.4.1-2.1.el9            yumrepofs_0     38 k
 libxcrypt-devel       x86_64    4.4.18-3.el9              yumrepofs_0     28 k
 llvm-libs             x86_64    16.0.6-4.el9              yumrepofs_0     24 M

The build logs are here if you wish to look at them. (Warning, it is difficult to parse through the logs. We are working in the background to make the UI better.).

Whether or not that you look through the build logs, if you believe something is indeed wrong, we ask that you file a bug report with as much detail as possible. This includes debug data (e.g. gdb), related installed packages, example broken scripts/programs. Bug Tracker.

It works if you have glibc-devel.i686 (and other i686 packages) and you use -m32, which I’m sure goes against yours (and others) expectations.

[root@awx ~]# dnf install lapack lapack.i686 lapack64 lapack-devel.i686 lapack-devel glibc-devel.i686 -y
Last metadata expiration check: 3:02:57 ago on Thu 18 Jan 2024 11:04:47 PM MST.
Dependencies resolved.
================================================================================
 Package               Arch       Version                   Repository     Size
================================================================================
Installing:
 glibc-devel           i686       2.34-83.el9.7             appstream      50 k
 lapack                i686       3.9.0-10.el9              appstream     8.5 M
 lapack                x86_64     3.9.0-10.el9              appstream     8.5 M
 lapack-devel          i686       3.9.0-10.el9              crb            62 k
 lapack-devel          x86_64     3.9.0-10.el9              crb            63 k
 lapack64              x86_64     3.9.0-10.el9              crb           2.2 M
Installing dependencies:
 blas                  i686       3.9.0-10.el9              appstream     456 k
 blas                  x86_64     3.9.0-10.el9              appstream     479 k
 blas-devel            i686       3.9.0-10.el9              crb            14 k
 blas-devel            x86_64     3.9.0-10.el9              crb            15 k
 blas64                x86_64     3.9.0-10.el9              crb           209 k
 blas64_               x86_64     3.9.0-10.el9              crb           209 k
 cpp                   x86_64     11.4.1-2.1.el9            appstream      11 M
 gcc                   x86_64     11.4.1-2.1.el9            appstream      32 M
 gcc-gfortran          x86_64     11.4.1-2.1.el9            appstream      13 M
 glibc                 i686       2.34-83.el9.7             baseos        1.9 M
 glibc-devel           x86_64     2.34-83.el9.7             appstream      50 k
 glibc-headers         x86_64     2.34-83.el9.7             appstream     450 k
 kernel-headers        x86_64     5.14.0-362.13.1.el9_3     appstream     6.4 M
 lapack64_             x86_64     3.9.0-10.el9              crb           2.2 M
 libgcc                i686       11.4.1-2.1.el9            baseos        104 k
 libgfortran           i686       11.4.1-2.1.el9            baseos        774 k
 libgfortran           x86_64     11.4.1-2.1.el9            baseos        807 k
 libmpc                x86_64     1.2.1-4.el9               appstream      61 k
 libquadmath           i686       11.4.1-2.1.el9            baseos        245 k
 libquadmath           x86_64     11.4.1-2.1.el9            baseos        197 k
 libquadmath-devel     x86_64     11.4.1-2.1.el9            appstream      38 k
 libxcrypt             i686       4.4.18-3.el9              baseos        121 k
 libxcrypt-devel       i686       4.4.18-3.el9              appstream      28 k
 libxcrypt-devel       x86_64     4.4.18-3.el9              appstream      28 k
 make                  x86_64     1:4.3-7.el9               baseos        530 k
Installing weak dependencies:
 glibc-gconv-extra     i686       2.34-83.el9.7             baseos        1.6 M

Transaction Summary
================================================================================
Install  32 Packages

Total download size: 92 M
Installed size: 225 M
Downloading Packages:
(1/32): libxcrypt-4.4.18-3.el9.i686.rpm         369 kB/s | 121 kB     00:00
(2/32): make-4.3-7.el9.x86_64.rpm               1.3 MB/s | 530 kB     00:00
(3/32): libquadmath-11.4.1-2.1.el9.i686.rpm     577 kB/s | 245 kB     00:00
(4/32): libgcc-11.4.1-2.1.el9.i686.rpm          1.8 MB/s | 104 kB     00:00
(5/32): libquadmath-11.4.1-2.1.el9.x86_64.rpm   2.6 MB/s | 197 kB     00:00
(6/32): libgfortran-11.4.1-2.1.el9.i686.rpm     3.7 MB/s | 774 kB     00:00
(7/32): libgfortran-11.4.1-2.1.el9.x86_64.rpm   5.3 MB/s | 807 kB     00:00
(8/32): glibc-gconv-extra-2.34-83.el9.7.i686.rp 7.9 MB/s | 1.6 MB     00:00
(9/32): glibc-2.34-83.el9.7.i686.rpm            8.3 MB/s | 1.9 MB     00:00
(10/32): libxcrypt-devel-4.4.18-3.el9.x86_64.rp  97 kB/s |  28 kB     00:00
(11/32): libmpc-1.2.1-4.el9.x86_64.rpm          146 kB/s |  61 kB     00:00
(12/32): libxcrypt-devel-4.4.18-3.el9.i686.rpm   97 kB/s |  28 kB     00:00
(13/32): blas-3.9.0-10.el9.x86_64.rpm           1.0 MB/s | 479 kB     00:00
(14/32): lapack-3.9.0-10.el9.x86_64.rpm          10 MB/s | 8.5 MB     00:00
(15/32): kernel-headers-5.14.0-362.13.1.el9_3.x 6.9 MB/s | 6.4 MB     00:00
(16/32): blas-3.9.0-10.el9.i686.rpm             3.5 MB/s | 456 kB     00:00
(17/32): libquadmath-devel-11.4.1-2.1.el9.x86_6 389 kB/s |  38 kB     00:00
(18/32): lapack-3.9.0-10.el9.i686.rpm            12 MB/s | 8.5 MB     00:00
(19/32): gcc-gfortran-11.4.1-2.1.el9.x86_64.rpm  19 MB/s |  13 MB     00:00
(20/32): glibc-headers-2.34-83.el9.7.x86_64.rpm 3.7 MB/s | 450 kB     00:00
(21/32): cpp-11.4.1-2.1.el9.x86_64.rpm           16 MB/s |  11 MB     00:00
(22/32): glibc-devel-2.34-83.el9.7.x86_64.rpm   411 kB/s |  50 kB     00:00
(23/32): glibc-devel-2.34-83.el9.7.i686.rpm     514 kB/s |  50 kB     00:00
(24/32): lapack-devel-3.9.0-10.el9.x86_64.rpm   226 kB/s |  63 kB     00:00
(25/32): blas64_-3.9.0-10.el9.x86_64.rpm        1.3 MB/s | 209 kB     00:00
(26/32): gcc-11.4.1-2.1.el9.x86_64.rpm           19 MB/s |  32 MB     00:01
(27/32): blas-devel-3.9.0-10.el9.x86_64.rpm      48 kB/s |  15 kB     00:00
(28/32): lapack-devel-3.9.0-10.el9.i686.rpm     812 kB/s |  62 kB     00:00
(29/32): lapack64_-3.9.0-10.el9.x86_64.rpm      2.2 MB/s | 2.2 MB     00:00
(30/32): blas-devel-3.9.0-10.el9.i686.rpm        64 kB/s |  14 kB     00:00
(31/32): blas64-3.9.0-10.el9.x86_64.rpm         2.3 MB/s | 209 kB     00:00
(32/32): lapack64-3.9.0-10.el9.x86_64.rpm       5.6 MB/s | 2.2 MB     00:00
--------------------------------------------------------------------------------
Total                                            18 MB/s |  92 MB     00:05
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1
  Installing       : libgcc-11.4.1-2.1.el9.i686                            1/32
  Running scriptlet: libgcc-11.4.1-2.1.el9.i686                            1/32
  Installing       : kernel-headers-5.14.0-362.13.1.el9_3.x86_64           2/32
  Installing       : glibc-gconv-extra-2.34-83.el9.7.i686                  3/32
  Running scriptlet: glibc-gconv-extra-2.34-83.el9.7.i686                  3/32
  Running scriptlet: glibc-2.34-83.el9.7.i686                              4/32
  Installing       : glibc-2.34-83.el9.7.i686                              4/32
  Running scriptlet: glibc-2.34-83.el9.7.i686                              4/32
  Installing       : libquadmath-11.4.1-2.1.el9.x86_64                     5/32
  Installing       : libgfortran-11.4.1-2.1.el9.x86_64                     6/32
  Installing       : blas-3.9.0-10.el9.x86_64                              7/32
  Installing       : libmpc-1.2.1-4.el9.x86_64                             8/32
  Installing       : blas64_-3.9.0-10.el9.x86_64                           9/32
  Installing       : blas64-3.9.0-10.el9.x86_64                           10/32
  Installing       : glibc-headers-2.34-83.el9.7.x86_64                   11/32
  Installing       : libxcrypt-devel-4.4.18-3.el9.x86_64                  12/32
  Installing       : glibc-devel-2.34-83.el9.7.x86_64                     13/32
  Installing       : lapack64-3.9.0-10.el9.x86_64                         14/32
  Installing       : lapack64_-3.9.0-10.el9.x86_64                        15/32
  Installing       : cpp-11.4.1-2.1.el9.x86_64                            16/32
  Installing       : lapack-3.9.0-10.el9.x86_64                           17/32
  Installing       : make-1:4.3-7.el9.x86_64                              18/32
  Installing       : gcc-11.4.1-2.1.el9.x86_64                            19/32
  Installing       : libquadmath-devel-11.4.1-2.1.el9.x86_64              20/32
  Installing       : gcc-gfortran-11.4.1-2.1.el9.x86_64                   21/32
  Installing       : blas-devel-3.9.0-10.el9.x86_64                       22/32
  Installing       : libxcrypt-4.4.18-3.el9.i686                          23/32
  Installing       : glibc-devel-2.34-83.el9.7.i686                       24/32
  Installing       : libxcrypt-devel-4.4.18-3.el9.i686                    25/32
  Installing       : libquadmath-11.4.1-2.1.el9.i686                      26/32
  Installing       : lapack-devel-3.9.0-10.el9.x86_64                     27/32
  Installing       : libgfortran-11.4.1-2.1.el9.i686                      28/32
  Installing       : blas-3.9.0-10.el9.i686                               29/32
  Installing       : blas-devel-3.9.0-10.el9.i686                         30/32
  Installing       : lapack-3.9.0-10.el9.i686                             31/32
  Installing       : lapack-devel-3.9.0-10.el9.i686                       32/32
  Running scriptlet: lapack-devel-3.9.0-10.el9.i686                       32/32
  Verifying        : libxcrypt-4.4.18-3.el9.i686                           1/32
  Verifying        : make-1:4.3-7.el9.x86_64                               2/32
  Verifying        : libquadmath-11.4.1-2.1.el9.i686                       3/32
  Verifying        : libgfortran-11.4.1-2.1.el9.i686                       4/32
  Verifying        : libgcc-11.4.1-2.1.el9.i686                            5/32
  Verifying        : libquadmath-11.4.1-2.1.el9.x86_64                     6/32
  Verifying        : libgfortran-11.4.1-2.1.el9.x86_64                     7/32
  Verifying        : glibc-gconv-extra-2.34-83.el9.7.i686                  8/32
  Verifying        : glibc-2.34-83.el9.7.i686                              9/32
  Verifying        : libmpc-1.2.1-4.el9.x86_64                            10/32
  Verifying        : libxcrypt-devel-4.4.18-3.el9.x86_64                  11/32
  Verifying        : libxcrypt-devel-4.4.18-3.el9.i686                    12/32
  Verifying        : kernel-headers-5.14.0-362.13.1.el9_3.x86_64          13/32
  Verifying        : lapack-3.9.0-10.el9.x86_64                           14/32
  Verifying        : blas-3.9.0-10.el9.x86_64                             15/32
  Verifying        : lapack-3.9.0-10.el9.i686                             16/32
  Verifying        : blas-3.9.0-10.el9.i686                               17/32
  Verifying        : libquadmath-devel-11.4.1-2.1.el9.x86_64              18/32
  Verifying        : gcc-gfortran-11.4.1-2.1.el9.x86_64                   19/32
  Verifying        : gcc-11.4.1-2.1.el9.x86_64                            20/32
  Verifying        : cpp-11.4.1-2.1.el9.x86_64                            21/32
  Verifying        : glibc-headers-2.34-83.el9.7.x86_64                   22/32
  Verifying        : glibc-devel-2.34-83.el9.7.x86_64                     23/32
  Verifying        : glibc-devel-2.34-83.el9.7.i686                       24/32
  Verifying        : lapack64_-3.9.0-10.el9.x86_64                        25/32
  Verifying        : lapack-devel-3.9.0-10.el9.x86_64                     26/32
  Verifying        : blas64_-3.9.0-10.el9.x86_64                          27/32
  Verifying        : blas-devel-3.9.0-10.el9.x86_64                       28/32
  Verifying        : lapack-devel-3.9.0-10.el9.i686                       29/32
  Verifying        : blas-devel-3.9.0-10.el9.i686                         30/32
  Verifying        : lapack64-3.9.0-10.el9.x86_64                         31/32
  Verifying        : blas64-3.9.0-10.el9.x86_64                           32/32

Installed:
  blas-3.9.0-10.el9.i686
  blas-3.9.0-10.el9.x86_64
  blas-devel-3.9.0-10.el9.i686
  blas-devel-3.9.0-10.el9.x86_64
  blas64-3.9.0-10.el9.x86_64
  blas64_-3.9.0-10.el9.x86_64
  cpp-11.4.1-2.1.el9.x86_64
  gcc-11.4.1-2.1.el9.x86_64
  gcc-gfortran-11.4.1-2.1.el9.x86_64
  glibc-2.34-83.el9.7.i686
  glibc-devel-2.34-83.el9.7.i686
  glibc-devel-2.34-83.el9.7.x86_64
  glibc-gconv-extra-2.34-83.el9.7.i686
  glibc-headers-2.34-83.el9.7.x86_64
  kernel-headers-5.14.0-362.13.1.el9_3.x86_64
  lapack-3.9.0-10.el9.i686
  lapack-3.9.0-10.el9.x86_64
  lapack-devel-3.9.0-10.el9.i686
  lapack-devel-3.9.0-10.el9.x86_64
  lapack64-3.9.0-10.el9.x86_64
  lapack64_-3.9.0-10.el9.x86_64
  libgcc-11.4.1-2.1.el9.i686
  libgfortran-11.4.1-2.1.el9.i686
  libgfortran-11.4.1-2.1.el9.x86_64
  libmpc-1.2.1-4.el9.x86_64
  libquadmath-11.4.1-2.1.el9.i686
  libquadmath-11.4.1-2.1.el9.x86_64
  libquadmath-devel-11.4.1-2.1.el9.x86_64
  libxcrypt-4.4.18-3.el9.i686
  libxcrypt-devel-4.4.18-3.el9.i686
  libxcrypt-devel-4.4.18-3.el9.x86_64
  make-1:4.3-7.el9.x86_64

Complete!
[root@awx ~]# cat > main.f90 <<EOF
program bench

  implicit none

  real(8), allocatable :: mat(:,:), a(:,:)

  integer :: n

  n = 100

  ! Create random positive definite matrices
  write(6,'(3x,a,i8)') "I will create matrices of size",n

  allocate(a(n,n), mat(n,n))

  call random_number(a)

  call dgemm('n', 't', n, n, n, 1.d0, a, n, a, n, 0.d0, mat, n)

end program
EOF

# Compiling for 32 bit (i686) only
[root@awx ~]# gfortran -m32 -fallow-argument-mismatch main.f90 -lblas -llapack -o a1.out
[root@awx ~]# ./a1.out
   I will create matrices of size     100
[root@awx ~]# ldd a1.out
        linux-gate.so.1 (0xf7efe000)
        libblas.so.3 => /lib/libblas.so.3 (0xf7e7b000)
        liblapack.so.3 => /lib/liblapack.so.3 (0xf7600000)
        libgfortran.so.5 => /lib/libgfortran.so.5 (0xf7200000)
        libm.so.6 => /lib/libm.so.6 (0xf7db0000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf7d92000)
        libquadmath.so.0 => /lib/libquadmath.so.0 (0xf7d1b000)
        libc.so.6 => /lib/libc.so.6 (0xf7009000)
        /lib/ld-linux.so.2 (0xf7f00000)

# Compiling for 64 bit (x86_64) only
[root@awx ~]# gfortran -fallow-argument-mismatch main.f90 -lblas -llapack -o a2.out
[root@awx ~]# ./a2.out
   I will create matrices of size     100

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f6db4423860 in ???
#1  0x7f6db4422a05 in ???
#2  0x7f6db4054daf in ???
#3  0x7f6db402b587 in ???
#4  0x7f6db4fbd664 in ???
#5  0x4016fc in ???
#6  0x401740 in ???
#7  0x7f6db403feaf in ???
#8  0x7f6db403ff5f in ???
#9  0x401114 in ???
#10  0xffffffffffffffff in ???
Segmentation fault (core dumped)
[root@awx ~]# ldd a2.out
        linux-vdso.so.1 (0x00007fff8dd80000)
        libblas.so.3 => /lib64/libblas.so.3 (0x00007f66ec87d000)
        liblapack.so.3 => /lib64/liblapack.so.3 (0x00007f66ec000000)
        libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007f66ebc00000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f66ec7a2000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f66ec787000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f66ec73d000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f66eb800000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f66ec90e000)

# Fails also when asking for lapack64 and blas64
[root@awx ~]# gfortran -fallow-argument-mismatch main.f90 -lblas64 -llapack64 -o a3.out
[root@awx ~]# ./a3.out
   I will create matrices of size     100

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f6183a23860 in ???
#1  0x7f6183a22a05 in ???
#2  0x7f6183654daf in ???
#3  0x7f618362b587 in ???
#4  0x7f61845b1664 in ???
#5  0x4016fc in ???
#6  0x401740 in ???
#7  0x7f618363feaf in ???
#8  0x7f618363ff5f in ???
#9  0x401114 in ???
#10  0xffffffffffffffff in ???
Segmentation fault (core dumped)
[root@awx ~]# ldd a3.out
        linux-vdso.so.1 (0x00007ffc444e3000)
        libblas.so.3 => /lib64/libblas.so.3 (0x00007fac282f6000)
        liblapack.so.3 => /lib64/liblapack.so.3 (0x00007fac27a00000)
        libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007fac27600000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fac2821b000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fac28200000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007fac281b6000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fac27200000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fac28387000)

Several lapack packages are multilib and provide both x86_64 and i686 packages, if this is what you mean. With that said, below is what changed with lapack. Left side is 9.2. Right side is 9.3. No packages were removed. (Also note the i686 packages mentioned here)