Broken lapack/blas in Rocky Linux 9.3

Rocky Linux General
1

I’m lacking a minimal example, but wanted to reach out to the community. Maybe someone has similar issues or does know more about the lapack package for 9.3

Recently we upgraded in our group (working in the field of computational chemistry) from 9.2 to 9.3. Since then our codes using lapack routines are broken. The programs terminate with internal lapack errors indicating that wrong dimensions were passed to the routines. The problems disappears as soon as we link to a different lapack implementation e.g. Intel MKL (which is anyway the preferred choice)

Did someone experience something similar? Or does someone know more details about the lapack rpm?

2

Already a very simple example is broken

program bench

  implicit none

  real(8), allocatable :: mat(:,:), a(:,:)

  integer :: n

  n = 100

  ! Create random positive definite matrices
  write(6,'(3x,a,i8)') "I will create matrices of size",n

  allocate(a(n,n), mat(n,n))

  call random_number(a)

  call dgemm('n', 't', n, n, n, 1.d0, a, n, a, n, 0.d0, mat, n)

end program

Compiling it with gfortran an linking against the lapack installation

gfortran -fallow-argument-mismatch main.f90 -L /usr/lib64/ -lblas -L/usr/lib64 -llapack

produces an executable which seg-faults in the dgemm call. If the program is compiled an linked against the MKL it works

On the machine lapack-devel is installed.

EDIT:
I think I understand now what is going on. The lapack/blas version is compiled for a 8 byte integer and not a 4 byte integer, which is the fortran standard. There are flags for the compiler to convert all 4 byte integers into 8 byte integers, but maybe there is also a flag which I can use to link to the 4 byte integer version assuming that it is also shipped

3

Does anyone know why Rocky 9.2 has lapack64_ but Rocky 9.3 has lapack64_ and lapack64 (with no underscore)?

4

Hey, smith, thanks for the hint about the size of integers.

I run into this issue myself in Rocky 9.3 and couldn’t figure out the problem even after testing on Rocky 9.2 and seeing my program work just fine.

Then, I noticed that even though I am requesting “-llapack” at compile time, the run time error message says that the problem is inside lapack64!

The way I see it, if I say “-llapack”, it should pick up lapack; if I say “-llapack64”, it should then pick up lapack64…but that does not seem to be happening OR, lapack is also 64…and it shouldn’t be?

5

Hi gsal, I think 9.3 is missing a 32 bit integer version of lapack. I’m wondering if this is a Rocky issue or a Red Hat issue. Rocky should be bit by bit compatible to RHEL, but I don’t know if the rpms are maybe rebuild on the Rocky side and something went wrong in the lapack case.

6

To preface this post, this is all that changed with the RPM: import lapack-3.9.0-10.el9 (d6d9cc1a) · Commits · staging / rpms / lapack · GitLab - Not much changed here, except for the addition of explicit requires for some packages.

This is how pungi (our initial composing tool) determined what was needed to be in the repositories. The packages have always been built (you can actually verify that in the vault devel 9.2 repo, but pungi is what ultimately determines what will be in the repos by looking at dependencies of each package. Pungi is what decided it needed to be there.

If you look at the rpm description, they both say they are for 64bit integars, but lapack64_ has a symbol name suffix. see dnf repoquery -q lapack64 -i and dnf repoquery -q lapack64_ -i.

Below are the dependencies used to build lapack for 9.3.

================================================================================
 Package               Arch      Version                   Repository      Size
================================================================================
Installing:
 gcc-gfortran          x86_64    11.4.1-2.1.el9            yumrepofs_0     13 M
Installing dependencies:
 annobin               x86_64    12.12-1.el9               yumrepofs_0    977 k
 cpp                   x86_64    11.4.1-2.1.el9            yumrepofs_0     11 M
 gcc                   x86_64    11.4.1-2.1.el9            yumrepofs_0     32 M
 gcc-plugin-annobin    x86_64    11.4.1-2.1.el9            yumrepofs_0     48 k
 glibc-devel           x86_64    2.34-83.el9.7             yumrepofs_0     50 k
 glibc-headers         x86_64    2.34-83.el9.7             yumrepofs_0    450 k
 kernel-headers        x86_64    5.14.0-362.el9            yumrepofs_0    6.4 M
 libedit               x86_64    3.1-38.20210216cvs.el9    yumrepofs_0    103 k
 libgfortran           x86_64    11.4.1-2.1.el9            yumrepofs_0    807 k
 libmpc                x86_64    1.2.1-4.el9               yumrepofs_0     61 k
 libquadmath           x86_64    11.4.1-2.1.el9            yumrepofs_0    197 k
 libquadmath-devel     x86_64    11.4.1-2.1.el9            yumrepofs_0     38 k
 libxcrypt-devel       x86_64    4.4.18-3.el9              yumrepofs_0     28 k
 llvm-libs             x86_64    16.0.6-4.el9              yumrepofs_0     24 M

The build logs are here if you wish to look at them. (Warning, it is difficult to parse through the logs. We are working in the background to make the UI better.).

Whether or not that you look through the build logs, if you believe something is indeed wrong, we ask that you file a bug report with as much detail as possible. This includes debug data (e.g. gdb), related installed packages, example broken scripts/programs. Bug Tracker.

It works if you have glibc-devel.i686 (and other i686 packages) and you use -m32, which I’m sure goes against yours (and others) expectations.

[root@awx ~]# dnf install lapack lapack.i686 lapack64 lapack-devel.i686 lapack-devel glibc-devel.i686 -y
Last metadata expiration check: 3:02:57 ago on Thu 18 Jan 2024 11:04:47 PM MST.
Dependencies resolved.
================================================================================
 Package               Arch       Version                   Repository     Size
================================================================================
Installing:
 glibc-devel           i686       2.34-83.el9.7             appstream      50 k
 lapack                i686       3.9.0-10.el9              appstream     8.5 M
 lapack                x86_64     3.9.0-10.el9              appstream     8.5 M
 lapack-devel          i686       3.9.0-10.el9              crb            62 k
 lapack-devel          x86_64     3.9.0-10.el9              crb            63 k
 lapack64              x86_64     3.9.0-10.el9              crb           2.2 M
Installing dependencies:
 blas                  i686       3.9.0-10.el9              appstream     456 k
 blas                  x86_64     3.9.0-10.el9              appstream     479 k
 blas-devel            i686       3.9.0-10.el9              crb            14 k
 blas-devel            x86_64     3.9.0-10.el9              crb            15 k
 blas64                x86_64     3.9.0-10.el9              crb           209 k
 blas64_               x86_64     3.9.0-10.el9              crb           209 k
 cpp                   x86_64     11.4.1-2.1.el9            appstream      11 M
 gcc                   x86_64     11.4.1-2.1.el9            appstream      32 M
 gcc-gfortran          x86_64     11.4.1-2.1.el9            appstream      13 M
 glibc                 i686       2.34-83.el9.7             baseos        1.9 M
 glibc-devel           x86_64     2.34-83.el9.7             appstream      50 k
 glibc-headers         x86_64     2.34-83.el9.7             appstream     450 k
 kernel-headers        x86_64     5.14.0-362.13.1.el9_3     appstream     6.4 M
 lapack64_             x86_64     3.9.0-10.el9              crb           2.2 M
 libgcc                i686       11.4.1-2.1.el9            baseos        104 k
 libgfortran           i686       11.4.1-2.1.el9            baseos        774 k
 libgfortran           x86_64     11.4.1-2.1.el9            baseos        807 k
 libmpc                x86_64     1.2.1-4.el9               appstream      61 k
 libquadmath           i686       11.4.1-2.1.el9            baseos        245 k
 libquadmath           x86_64     11.4.1-2.1.el9            baseos        197 k
 libquadmath-devel     x86_64     11.4.1-2.1.el9            appstream      38 k
 libxcrypt             i686       4.4.18-3.el9              baseos        121 k
 libxcrypt-devel       i686       4.4.18-3.el9              appstream      28 k
 libxcrypt-devel       x86_64     4.4.18-3.el9              appstream      28 k
 make                  x86_64     1:4.3-7.el9               baseos        530 k
Installing weak dependencies:
 glibc-gconv-extra     i686       2.34-83.el9.7             baseos        1.6 M

Transaction Summary
================================================================================
Install  32 Packages

Total download size: 92 M
Installed size: 225 M
Downloading Packages:
(1/32): libxcrypt-4.4.18-3.el9.i686.rpm         369 kB/s | 121 kB     00:00
(2/32): make-4.3-7.el9.x86_64.rpm               1.3 MB/s | 530 kB     00:00
(3/32): libquadmath-11.4.1-2.1.el9.i686.rpm     577 kB/s | 245 kB     00:00
(4/32): libgcc-11.4.1-2.1.el9.i686.rpm          1.8 MB/s | 104 kB     00:00
(5/32): libquadmath-11.4.1-2.1.el9.x86_64.rpm   2.6 MB/s | 197 kB     00:00
(6/32): libgfortran-11.4.1-2.1.el9.i686.rpm     3.7 MB/s | 774 kB     00:00
(7/32): libgfortran-11.4.1-2.1.el9.x86_64.rpm   5.3 MB/s | 807 kB     00:00
(8/32): glibc-gconv-extra-2.34-83.el9.7.i686.rp 7.9 MB/s | 1.6 MB     00:00
(9/32): glibc-2.34-83.el9.7.i686.rpm            8.3 MB/s | 1.9 MB     00:00
(10/32): libxcrypt-devel-4.4.18-3.el9.x86_64.rp  97 kB/s |  28 kB     00:00
(11/32): libmpc-1.2.1-4.el9.x86_64.rpm          146 kB/s |  61 kB     00:00
(12/32): libxcrypt-devel-4.4.18-3.el9.i686.rpm   97 kB/s |  28 kB     00:00
(13/32): blas-3.9.0-10.el9.x86_64.rpm           1.0 MB/s | 479 kB     00:00
(14/32): lapack-3.9.0-10.el9.x86_64.rpm          10 MB/s | 8.5 MB     00:00
(15/32): kernel-headers-5.14.0-362.13.1.el9_3.x 6.9 MB/s | 6.4 MB     00:00
(16/32): blas-3.9.0-10.el9.i686.rpm             3.5 MB/s | 456 kB     00:00
(17/32): libquadmath-devel-11.4.1-2.1.el9.x86_6 389 kB/s |  38 kB     00:00
(18/32): lapack-3.9.0-10.el9.i686.rpm            12 MB/s | 8.5 MB     00:00
(19/32): gcc-gfortran-11.4.1-2.1.el9.x86_64.rpm  19 MB/s |  13 MB     00:00
(20/32): glibc-headers-2.34-83.el9.7.x86_64.rpm 3.7 MB/s | 450 kB     00:00
(21/32): cpp-11.4.1-2.1.el9.x86_64.rpm           16 MB/s |  11 MB     00:00
(22/32): glibc-devel-2.34-83.el9.7.x86_64.rpm   411 kB/s |  50 kB     00:00
(23/32): glibc-devel-2.34-83.el9.7.i686.rpm     514 kB/s |  50 kB     00:00
(24/32): lapack-devel-3.9.0-10.el9.x86_64.rpm   226 kB/s |  63 kB     00:00
(25/32): blas64_-3.9.0-10.el9.x86_64.rpm        1.3 MB/s | 209 kB     00:00
(26/32): gcc-11.4.1-2.1.el9.x86_64.rpm           19 MB/s |  32 MB     00:01
(27/32): blas-devel-3.9.0-10.el9.x86_64.rpm      48 kB/s |  15 kB     00:00
(28/32): lapack-devel-3.9.0-10.el9.i686.rpm     812 kB/s |  62 kB     00:00
(29/32): lapack64_-3.9.0-10.el9.x86_64.rpm      2.2 MB/s | 2.2 MB     00:00
(30/32): blas-devel-3.9.0-10.el9.i686.rpm        64 kB/s |  14 kB     00:00
(31/32): blas64-3.9.0-10.el9.x86_64.rpm         2.3 MB/s | 209 kB     00:00
(32/32): lapack64-3.9.0-10.el9.x86_64.rpm       5.6 MB/s | 2.2 MB     00:00
--------------------------------------------------------------------------------
Total                                            18 MB/s |  92 MB     00:05
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1
  Installing       : libgcc-11.4.1-2.1.el9.i686                            1/32
  Running scriptlet: libgcc-11.4.1-2.1.el9.i686                            1/32
  Installing       : kernel-headers-5.14.0-362.13.1.el9_3.x86_64           2/32
  Installing       : glibc-gconv-extra-2.34-83.el9.7.i686                  3/32
  Running scriptlet: glibc-gconv-extra-2.34-83.el9.7.i686                  3/32
  Running scriptlet: glibc-2.34-83.el9.7.i686                              4/32
  Installing       : glibc-2.34-83.el9.7.i686                              4/32
  Running scriptlet: glibc-2.34-83.el9.7.i686                              4/32
  Installing       : libquadmath-11.4.1-2.1.el9.x86_64                     5/32
  Installing       : libgfortran-11.4.1-2.1.el9.x86_64                     6/32
  Installing       : blas-3.9.0-10.el9.x86_64                              7/32
  Installing       : libmpc-1.2.1-4.el9.x86_64                             8/32
  Installing       : blas64_-3.9.0-10.el9.x86_64                           9/32
  Installing       : blas64-3.9.0-10.el9.x86_64                           10/32
  Installing       : glibc-headers-2.34-83.el9.7.x86_64                   11/32
  Installing       : libxcrypt-devel-4.4.18-3.el9.x86_64                  12/32
  Installing       : glibc-devel-2.34-83.el9.7.x86_64                     13/32
  Installing       : lapack64-3.9.0-10.el9.x86_64                         14/32
  Installing       : lapack64_-3.9.0-10.el9.x86_64                        15/32
  Installing       : cpp-11.4.1-2.1.el9.x86_64                            16/32
  Installing       : lapack-3.9.0-10.el9.x86_64                           17/32
  Installing       : make-1:4.3-7.el9.x86_64                              18/32
  Installing       : gcc-11.4.1-2.1.el9.x86_64                            19/32
  Installing       : libquadmath-devel-11.4.1-2.1.el9.x86_64              20/32
  Installing       : gcc-gfortran-11.4.1-2.1.el9.x86_64                   21/32
  Installing       : blas-devel-3.9.0-10.el9.x86_64                       22/32
  Installing       : libxcrypt-4.4.18-3.el9.i686                          23/32
  Installing       : glibc-devel-2.34-83.el9.7.i686                       24/32
  Installing       : libxcrypt-devel-4.4.18-3.el9.i686                    25/32
  Installing       : libquadmath-11.4.1-2.1.el9.i686                      26/32
  Installing       : lapack-devel-3.9.0-10.el9.x86_64                     27/32
  Installing       : libgfortran-11.4.1-2.1.el9.i686                      28/32
  Installing       : blas-3.9.0-10.el9.i686                               29/32
  Installing       : blas-devel-3.9.0-10.el9.i686                         30/32
  Installing       : lapack-3.9.0-10.el9.i686                             31/32
  Installing       : lapack-devel-3.9.0-10.el9.i686                       32/32
  Running scriptlet: lapack-devel-3.9.0-10.el9.i686                       32/32
  Verifying        : libxcrypt-4.4.18-3.el9.i686                           1/32
  Verifying        : make-1:4.3-7.el9.x86_64                               2/32
  Verifying        : libquadmath-11.4.1-2.1.el9.i686                       3/32
  Verifying        : libgfortran-11.4.1-2.1.el9.i686                       4/32
  Verifying        : libgcc-11.4.1-2.1.el9.i686                            5/32
  Verifying        : libquadmath-11.4.1-2.1.el9.x86_64                     6/32
  Verifying        : libgfortran-11.4.1-2.1.el9.x86_64                     7/32
  Verifying        : glibc-gconv-extra-2.34-83.el9.7.i686                  8/32
  Verifying        : glibc-2.34-83.el9.7.i686                              9/32
  Verifying        : libmpc-1.2.1-4.el9.x86_64                            10/32
  Verifying        : libxcrypt-devel-4.4.18-3.el9.x86_64                  11/32
  Verifying        : libxcrypt-devel-4.4.18-3.el9.i686                    12/32
  Verifying        : kernel-headers-5.14.0-362.13.1.el9_3.x86_64          13/32
  Verifying        : lapack-3.9.0-10.el9.x86_64                           14/32
  Verifying        : blas-3.9.0-10.el9.x86_64                             15/32
  Verifying        : lapack-3.9.0-10.el9.i686                             16/32
  Verifying        : blas-3.9.0-10.el9.i686                               17/32
  Verifying        : libquadmath-devel-11.4.1-2.1.el9.x86_64              18/32
  Verifying        : gcc-gfortran-11.4.1-2.1.el9.x86_64                   19/32
  Verifying        : gcc-11.4.1-2.1.el9.x86_64                            20/32
  Verifying        : cpp-11.4.1-2.1.el9.x86_64                            21/32
  Verifying        : glibc-headers-2.34-83.el9.7.x86_64                   22/32
  Verifying        : glibc-devel-2.34-83.el9.7.x86_64                     23/32
  Verifying        : glibc-devel-2.34-83.el9.7.i686                       24/32
  Verifying        : lapack64_-3.9.0-10.el9.x86_64                        25/32
  Verifying        : lapack-devel-3.9.0-10.el9.x86_64                     26/32
  Verifying        : blas64_-3.9.0-10.el9.x86_64                          27/32
  Verifying        : blas-devel-3.9.0-10.el9.x86_64                       28/32
  Verifying        : lapack-devel-3.9.0-10.el9.i686                       29/32
  Verifying        : blas-devel-3.9.0-10.el9.i686                         30/32
  Verifying        : lapack64-3.9.0-10.el9.x86_64                         31/32
  Verifying        : blas64-3.9.0-10.el9.x86_64                           32/32

Installed:
  blas-3.9.0-10.el9.i686
  blas-3.9.0-10.el9.x86_64
  blas-devel-3.9.0-10.el9.i686
  blas-devel-3.9.0-10.el9.x86_64
  blas64-3.9.0-10.el9.x86_64
  blas64_-3.9.0-10.el9.x86_64
  cpp-11.4.1-2.1.el9.x86_64
  gcc-11.4.1-2.1.el9.x86_64
  gcc-gfortran-11.4.1-2.1.el9.x86_64
  glibc-2.34-83.el9.7.i686
  glibc-devel-2.34-83.el9.7.i686
  glibc-devel-2.34-83.el9.7.x86_64
  glibc-gconv-extra-2.34-83.el9.7.i686
  glibc-headers-2.34-83.el9.7.x86_64
  kernel-headers-5.14.0-362.13.1.el9_3.x86_64
  lapack-3.9.0-10.el9.i686
  lapack-3.9.0-10.el9.x86_64
  lapack-devel-3.9.0-10.el9.i686
  lapack-devel-3.9.0-10.el9.x86_64
  lapack64-3.9.0-10.el9.x86_64
  lapack64_-3.9.0-10.el9.x86_64
  libgcc-11.4.1-2.1.el9.i686
  libgfortran-11.4.1-2.1.el9.i686
  libgfortran-11.4.1-2.1.el9.x86_64
  libmpc-1.2.1-4.el9.x86_64
  libquadmath-11.4.1-2.1.el9.i686
  libquadmath-11.4.1-2.1.el9.x86_64
  libquadmath-devel-11.4.1-2.1.el9.x86_64
  libxcrypt-4.4.18-3.el9.i686
  libxcrypt-devel-4.4.18-3.el9.i686
  libxcrypt-devel-4.4.18-3.el9.x86_64
  make-1:4.3-7.el9.x86_64

Complete!
[root@awx ~]# cat > main.f90 <<EOF
program bench

  implicit none

  real(8), allocatable :: mat(:,:), a(:,:)

  integer :: n

  n = 100

  ! Create random positive definite matrices
  write(6,'(3x,a,i8)') "I will create matrices of size",n

  allocate(a(n,n), mat(n,n))

  call random_number(a)

  call dgemm('n', 't', n, n, n, 1.d0, a, n, a, n, 0.d0, mat, n)

end program
EOF

# Compiling for 32 bit (i686) only
[root@awx ~]# gfortran -m32 -fallow-argument-mismatch main.f90 -lblas -llapack -o a1.out
[root@awx ~]# ./a1.out
   I will create matrices of size     100
[root@awx ~]# ldd a1.out
        linux-gate.so.1 (0xf7efe000)
        libblas.so.3 => /lib/libblas.so.3 (0xf7e7b000)
        liblapack.so.3 => /lib/liblapack.so.3 (0xf7600000)
        libgfortran.so.5 => /lib/libgfortran.so.5 (0xf7200000)
        libm.so.6 => /lib/libm.so.6 (0xf7db0000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf7d92000)
        libquadmath.so.0 => /lib/libquadmath.so.0 (0xf7d1b000)
        libc.so.6 => /lib/libc.so.6 (0xf7009000)
        /lib/ld-linux.so.2 (0xf7f00000)

# Compiling for 64 bit (x86_64) only
[root@awx ~]# gfortran -fallow-argument-mismatch main.f90 -lblas -llapack -o a2.out
[root@awx ~]# ./a2.out
   I will create matrices of size     100

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f6db4423860 in ???
#1  0x7f6db4422a05 in ???
#2  0x7f6db4054daf in ???
#3  0x7f6db402b587 in ???
#4  0x7f6db4fbd664 in ???
#5  0x4016fc in ???
#6  0x401740 in ???
#7  0x7f6db403feaf in ???
#8  0x7f6db403ff5f in ???
#9  0x401114 in ???
#10  0xffffffffffffffff in ???
Segmentation fault (core dumped)
[root@awx ~]# ldd a2.out
        linux-vdso.so.1 (0x00007fff8dd80000)
        libblas.so.3 => /lib64/libblas.so.3 (0x00007f66ec87d000)
        liblapack.so.3 => /lib64/liblapack.so.3 (0x00007f66ec000000)
        libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007f66ebc00000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f66ec7a2000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f66ec787000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f66ec73d000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f66eb800000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f66ec90e000)

# Fails also when asking for lapack64 and blas64
[root@awx ~]# gfortran -fallow-argument-mismatch main.f90 -lblas64 -llapack64 -o a3.out
[root@awx ~]# ./a3.out
   I will create matrices of size     100

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7f6183a23860 in ???
#1  0x7f6183a22a05 in ???
#2  0x7f6183654daf in ???
#3  0x7f618362b587 in ???
#4  0x7f61845b1664 in ???
#5  0x4016fc in ???
#6  0x401740 in ???
#7  0x7f618363feaf in ???
#8  0x7f618363ff5f in ???
#9  0x401114 in ???
#10  0xffffffffffffffff in ???
Segmentation fault (core dumped)
[root@awx ~]# ldd a3.out
        linux-vdso.so.1 (0x00007ffc444e3000)
        libblas.so.3 => /lib64/libblas.so.3 (0x00007fac282f6000)
        liblapack.so.3 => /lib64/liblapack.so.3 (0x00007fac27a00000)
        libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00007fac27600000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fac2821b000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fac28200000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007fac281b6000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fac27200000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fac28387000)

Several lapack packages are multilib and provide both x86_64 and i686 packages, if this is what you mean. With that said, below is what changed with lapack. Left side is 9.2. Right side is 9.3. No packages were removed. (Also note the i686 packages mentioned here)

image
image1911×1005 54.4 KB

7

Thank you nazunalika for this detailed post. I think I will file a bug report, although there is a way to link to the correct lapack version. It behaves at least not as I expect it and also many codes we are using. Still, it is good to know that there is an additional way to make them run again for the mean time

1 Like
8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.