Trouble using gcc-toolset-15 on Rocky Linux 8: missing symbol (but only on some machines)

Rocky Linux Help & Support
1

I wrote an application that I uses c++20 features that needs to run on Rocky 8, so I installed the gcc-toolset-15 package. I do this on a development machine (this is a virtual machine but that’s not important for this discussion). I’ve deployed this application to several machines running Rocky 8, but it does not work everywhere, and I’m wondering why. On some machines the application works fine, but on others I get the following error message when I try to run it:

symbol lookup error: depc_bin: undefined symbol: _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC2Ev

I’ve looked at libstdc++.so.6 on both machines and they are identical (with identical sizes, package origin and version, and md5sum signatures). I don’t know why it works on some machines and not on others. Both sets of machines are completely up to date using “dnf update”.

I’ve even tried compiling the application on the machine that gives this error message, and the resulting binary also produces this error on that machine. This binary works fine on the machines where the other applications work. So it seems like there is something missing on the machines where it doesn’t work, but I do not know what.

Thank you in advance for any suggestions.

Eric

2

Do you know what line/function in your program causes the error to appear?

Have you tried isolating that single function into a test program that does just that one thing? Does it work if the function is isolated like that?

3

I have an executable built with GCC 15, (although on el9), and it shows:

$ nm example | grep ostringstream
                 U _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev@GLIBCXX_3.4.26
                 U _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEED1Ev@GLIBCXX_3.4.21
                 U _ZTTNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEE@GLIBCXX_3.4.21
                 U _ZTVNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEE@GLIBCXX_3.4.21

$ nm -C example | grep ostringstream
                 U std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream()@GLIBCXX_3.4.26
                 U std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()@GLIBCXX_3.4.21
                 U VTT for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21
                 U vtable for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21

What do you get from your depc_bin?

The libstdc++-8.5.0-28.el8_10.x86_64 package?

The library file does not have symbols, does it?
They are probably in separate package: https://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/debug/tree/Packages/l/libstdc%2B%2B-debuginfo-8.5.0-28.el8_10.x86_64.rpm

4

This error is not produced by my program. It’s produced by the system as it attempts to load and start the program. The missing symbol is part of the C++ standard library: it’s one of the constructors for std::ostringstream.

5

Here’s what I get with nm:

nm depc_bin | grep ostringstream
00000000005cbcb0 W _ZNKRSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEE3strEv
00000000005cbfe0 W _ZNKRSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEE3strEv
00000000005c70c0 W _ZNKSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEE4viewEv
00000000005c77d0 W _ZNKSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEE4viewEv
00000000005c9620 W _ZNOSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEE3strEv
00000000005cabe0 W _ZNOSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEE3strEv
00000000005c7100 W _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEE3strEONS_12basic_stringIcS2_S3_EE
00000000005c9d60 W _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1EONS_12basic_stringIcS2_S3_EESt13_Ios_Openmode
00000000005c8160 W _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1ESt13_Ios_OpenmodeRKS3_
                 U _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev
00000000005c98e0 W _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC2EONS_12basic_stringIcS2_S3_EESt13_Ios_Openmode
00000000005c8050 W _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC2ESt13_Ios_OpenmodeRKS3_
                 U _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC2Ev
                 U _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEED1Ev@GLIBCXX_3.4.21
                 U _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEED2Ev@GLIBCXX_3.4.21
00000000005c7810 W _ZNSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEE3strEONS_12basic_stringIwS2_S3_EE
00000000005cb160 W _ZNSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEEC1EONS_12basic_stringIwS2_S3_EESt13_Ios_Openmode
00000000005c8cc0 W _ZNSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEEC1ESt13_Ios_OpenmodeRKS3_
00000000005cae00 W _ZNSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEEC2EONS_12basic_stringIwS2_S3_EESt13_Ios_Openmode
00000000005c8bb0 W _ZNSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEEC2ESt13_Ios_OpenmodeRKS3_
00000000005fb580 V _ZTC5log_t0_NSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEE
00000000006ae4f8 V _ZTINSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEE@GLIBCXX_3.4.21
                 U _ZTTNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEE@GLIBCXX_3.4.21
                 U _ZTTNSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEEE@GLIBCXX_3.4.21
                 U _ZTVNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEE@GLIBCXX_3.4.21
                 U _ZTVNSt7__cxx1119basic_ostringstreamIwSt11char_traitsIwESaIwEEE@GLIBCXX_3.4.21

Here’s the second part:

nm -C depc_bin | grep ostringstream
00000000005cbcb0 W std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::str() const &
00000000005cbfe0 W std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::str() const &
00000000005c70c0 W std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::view() const
00000000005c77d0 W std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::view() const
00000000005c9620 W std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::str() &&
00000000005cabe0 W std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::str() &&
00000000005c7100 W std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::str(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)
00000000005c9d60 W std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::_Ios_Openmode)
00000000005c8160 W std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream(std::_Ios_Openmode, std::allocator<char> const&)
                 U std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream()
00000000005c98e0 W std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::_Ios_Openmode)
00000000005c8050 W std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream(std::_Ios_Openmode, std::allocator<char> const&)
                 U std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream()
                 U std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()@GLIBCXX_3.4.21
                 U std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()@GLIBCXX_3.4.21
00000000005c7810 W std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::str(std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >&&)
00000000005cb160 W std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::basic_ostringstream(std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >&&, std::_Ios_Openmode)
00000000005c8cc0 W std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::basic_ostringstream(std::_Ios_Openmode, std::allocator<wchar_t> const&)
00000000005cae00 W std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::basic_ostringstream(std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >&&, std::_Ios_Openmode)
00000000005c8bb0 W std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::basic_ostringstream(std::_Ios_Openmode, std::allocator<wchar_t> const&)
00000000005fb580 V construction vtable for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >-in-log_t
00000000006ae4f8 V typeinfo for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21
                 U VTT for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21
                 U VTT for std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4.21
                 U vtable for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21
                 U vtable for std::__cxx11::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4.21

Yes, the package is the same as what you show:

rpm -qf /lib64/libstdc++.so.6*
libstdc++-8.5.0-28.el8_10.x86_64
libstdc++-8.5.0-28.el8_10.x86_64

It’s just so strange especially because I get the same runtime error when I compile the program on the same machine that causes the error, yet that same executable works on the other machines that all work. This leads me to believe that there is something missing from the system, not the executable.

6

I get less with nm most likely because my code does not use std::ostringstream (explicitly).
The man nm tells:

“U” The symbol is undefined.
“V” The symbol is a weak object.
“W” The symbol is a weak symbol that has not been specifically tagged as a weak object symbol.

Therefore, the U symbols are of interest.

They (constructor, destructor, VTT, and vtable) seem to be twice there. The constructors:

The demangled versions:

The first symbol, which ends “C1Ev”, is even on my el9 binary.
The second, ending “C2Ev”, is the one that the libstd++ does not have.

Overall, it is odd that there are references two different(?) std::ostringstream.

On build the compiler does first create object files for each translation unit and then the linker combines the object files into executable. Part of the linking is to insert the (undefined) symbols that link to implementations in shared objects (the *.so libraries).

Therefore, the linker of GCC 15 must on that linking phase see the “C2Ev” symbol somewhere. Somewhere that the dynamic linker does not know of when the program is run.

It seems clear that the libstdc++.so.6 does not implement the “C2Ev”.
Therefore, the “C2Ev” must be somewhere else.
It might be in files of gcc-toolset-15.

The dynamic linker (see man ld.so) which on runtime connects shared objects to the executable, does look from “places”. The /etc/ld.so.conf.d/*.conf files do list those places.

Go to machine where the program “works”. Run:

strace ./depc_bin 2>&1 | grep lib | grep -v ENOENT

An example of what that gives for my program:

$ strace build/example 2>&1 | grep lib | grep -v ENOENT
newfstatat(AT_FDCWD, "/opt/rh/gcc-toolset-15/root/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/opt/rh/gcc-toolset-15/root/usr/lib64", {st_mode=S_IFDIR|0555, st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/opt/rh/gcc-toolset-15/root/usr/lib", {st_mode=S_IFDIR|0555, st_size=4096, ...}, 0) = 0
openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

Perhaps such strace for your program reveals a library file of interest.

Then peek in /etc/ld.so.conf.d/*.conf whether that mystery location is listed there (but not in the machines where symbol cannot be found).

7

Thank you for this suggestion. It let me to the fact that on the system where this doesn’t work, some of the libraries were in a shared location (not installed locally). When I removed that path from LD_LIBRARY_PATH, it worked. Now I know where the problem is. I appreciate your help! Thank you!

1 Like