Ports "in use" permanently, even after reboot

Rocky Linux General
1

I have a pretty strange issue on Rocky 9.2. I’ve scoured Stack Overflow and elsewhere, and haven’t found anything that gives me any usable insight or results.

Got a Node.JS Express app which listens on a few ports - both HTTP, HTTPS, and WebSocket. Trying to restart it, I got "Error: listen EADDRINUSE: address already in use 0.0.0.0:8000". It’s reproducible: If I then run the process specifying port 8001, it works. CTRL + C, re-run with port 8001 again: error. Re-run with port 8002: works the first time, error the second time.

Here’s what I’ve observed:

  • This only happens when I specify 0.0.0.0:8000. When I specify 8000 only, it binds only to IPv6, and re-starts without any issues.
  • At first I thought it was an issue with Podman, but then found that it occurs when running the process directly on the host.
  • netstat -na doesn’t output any lines with the afflicted ports (including w/ sudo).
  • lsof doesn’t output any lines with the afflicted ports (including w/ sudo).
  • ss doesn’t output any lines with the afflicted ports (including w/ sudo)
  • sudo sysctl -w net.ipv4.tcp_fin_timeout=1 + sudo sysctl -w net.ipv4.tcp_tw_reuse=1 doesn’t solve the issue
  • The really perplexing part: rebooting doesn’t free up the ports.

Where is this state being stored? How can I probe the kernel to figure out what’s going on under the hood to cause this issue?

Running the same software on an Ubuntu server, I don’t get this issue.

2

Well, I ended up figuring out the solution…

[mpixel@server NodeApp]$ ls -la
total 140
drwxr-xr-x.  11 mpixel mpixel  4096 Aug  8 18:58  .
drwxr-xr-x.   6 mpixel mpixel   181 Aug  1 13:28  ..
srwxr-xr-x.   1 root root     0 Aug  8 12:50  0.0.0.0:443
srwxr-xr-x.   1 root root     0 Aug  8 12:50  0.0.0.0:80
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 14:05  0.0.0.0:8000
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 14:23  0.0.0.0:8001
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 14:38  0.0.0.0:8002
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 14:05  0.0.0.0:8443
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 14:23  0.0.0.0:8444
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 14:38  0.0.0.0:8445
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 14:56  0.0.0.0:9000
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 14:56  0.0.0.0:9001
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 15:42  0.0.0.0:9002
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 15:42  0.0.0.0:9003
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 16:05  192.168.23.77:8000
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 16:05  192.168.23.77:8443
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 16:46 '*:443'
srwxr-xr-x.   1 mpixel mpixel     0 Aug  8 16:46 '*:80'
...

Evidently, when given “0.0.0.0:####” as a port, Express / NodeJS will create a Unix socket on the filesystem, relative to the current working directory, and doesn’t clean it up. These can be simply removed with rm.

1 Like