AWS EC2 - Rocky Linux 9 - logs flooded with "state changed new lease"

Hi everyone
on a brand-new Rocky Linux 9 AWS EC2 instance (official image Rocky-9-EC2-Base-9.5-20241118.0.aarch64-0d51926d-1cd1-4223-bda9-346993accc16 AMI ami-0793f1de745d09710) I keep getting the logs flooded by NetworkManager with lines like the below

dhcp6 (eth0): state changed new lease, address=xyz

Any hint about this? Please note: leveraging docker and IPv6.

thanks a lot
nicola

It would suggest dhcp6 is attempting to get an address and not getting one or getting a different one regularly for some reason. If you are not using dhcp6, perhaps edit the network connection to disable IPv6 by setting it to manual?

Or perhaps ask AWS what is going on with dhcp6 at their end, as it could well be a problem with their environment.

thanks @iwalker - IPv6 is mandatory for my purposes (avoiding AWS IPv4 charges, FYI). Any other hint apart from raising an AWS support?

Sadly not, I have a server with OVH and I don’t have these messages, so would suggest it’s a problem with their configuration.

1 Like

You can check the lease expire date for the DHCP6 leases you get. May be they are really short (instead of the common default of 24h).
You can try this ( replace DHCP4 with DHCP6 and “enp1s0” with “eth0”):

[root@localhost ~]# nmcli -f DHCP4 dev show enp1s0 |grep -i exp
DHCP4.OPTION[6]:                        expiry = 1738258354
[root@localhost ~]# date -d "@1738258354"
Thu Jan 30 06:32:34 PM CET 2025
[root@localhost ~]#

If you cannot find the reason for those messages but want to suppress them, you can reconfigure rsyslogd via configuration file /etc/rsyslog.conf to not log the “info” level of facility “daemon” to /var/log/messages anymore.
Rocky 9 has this line:
*.info;mail.none;authpriv.none;cron.none /var/log/messages
To ignore info level for all daemon processes ( including NetworkManager)
and start logging with next higher level “notice” this might work:
*.info;mail.none;authpriv.none;cron.none;daemon.notice /var/log/messages

( appending “daemon.notice” should overwrite “*.info” from the first entry. There might be better ways to do this with advanced rsyslog filtering, e.g. by only ignoring messages with text “changed new lease”)

In my daily work we see many different reasons why processes are spamming our log files and often we cannot change that behaviour. So a common thing we do is to enable “logrotate” to run every day. This way our logs are rotated and compressed.

Thanks a lot for your feedback! don’t know if something is truly broken or not, but the IPv6 output from

nmcli -f DHCP6 dev show eth0

is really short

DHCP6.OPTION[1]:                        dhcp6_client_id = xyz
DHCP6.OPTION[2]:                        iaid = 2b:50:67:35
DHCP6.OPTION[3]:                        ip6_address = xxx

Is this supposed to be?? The IPv4 is 17 lines long indeed and having the expiry field

expiry = 1738263024

BTW just replicated this with a brand new EC2 instance just to try out: issue happens immediately at first boot, without any docker or anything else installed. Therefore I assume it’s something embedded in the official Rocky image when IPv6 is used.

thanks
nicola

If the lease is short, then the lease is configured on the DHCP server - which is obviously what AWS manages. Your Rocky image is just a DHCP client that obtains an address from that server. This is why I believe the Rocky image isn’t the problem here, but the DHCP server providing the address. You really need to be talking to AWS about this.

It would help seeing the data/time from your logs as well to see how often the messages are appearing in the log. Your screenshot above doesn’t include that.

The timestamps in the log look like epoch numbers, so there is a message every 60-70 seconds i think …

Yeah good catch, didn’t realise those numbers to think about checking them, but on epochconvert.com it checks out:

1738143713.6562 09:41:53.656
1738143771.2992 09:42:51.299

I confirm

  • it’s around every minute
  • this happens with a brand new official Rocky-provided ABI image (just fire up the instance and here it is)
  • I tried to raise a bug to Rocky but I cannot even access the register page for https://bugs.rockylinux.org/ - asked in chats.rockylinux.org if it’s a temporary issue

I have no way to contact AWS since don’t have a support plan at the moment (still evaluating before making the switch).

All in all, I see no light at the end of the tunnel because I’m not aware of DHCP settings I could access on AWS side :sweat_smile:

Yes true, it would require AWS to check it. I’m curious, if you can test another image other than Rocky, say CentOS or Alma if they have it and see if they also have the same problem with Network Manager and IPv6? If they do it would suggest it’s not specific to Rocky as such if other images suffer the same problem. If they don’t suffer the same problem, then it would suggest a problem with the Rocky one.

I confirm the same happens with AlmaLinux as well (AMI AlmaLinux OS 9.5.20241122 aarch64-2d219cc1-aa44-4a1e-b6fe-258d4ebd3cdb)

Maybe this helps:

I think you’re right (I even remember I stumbled upon that page but discarded on the spot given the Nitro reference). The relevant part is

By default, the IPv6 preferred lease time is 140 seconds and DHCP lease renewals for IPv6 address assignments occur at the halfway mark of 70 seconds

therefore it makes sense seeing logs around every minute or so (precisely, every 70 seconds).

This is for sure something not clear from AWS console VPC -> DHCP option sets where no such default 140s default lease time is reported and cannot be changed at all in the existing DHCP config set (no edit button at all):

but you can set the IPv6 DHCP lease time for a brand new DHCP config set!

Due to this, I think we can consider closed the issue!

thanks again @iwalker @Hedges

1 Like

Yeah you prob cannot edit the default one, but create your own to override the default.