Ookla speedtest issues

I know this is not a Rocky problem, but I’m running it on R8 so that’s my excuse :slight_smile:

I have the ookla speedtest CLI binary from https://packagecloud.io/ookla/speedtest-cli/el/8/$basearch

Name        : speedtest
Version     : 1.2.0.84_1.ea6b6773cf
Release     : 1

Every 15 minutes I call

speedtest -s $SERVER -f csv

(where, at the moment, $SERVER is 0514 # Optimum Online - New York, NY)

I hard-code the server rather than letting ookla pick its own so I can get some consistency of results.

This has worked well for years, except when a server leaves; then I have to pick a new one.

But these past two weeks I’ve been getting a timeout error with

ConfigurationError - Could not retrieve or read configuration (Configuration)

Not every time. Last 24 hours (96 calls) it happened 31 times.

It’s not evenly distributed; sometimes 3 tests in succession fail, othertimes it can go hours without problem.

Is anyone else seeing this?

I’ve had different issues on different distros when installing the packaged version of speedtest (I think it was Debian), but on those systems I used pip to install it instead after removing the package first. Perhaps try using the one installed with pip to see if it’s more stable? That would at least rule it out.

I think I’ve worked out what’s going on.

For some reason the ookla code has started using IPv6 by default, even though gai.conf is set to prefer IPv4. (eg telnet speedtest.net 80 will use IPv4). Since I don’t have native IPv6 I have to use tunnelbroker.

It works well, but I’m guessing all these speedtests might be being considered abuse and it’s blocking traffic. Or else speedtest.net doesn’t work properly with it.

Hmm. I don’t want to disable IPv6 permanently, but I want IPv4 to always be a priority. I thought gai.conf would do this

precedence ::ffff:0:0/96  100
scopev4 ::ffff:0.0.0.0/96       14

I looked at the python code (pip3 install -user speedtest-cli) and it does work… but the results are slower than the ookla binary. Not sure why, but it consistently is.

I wonder if there’s a better way of forcing ipv4 to have priority, or if there’s a wrapper (maybe using LD_PRELOAD) which hides the IPv6 address from teh app!

Well, that’s annoying… I used code from https://gist.githubusercontent.com/winny-/7367518/raw/f098ef540c7ebbb9e148055365cec4205c4484c9/force-inet4-or-inet6.c and it works with telnet (eg LD_PRELOAD=$PWD/force_ipv6.so telnet speedtest.net 80 uses IPv6).

But it doesn’t work with the ookla binary. Because, of course, it’s a static binary, so we can’t preload. Doh. This is getting annoying!

Just been testing on Rocky 8. I installed this package:

dnf install speedtest-cli

only I have different commands than the ones you are using, where did you install the package from? Mine is coming from EPEL.

[root@rocky8 ~]# dnf info speedtest-cli
Last metadata expiration check: 1:50:06 ago on Tue 25 Apr 2023 06:29:47 AM CEST.
Installed Packages
Name         : speedtest-cli
Version      : 2.1.3
Release      : 1.el8
Architecture : noarch
Size         : 92 k
Source       : speedtest-cli-2.1.3-1.el8.src.rpm
Repository   : @System
From repo    : epel

When I run it the parameters are specified differently:

speedtest-cli --server xxx

with xxx being the ID. Despite getting a valid ID from the --list command, the first few times I tried it I got:

ERROR: No matched servers: 338

however after persisting and running a few times eventually it did find it, and then consistently started to work. There are odd occasions when it glitches and doesn’t find it, because the --list command doesn’t always return the same list and sometimes doesn’t include the ID I’m wanting to use. This could be looped though until it finds it and runs the test successfully.

My speeds using this package are consistent.

The version from EPEL is the python 3rd party version. The official ookla version comes from Speedtest CLI: Internet speed test for the command line

They have different command line options, just to confuse things!

Curiously the official binary doesn’t check the requested server against the returned list.

So I have a work-around, that uses namespaces. Basically I create a namespace that only has an IPv4 address and NAT that to my LAN. So anything running in that namespace will think it only has an IPv4 address.

In my case I have br-lan as a bridge for my LAN access so the setup for this namespace would be


ip netns del ip4only
ip netns add ip4only
ip link add ip4only-root type veth peer name ip4only-ns
ip link set ip4only-ns netns ip4only
ip addr add 192.168.200.1/24 dev ip4only-root
ip link set ip4only-root up

ip netns exec ip4only ip addr add 192.168.200.2/24 dev ip4only-ns
ip netns exec ip4only ip link set ip4only-ns up
ip netns exec ip4only ip link set lo up
ip netns exec ip4only ip route add default via 192.168.200.1

echo 1 > /proc/sys/net/ipv4/ip_forward

# Flush forward rules, policy DROP by default.
iptables -P FORWARD DROP
iptables -F FORWARD

# Flush nat rules.
iptables -t nat -F

# Enable masquerading of 192.168.200.0
iptables -t nat -A POSTROUTING -s 192.168.200.0/255.255.255.0 -o br-lan -j MASQUERADE

# Allow forwarding between br-lan and ip4only-root
iptables -A FORWARD -i br-lan -o ip4only-root -j ACCEPT
iptables -A FORWARD -o br-lan -i ip4only-root -j ACCEPT

So now we can see

%  sudo ip netns exec ip4only ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
21: ip4only-ns@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 86:58:d8:c4:b7:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.200.2/24 scope global ip4only-ns
       valid_lft forever preferred_lft forever
    inet6 fe80::8458:d8ff:fec4:b747/64 scope link 
       valid_lft forever preferred_lft forever

Yes it does look like there are inet6 addresses but these are link-local so aren’t used.

Now I don’t want to use sudo each time, so I created a wrapper that enters the name space, and uses capabilities for the permissions.

% cat ip4only.c
// ip4only
//   gcc -o ip4only ip4only.c
//   sudo setcap cap_sys_admin+ep ./ip4only
//
// Now we can do "ip4only command" (eg "ip4only ip addr")
// and it will run in the ip4only network namespace

// We need this for setns()
#define _GNU_SOURCE

#include <fcntl.h>
#include <sched.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>

#define errExit(msg) { perror(msg); exit(EXIT_FAILURE); }

int main(int argc, char *argv[])
{
  int fd;

  if (argc < 2)
  {
      fprintf(stderr, "%s cmd args...\n", argv[0]);
      exit(EXIT_FAILURE);
  }

  // To set a namespace we need to have an open file handle.
  // Network namespaces live in /var/run/ns so that's easy
  
  fd = open("/var/run/netns/ip4only", O_RDONLY);
  if (fd == -1)
    errExit("open");

  // Join the namespace
  if (setns(fd, 0) == -1)
    errExit("setns");

  // Run the specified command
  execvp(argv[1], &argv[1]);

  // If we got here, there's an error!
  errExit("execvp");
}

So now I can do

% ip4only speedtest
 
   Speedtest by Ookla

      Server: i3D.net - Newark, NJ (id: 43263)
         ISP: Verizon Fios

And we can see it’s picking up my ISP’s IPv4 address and not HE Tunnel.

This wrapper might be useful for other purposes as well!