Sorry. Once again a problem with repos:
cannot sync appstream repo for rocky8
A file located at the url http://dl.rockylinux.org/pub/rocky/8/AppStream/x86_64/os/Packages/j/java-17-openjdk-headless-17.0.5.0.8-1.el8_7.x86_64.rpm failed validation due to checksum.
Hello Rocky-team:
Can someone help to fix the problem in the above repo? My problem is, that I actually cannot deploy Rocky 8.7 to my people.
We are syncing the repository via Red Hat satellite server. The local repo is only created if every rpm can be found. And so I actually cannot provide Appstream and Baseos
I am not able to replicate your issue. I was able to run a full dnf reposync for the appstream repo and received no checksum errors.
[root@router j]# sha256sum appstream/Packages/j/java-17-openjdk-headless-17.0.5.0.8-1.el8_7.x86_64.rpm
7ffee64f4d6987f5a2ec5be9980b2066dd61f7f865d5e81ee88646d13a63dd41 appstream/Packages/j/java-17-openjdk-headless-17.0.5.0.8-1.el8_7.x86_64.rpm
<name>java-17-openjdk-headless</name>
<arch>x86_64</arch>
<version epoch="1" ver="17.0.5.0.8" rel="1.el8_7"/>
<checksum type="sha256" pkgid="YES">7ffee64f4d6987f5a2ec5be9980b2066dd61f7f865d5e81ee88646d13a63dd41</checksum>
<summary>OpenJDK 17 Headless Runtime Environment</summary>
<description>The OpenJDK 17 runtime environment without audio and video support.</description>
<packager>infrastructure@rockylinux.org</packager>
<url>http://openjdk.java.net/</url>
<time file="1667911950" build="1667910742"/>
<size package="45225116" installed="205225840" archive="191617544"/>
<location href="Packages/j/java-17-openjdk-headless-17.0.5.0.8-1.el8_7.x86_64.rpm"/>
I can’t download python3-dnf-plugin-versionlock-4.0.21-14.1.el8.noarch.rpm, I get a 503 Service Unavailable:
$ curl -v http://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-dnf-plugin-versionlock-4.0.21-14.1.el8.noarch.rpm
* Trying 146.75.118.132...
* TCP_NODELAY set
* Connected to download.rockylinux.org (146.75.118.132) port 80 (#0)
> GET /pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-dnf-plugin-versionlock-4.0.21-14.1.el8.noarch.rpm HTTP/1.1
> Host: download.rockylinux.org
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Connection: keep-alive
< Content-Length: 107
< Content-Type: text/html
< Fastly-Restarts: 2
< Via: 1.1 varnish, 1.1 varnish
< Cache-Control: max-age=11059200, public, stale-while-revalidate=86400, stale-if-error=259200
< Accept-Ranges: bytes
< Date: Fri, 25 Nov 2022 09:34:07 GMT
< Age: 826219
< X-Served-By: cache-chi-kigq8000033-CHI, cache-fra-eddf8230057-FRA
< X-Cache: MISS, HIT
< X-Cache-Hits: 0, 1
< X-Timer: S1669368847.352553,VS0,VE1
<
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
$ dig download.rockylinux.org
; <<>> DiG 9.11.36-RedHat-9.11.36-5.el8_7.2 <<>> download.rockylinux.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27111
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; MBZ: 0x0005, udp: 512
;; QUESTION SECTION:
;download.rockylinux.org. IN A
;; ANSWER SECTION:
download.rockylinux.org. 5 IN CNAME dualstack.dl.map.rockylinux.org.
dualstack.dl.map.rockylinux.org. 5 IN CNAME rockylinux.map.fastly.net.
rockylinux.map.fastly.net. 5 IN A 146.75.118.132
;; Query time: 18 msec
;; SERVER: 192.168.230.2#53(192.168.230.2)
;; WHEN: Fri Nov 25 10:37:00 CET 2022
;; MSG SIZE rcvd: 138
[label@sani tmp]$ curl -v http://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-dnf-plugin-versionlock-4.0.21-14.1.el8.noarch.rpm
* Trying 199.232.194.132:80...
* Connected to download.rockylinux.org (199.232.194.132) port 80 (#0)
> GET /pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-dnf-plugin-versionlock-4.0.21-14.1.el8.noarch.rpm HTTP/1.1
> Host: download.rockylinux.org
> User-Agent: curl/7.85.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: keep-alive
< Content-Length: 64792
< Content-Type: application/x-redhat-package-manager
< Server: nginx
< Last-Modified: Tue, 08 Nov 2022 10:20:29 GMT
< ETag: "636a2d6d-fd18"
< Via: 1.1 varnish, 1.1 varnish
< Cache-Control: max-age=11059200, public, stale-while-revalidate=86400, stale-if-error=259200
< Accept-Ranges: bytes
< Date: Fri, 25 Nov 2022 17:37:26 GMT
< Age: 821912
< X-Served-By: cache-chi-kigq8000033-CHI, cache-phx12426-PHX
< X-Cache: MISS, HIT
< X-Cache-Hits: 0, 1
< X-Timer: S1669397846.015319,VS0,VE5
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* Closing connection 0
I’m not able to reproduce. Pinging @neil
503 errors tend to come and go, so people get different results. It’s usually when a proxy sits in front of the real server; the proxy stays up, but the real server goes down or is too slow to respond.
It depends what IP is returned for download.rockylinux.org (=rockylinux.map.fastly.net). In my case the DNS returns 146.75.118.132.
With this command it is reproducible:
$ curl -v -H "Host: download.rockylinux.org" http://146.75.118.132/pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-dnf-plugin-versionlock-4.0.21-14.1.el8.noarch.rpm
* Trying 146.75.118.132:80...
* Connected to 146.75.118.132 (146.75.118.132) port 80 (#0)
> GET /pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-dnf-plugin-versionlock-4.0.21-14.1.el8.noarch.rpm HTTP/1.1
> Host: download.rockylinux.org
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: keep-alive
< Content-Length: 107
< Content-Type: text/html
< Fastly-Restarts: 2
< Via: 1.1 varnish, 1.1 varnish
< Cache-Control: max-age=11059200, public, stale-while-revalidate=86400, stale-if-error=259200
< Accept-Ranges: bytes
< Date: Fri, 25 Nov 2022 21:30:04 GMT
< Age: 869176
< X-Served-By: cache-chi-kigq8000033-CHI, cache-fra-eddf8230086-FRA
< X-Cache: MISS, HIT
< X-Cache-Hits: 0, 1
< X-Timer: S1669411804.378253,VS0,VE4
<
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
Other packages can be downloaded from 146.75.118.132:
$ curl -v -H "Host: download.rockylinux.org" http://146.75.118.132/pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-dnf-4.7.0-11.el8.noarch.rpm
* Trying 146.75.118.132:80...
* Connected to 146.75.118.132 (146.75.118.132) port 80 (#0)
> GET /pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-dnf-4.7.0-11.el8.noarch.rpm HTTP/1.1
> Host: download.rockylinux.org
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: keep-alive
< Content-Length: 559464
< Content-Type: application/x-redhat-package-manager
< Server: nginx
< Last-Modified: Sun, 02 Oct 2022 14:19:39 GMT
< ETag: "63399dfb-88968"
< Via: 1.1 varnish, 1.1 varnish
< Cache-Control: max-age=11059200, public, stale-while-revalidate=86400, stale-if-error=259200
< Accept-Ranges: bytes
< Date: Fri, 25 Nov 2022 21:35:53 GMT
< Age: 869525
< X-Served-By: cache-chi-klot8100150-CHI, cache-fra-eddf8230128-FRA
< X-Cache: MISS, HIT
< X-Cache-Hits: 0, 1
< X-Timer: S1669412154.920917,VS0,VE2
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* Closing connection 0
Sorry, I responded without trying anything first, so I deleted the old one.
@wagegede , can you share the results when you do nslookup dl.rockylinux.org on the affected system? I can confirm that the endpoint node 151.101.114.132 gives me a 503 error for this file.
Some of the Rocky infra folks are looking at this, because it’s a weird issue. In the meantime, there are 2 ways to work around this that I can think of:
1: Force a working endpoint via /etc/hosts:
You can force your system to connect to a single known-good Fastly node temporarily, until we figure this out. Add this line to the end of your /etc/hosts file:
199.232.198.132 dl.rockylinux.org
2: Download the file with an alternate URL:
You can force Fastly to re-cache the file by adding a random variable at the end of the URL. If you grab this URL, for example: dl.rockylinux.org/pub/rocky/8/AppStream/x86_64/os/Packages/j/java-17-openjdk-headless-17.0.5.0.8-1.el8_7.x86_64.rpm?testvar=blah , it will download (a little bit slower, you are forcing a cache grab from the endpoint).
This may not be useful depending on your use-case. If syncing via dnf install or reposync, you may need to download this artifact directly to the local cache using this method.
Hope this helps while we figure this out, thanks!
-Skip
Heya folks,
This should be fully resolved now. I identified a race condition in our CDN code which was improperly restarting and caching 503 and 500 errors. Along with the fix I made last week to not cache 404s, I believe the CDN should be in a really good place in terms of not caching errors anymore!
On the plus side, we’ve only seen these issues as we’ve grown and had to handle even more requests per second. That said, I am thankful for your reports and hopeful that this will be the last set of them.
I issued a soft purge request for all RPM artifacts on the CDN as they were the only cacheable items affected by the bad restart logic. This means you may see one or two more errors, but subsequent requests should be unaffected once the item is revalidated in cache.
Please let me know if you have any questions or if you are still seeing this on any artifacts.
Best,
Neil
Thx so much. Repo Download via Red Hat Satellite Server is now working very well. Good Job!