When we migrated our application from CentOS 7 to RockyLinux 8, we encountered a nasty bug. The application worked correctly on CentOS 7 (Apache+mod_php).
Background
Apache has a configuration variable called “LimitRequestLine” that allows the maximum querystring length (useful for large GET requests) to be expanded beyond the system default. We use a large setting, which worked correctly with Apache+mod_php on CentOS 7. However, there is a hard-coded querystring limit in the version of Apache included in Rocky Linux 8, for mod_proxy_fcgi, that limits the max querystring length to 16384 bytes irrespective of the LimitRequestLine setting. This means that PHP scripts cannot use longer querystrings.
Manifestation
This bug was difficult to track down. It produces a line in the HTTPD error_log file as something like:
AH02536: couldn’t encode envvar ‘QUERY_STRING’ in 16384 bytes
Correct Behavior
The correct behavior would be for mod_proxy_fcgi to use the active LimitRequestLine value (or the default if there is not an explicit setting) as the maximum querystring (and environment variable) length rather than a hard coded limit.
Other Info
According to other online sources, the only way to overcome this bug is either to use the deprecated mod_php instead of php-fpm or manually recompile Apache with a different hard-coded value for the maximum environment variable size for mod_proxy_fcgi. Neither of these options is a good one.
This is a patch that incorporates the fix from this bug report. This was backported as a result of this bugzilla. I see the same code (#define MAX_MEM_SPOOL 16384) is defined upstream, see here.
When I look at the git blame of upstream, I see this PR, which resulted in 16384 being defined. Very likely the reason for what you’re seeing.
If this is indeed a bug, this might be a better question for the maintainers at Apache, seeing as this is simply a backport of what they have in their upstream code.
As an aside, would you be willing to try this out on Rocky Linux 9? I’d be curious to know if you get the same results, where the httpd version is 2.4.57. (When 9.5 is released, 2.4.62 of httpd will be available, too, which is or is closet to the official GA version).
I’m thinking if the issue isn’t there, the backported patch may be right that it is a bug and that patch may have inadvertently created it. I’m quite interested to know if that’s the case. And at least testing a more recent version may help get some eyes on it.
As for contacting httpd folks, there’s a few places you can probably try.
They seem to have #httpd on Libera IRC if you happen to use it. This may be quicker to get an answer since it’s more real time. I’d read their support and bug report page too just in case I missed something.
I don’t know how active their mail lists or bug tracker is, but maybe IRC will be quicker. Doesn’t hurt to try both the users mail list and IRC channel though.
What’s even stranger is I looked through the mail archives and couldn’t really find anything beyond what I found a little bit ago. It seems like things go into a black hole in regards to this code change. So hopefully someone with deeper knowledge of the code base (and product in general) may have a better idea.
That’s a good suggestion with respect to testing Rocky Linux 9. Migrating our live app from CentOS 7 to Rocky Linux 8 was a major undertaking, so I don’t think we will be able to actually migrate our live application to Rocky Linux 9 just for testing (as a major version upgrade would likely be a pain). However, I may spin up a new Rocky Linux 9 server over the next couple weeks and try to send a specially constructed http request, to a new PHP script, designed to trigger the bug.
I would disagree that there is a design flaw in our app. The current behavior of FastCGI (which differs from the old mod_php) no longer allows for a successful override of the LimitRequestLine variable in Apache configuration files–and no hard-coded limit is memorialized in the Apache configuration documentation. Where is this limit “known” to Apache end-users today? In fact, the only indication of the root cause of the issue appears in a cryptic message in the Apache error_log files when the max environment size is exceeded.
Instead of using a hard-coded maximum environment size (which limits the max querystring), it should ideally be set to be to the greater of a pre-set value or the active LimitRequestLine value in Apache. If that logic were used, this bug would not exist.
sure, you can use long query strings if you want, but it would seem to be bad practice and bad design of the web application. Short query string, OK. Long ones? Start considering POST instead.
Just because it worked before, doesn’t mean that it was correct to do it that way in the first place and therefore doesn’t mean that Apache is the bug or problem here. The application should have been designed to be more efficient.
I converted our GET requests to equivalent POST requests, and that fixed the issue with our service. However, the absolute length limit for Apache querystrings should be documented for end-users (i.e. with LimitRequestLine). It is a new, unexpected, and undocumented behavior that was difficult to track down.