When using h2 frontend and h2 backend under low load, the proxy uses 1-2 connections to each backend and these connections can become idle. However, the 1 remaining idle backend connection appears to get into a state where the underlying TCP connection has closed but HAproxy keeps trying to use it and never attempts to create a new one despite every request failing (with flags SC iirc) or being redispatched. If all backends fall into this state, all requests will fail 503. The affected backends do not get put into down state though, as the http check on a secondary port remains working and using observe layer7 does not help. Testing with requests directly to the backend from the HAproxy host, it responds fine so it does not appear to be a connectivity issue (plus health checks are working).

Below is the stats UI when the proxy is in this state, you can see server_2 is redispatching everything to server_1. It reports 1 idle connection but the bytes in/out counters are not increasing over time as the sessions and redis counters are inrementing. Similar values are reported in show servers conn on the api.

Looking at socket stats with ss, the idle connection to server_2 does not exist, so it seems HAproxy is repeatedly trying to reuse a closed socket:

Observing tcpdump at both sides, I see zero traffic (connection attempts nor data) on the main port to server_2 but can see the health checks on the secondary port.

If I then do a set server ... state maint then set server ... state ready with the API on the affected backend server then the problem resolves.

Some environment specific details:

Things I have tried:

I have tried searching for related issues here and in discourse; the only potentially related topic I could find was this one with no solution: https://discourse.haproxy.org/t/frequent-connection-retries-and-timeouts-with-grpc/5689/1

If there's any API commands you would like me to try when in this state please let me know

The idle connection should be closed when it has failed and a new one opened, without having to manually force the server into maintenance

I have not yet found a way of determistically triggering this every time but it happens intermittently when both the backend and HAproxy are starting at the same time. Seems potentially race-condition'y as sometimes no servers are affected, often 1 is affected and sometimes multiple are.

I see a comment below on the HAproxy site:

pool-purge-delay: Frequency at which HAProxy will close idle and unused connections. Only half of the idle connections will be closed after this period of time The default is 5 seconds.

If there is only a single idle connection, will this 'half' of 1 mean none get closed?

Alternatively, is the use of srvtcpka preventing the connection ever looking idle?

No response

No response