329 Commits

Author SHA1 Message Date
Amaury Denoyelle
ffcd290ddd REGTESTS: fix http_reuse_conn_hash proxy test
Define pool-low-conn to 2 to force to have at least 2 idle connections
on the server side for the purpose of the test.
2021-02-17 09:11:52 +01:00
William Dauchy
e3f7bd5ae9 MEDIUM: contrib/prometheus-exporter: add listen stats
this was a missing piece for a while now even though it was planned. This
patch adds listen stats.
Nothing in particular but we make use of the status helper previously
added.  `promex_st_metrics` diff also looks scary, but I had to realign
all lines.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-15 14:14:59 +01:00
William Dauchy
d1a7b85a40 MEDIUM: server: support {check,agent}_addr, agent_port in server state
logical followup from cli commands addition, so that the state server
file stays compatible with the changes made at runtime; use previously
added helper to load server attributes.

also alloc a specific chunk to avoid mixing with other called functions
using it

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-12 16:04:52 +01:00
Amaury Denoyelle
7f583be822 REGTESTS: fix sni used in http_reuse_conn_hash for libressl 3.3.0
libressl 3.3.0 is stricter on the sni field and fails if it contains
illegal characters such as the underscore. Replace sni field with proper
name to pass the test on the CI environment.
2021-02-12 15:27:33 +01:00
Amaury Denoyelle
49aa5e9eca REGTESTS: deactivate a failed test on CI in http_reuse_conn_hash
The last request with proxy protocol fails on the CI, reporting
first_http_request as 1. Currently this is not reproducible on my
local machine.
2021-02-12 14:18:39 +01:00
Amaury Denoyelle
8abbd2dcc3 REGTESTS: adjust http_reuse_conn_hash requirements
This test is only valid for 2.4+ as the tested feature was introduced in
this version. Besides, it can only run with ssl support to test sni
reuse.
2021-02-12 14:18:16 +01:00
Amaury Denoyelle
7ef06c8253 MINOR: reg-tests: test http-reuse with proxy protocol
Complete the http-reuse test with the proxy protocol usage. A server
connection can be reused if the proxy protocol fields are the same.
2021-02-12 12:54:04 +01:00
Amaury Denoyelle
1921d20fff MINOR: connection: use proxy protocol as parameter for srv conn hash
Use the proxy protocol frame if proxy protocol is activated on the
server line. Do not add anymore these connections in the private list.
If some requests are made with the same proxy fields, they can reuse
the idle connection.

The reg-tests proxy_protocol_send_unique_id must be adapted has it
relied on the side effect behavior that every requests from a same
connection reused a private server connection. Now, a new connection is
created as expected if the proxy protocol fields differ.
2021-02-12 12:54:04 +01:00
Amaury Denoyelle
f679d9bbf3 MINOR: reg-test: test http-reuse with specific dst addr
Add a reg-test for connection reuse with a specific destination address.
It is based on the set-dst-port http rule.
2021-02-12 12:54:04 +01:00
Amaury Denoyelle
e676239922 MINOR: reg-tests: test http-reuse with sni
Add a reg-test for connection reuse with the sni parameter. Ensure that
only connections that use the same sni are reused.
2021-02-12 12:48:11 +01:00
Christopher Faulet
475fab397e REGTESTS: Add a script to test payload skipping for bodyless HTTP responses
This script is used to be sure unexpected payload in bodyless HTTP responses
is properly ignored on outgoing side, both by H1 and H2 muxes.
2021-02-10 16:25:42 +01:00
Willy Tarreau
826f3ab5e6 MINOR: stick-tables/counters: add http_fail_cnt and http_fail_rate data types
Historically we've been counting lots of client-triggered events in stick
tables to help detect misbehaving ones, but we've been missing the same on
the server side, and there's been repeated requests for being able to count
the server errors per URL in order to precisely monitor the quality of
service or even to avoid routing requests to certain dead services, which
is also called "circuit breaking" nowadays.

This commit introduces http_fail_cnt and http_fail_rate, which work like
http_err_cnt and http_err_rate in that they respectively count events and
their frequency, but they only consider server-side issues such as network
errors, unparsable and truncated responses, and 5xx status codes other
than 501 and 505 (since these ones are usually triggered by the client).
Note that retryable errors are purposely not accounted for, so that only
what the client really sees is considered.

With this it becomes very simple to put some protective measures in place
to perform a redirect or return an excuse page when the error rate goes
beyond a certain threshold for a given URL, and give more chances to the
server to recover from this condition. Typically it could look like this
to bypass a URL causing more than 10 requests per second:

  stick-table type string len 80 size 4k expire 1m store http_fail_rate(1m)
  http-request track-sc0 base       # track host+path, ignore query string
  http-request return status 503 content-type text/html \
      lf-file excuse.html if { sc0_http_fail_rate gt 10 }

A more advanced mechanism using gpt0 could even implement high/low rates
to disable/enable the service.

Reg-test converteers_ref_cnt_never_dec.vtc was updated to test it.
2021-02-10 12:27:01 +01:00
William Dauchy
69164228b8 MEDIUM: contrib/prometheus-exporter: export base stick table stats
I saw some people falling back to unix socket to collect some data they
could not find in prometheus exporter. One of them is base info from
stick tables (used/size).
I do not plan to extend it more for now; keys are quite a mess to
handle.

This should resolve github issue #1008.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-08 10:49:08 +01:00
Willy Tarreau
2da742933d REGTESTS: unbreak http-check-send.vtc
As noticed by Christopher, I messed up the version fix in commit
cb4ed02ef ("REGTESTS: mark http-check-send.vtc as 2.4-only"), as while
looking up the commit introducing the change I accidently reverted it.
Let's reinsert the contents of the file prior to that fix, except the
version, of course.
2021-02-05 10:13:15 +01:00
Willy Tarreau
23296f92f4 REGTESTS: mark sample_fetches/hashes.vtc as 2.4-only
Commit 9eea56009 ("REGTESTS: add tests for the xxh3 converter") introduced
the xxh3 to the tests thus made it incompatible with 2.3 and older, let's
upgrade the version requirement.
2021-02-04 18:07:59 +01:00
Willy Tarreau
cb4ed02ef0 REGTESTS: mark http-check-send.vtc as 2.4-only
Since commit 39ff8c519 ("REGTESTS: complete http-check test"), it breaks
on pre-2.4, let's update the required version.
2021-02-04 18:06:13 +01:00
William Dauchy
fe03e7d045 MEDIUM: server: adding support for check_port in server state
We can currently change the check-port using the cli command `set server
check-port` but there is a consistency issue when using server state.
This patch aims to fix this problem but will be also a good preparation
work to get rid of checkport flag, so we are able to know when checkport
was set by config.

I am fully aware this is not making github #953 moving forward, I
however think this might be acceptable while waiting for a proper
solution and resolve consistency problem faced with port settings.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-02-04 10:46:52 +01:00
William Lallemand
e0de0a6b32 MINOR: ssl/cli: flush the server session cache upon 'commit ssl cert'
Flush the SSL session cache when updating a certificate which is used on a
server line. This prevent connections to be established with a cached
session which was using the previous SSL_CTX.

This patch also replace the ha_barrier with a thread_isolate() since there
are more operations to do. The reg-test was also updated to remove the
'no-ssl-reuse' keyword which is now uneeded.
2021-02-03 18:51:01 +01:00
William Lallemand
8695ce0bae BUG/MEDIUM: ssl/cli: abort ssl cert is freeing the old store
The "abort ssl cert" command is buggy and removes the current ckch store,
and instances, leading to SNI removal. It must only removes the new one.

This patch also adds a check in set_ssl_cert.vtc and
set_ssl_server_cert.vtc.

Must be backported as far as 2.2.
2021-02-01 17:58:21 +01:00
William Lallemand
ff97edac3e REGTESTS: set_ssl_server_cert: cleanup the SSL caching option
Replace the tune.ssl.cachesize 0 and the no-tls-tickets by a
no-ssl-reuse option on the server line.
2021-02-01 14:57:31 +01:00
William Lallemand
a870a9cfdb REGTESTS: set_ssl_server_cert.vtc: remove SSL caching and set as working
In a previous commit this test was disabled because I though the
feature was broken, but in fact this is the test which is broken.

Indeed the connection between the server and the client was not
renegociated and was using the SSL cache or a ticket. To be work
correctly these 2 features must be disabled or a new connection must be
established after the ticket timeout, which is too long for a regtest.

Also a "nbthread 1" was added as it was easier to reproduce the problem
with it.
2021-02-01 14:50:17 +01:00
William Lallemand
e814321287 REGTESTS: set_ssl_server_cert.vtc: set as broken
It looks like this test is broken with a low nbthread value (1 for
example). Disable this test in the CI until the problem is solved.
2021-01-28 18:08:36 +01:00
Christopher Faulet
d2dcd8a88f REGTEST: Don't use the websocket to validate http-check
Now, some conformance tests are performed when an HTTP connection is
upgraded to websocket. This make the http-check-send.vtc script failed for
the backend <be6_ws>. Because the purpose of this health-check is to pass a
"Connection: Upgrade" header on an http-check send rule, we may use a dummy
protocal instead.
2021-01-28 16:37:14 +01:00
Christopher Faulet
85a813676f REGTESTS: Fix required versions for several scripts
The following scripts require HAProxy 2.4 :

 * cache/caching_rules.vtc
 * cache/post_on_entry.vtc
 * cache/vary.vtc
 * checks/1be_40srv_odd_health_checks.vtc
 * checks/40be_2srv_odd_health_checks.vtc
 * checks/4be_1srv_health_checks.vtc
 * converter/fix.vtc
 * converter/mqtt.vtc
 * http-messaging/protocol_upgrade.vtc
 * http-messaging/websocket.vtc
 * http-set-timeout/set_timeout.vtc
 * log/log_uri.vtc

However it may change is features are backported.
2021-01-28 16:37:14 +01:00
Amaury Denoyelle
8969c32d7e MINOR: vtc: add websocket test
Test the conformance of websocket rfc6455 in haproxy. In particular, if
a missing key is detected on a h1 message, haproxy must close the
connection.

Note that the case h2 client/h1 srv is not tested as I did not find a
way to calculate the key on the server side.
2021-01-28 16:37:14 +01:00
Amaury Denoyelle
bda34302d7 MINOR: vtc: add test for h1/h2 protocol upgrade translation
This test send HTTP/1.1 Get+Upgrade or HTTP/2 Extended Connect through a
haproxy instance.
2021-01-28 16:37:14 +01:00
Christopher Faulet
d8b44f0d9f REGTESTS: Don't run http_msg_full_on_eom script on the 2.4 anymore
The EOM block will be removed on the 2.4, thus this script will be broken on
this version. Now it is skipped for this version. It remains valid for 2.3
and 2.2.
2021-01-28 16:37:14 +01:00
William Lallemand
7b79424c05 REGTESTS: set_ssl_server_cert.vtc: check the sha1 from the server
Check the sha1 from the server side with the sample ssl_c_sha1 sample
fetch in order to evict a possible problem with "show/set ssl cert".
2021-01-28 16:00:22 +01:00
William Lallemand
7e69637ac5 REGTESTS: set_ssl_server_cert.vtc: check the Sha1 Fingerprint
Check the sha1 fingerprint once the certificate was changed with "show
ssl cert". This way the test is more reliable.
2021-01-28 15:11:59 +01:00
William Lallemand
8788c6ff13 REGTESTS: set_ssl_server_cert.vtc: remove the abort command
Temporarily remove the abort command as it seems to cause problems when
trying to do a "show ssl cert" after it.
2021-01-28 15:04:03 +01:00
Remi Tricot-Le Breton
f3eedfe195 MEDIUM: ssl: Enable backend certificate hot update
When trying to update a backend certificate, we should find a
server-side ckch instance thanks to which we can rebuild a new ssl
context and a new ckch instance that replace the previous ones in the
server structure. This way any new ssl session will be built out of the
new ssl context and the newly updated certificate.

This resolves a subpart of GitHub issue #427 (the certificate part)
2021-01-26 15:19:36 +01:00
Amaury Denoyelle
d86ae4bd03 MINOR: reg-tests: add http-reuse test
Add a serie of 4 tests for the various http-reuse modes : never, safe,
aggressive and always.
2021-01-26 14:48:39 +01:00
Christopher Faulet
6071c2d12d BUG/MEDIUM: filters/htx: Fix data forwarding when payload length is unknown
It is only a problem on the response path because the request payload length
it always known. But when a filter is registered to analyze the response
payload, the filtering may hang if the server closes just after the headers.

The root cause of the bug comes from an attempt to allow the filters to not
immediately forward the headers if necessary. A filter may choose to hold
the headers by not forwarding any bytes of the payload. For a message with
no payload but a known payload length, there is always a EOM block to
forward. Thus holding the EOM block for bodyless messages is a good way to
also hold the headers. However, messages with an unknown payload length,
there is no EOM block finishing the message, but only a SHUTR flag on the
channel to mark the end of the stream. If there is no payload when it
happens, there is no payload at all to forward. In the filters API, it is
wrongly detected as a condition to not forward the headers.

Because it is not the most used feature and not the obvious one, this patch
introduces another way to hold the message headers at the begining of the
forwarding. A filter flag is added to explicitly says the headers should be
hold. A filter may choose to set the STRM_FLT_FL_HOLD_HTTP_HDRS flag and not
forwad anything to hold the headers. This flag is removed at each call, thus
it must always be explicitly set by filters. This flag is only evaluated if
no byte has ever been forwarded because the headers are forwarded with the
first byte of the payload.

reg-tests/filters/random-forwarding.vtc reg-test is updated to also test
responses with unknown payload length (with and without payload).

This patch must be backported as far as 2.0.
2021-01-26 09:53:52 +01:00
Remi Tricot-Le Breton
6ca89162dc MINOR: cache: Do not store responses with an unknown encoding
If a server varies on the accept-encoding header and it sends a response
with an encoding we do not know (see parse_encoding_value function), we
will not store it. This will prevent unexpected errors caused by
cache collisions that could happen in accept_encoding_hash_cmp.
2021-01-15 22:33:05 +01:00
William Dauchy
1704efee89 MINOR: contrib/prometheus-exporter: avoid connection close header
it does not seem to have a reason to close connections after each
request; reflect that in tests by doing all requests within the same
client.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-13 15:11:38 +01:00
William Dauchy
76603f2552 MINOR: reg-tests: add base prometheus test
Add a base test to start with something, even though this is not
necessarily complete.
Also make use of the recent REQUIRE_SERVICE option to exclude it from
test list of it was not build with prometheus included.

note: I thought it was possible to send multiple requests within the
same client, but I'm getting "HTTP header is incomplete" from the second
request

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-11 14:16:06 +01:00
William Dauchy
5417e898ff CLEANUP: sample: remove uneeded check in json validation
- check functions are never called with a NULL args list, it is always
  an array, so first check can be removed
- the expression parser guarantees that we can't have anything else,
  because we mentioned json converter takes a mandatory string argument.
  Thus test on `ARGT_STR` can be removed as well
- also add breaking line between enum and function declaration

In order to validate it, add a simple json test testing very simple
cases but can be improved in the future:

- default json converter without args
- json converter failing on error (utf8)
- json converter with error being removed (utf8s)

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-10 07:39:58 +01:00
William Dauchy
888b0ae8cf MINOR: converter: adding support for url_enc
add base support for url encode following RFC3986, supporting `query`
type only.

- add test checking url_enc/url_dec/url_enc
- update documentation
- leave the door open for future changes

this should resolve github issue #941

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-06 23:43:04 +01:00
Ilya Shipitsin
1e9a66603f CLEANUP: assorted typo fixes in the code and comments
This is 14th iteration of typo fixes
2021-01-06 16:26:50 +01:00
Willy Tarreau
fae9a4cd2c REGTESTS: add unresolvable servers to srvkey-addr
This ensures that the previous fix remains operational, as without it
an unresolvable server causes a crash when it's been indexed.
2021-01-06 09:20:22 +01:00
Thayne McCombs
23cc52d34b REGTESTS: add test for stickiness using "srvkey addr"
This tests the new "srvkey addr" stick-table statement, which sticks based
on the server's address.
2020-12-31 10:08:54 +01:00
Tim Duesterhus
dc38bc4a1a BUG/MEDIUM: cache: Fix hash collision in accept-encoding handling for Vary
This patch fixes GitHub Issue #988. Commit ce9e7b25217c46db1ac636b2c885a05bf91ae57e
was not sufficient, because it fell back to a hash comparison if the bitmap
of known encodings was not acceptable instead of directly returning the the
cached response is not compatible.

This patch also extends the reg-test to test the hash collision that was
mentioned in #988.

Vary handling is 2.4, no backport needed.
2020-12-31 09:39:08 +01:00
Remi Tricot-Le Breton
e6cc5b5974 MINOR: cache: Replace the "process-vary" option's expected values
Replace the <0/1> expected values of the process-vary option by a more
usual <on/off> pair.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
14df1e1526 REGTESTS: cache: Add a specific test for the accept-encoding normalizer
Those tests check the explicit support of a subset of encodings added to
the accept-encoding normalizer.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
b054b6dec6 REGTESTS: cache: Simplify vary.vtc file
This test used chunked responses but it is not needed. All the chunked
responses are replaced by more simple 'bodylen' directives.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
e4421dec7e BUG/MINOR: cache: Manage multiple headers in accept-encoding normalization
The accept-encoding part of the secondary key (vary) was only built out
of the first occurrence of the header. So if a client had two
accept-encoding headers, gzip and br for instance, the key would have
been built out of the gzip string. So another client that only managed
gzip would have been sent the cached resource, even if it was a br resource.
The http_find_header function is now called directly by the normalizers
so that they can manage multiple headers if needed.
A request that has more than 16 encodings will be considered as an
illegitimate request and its response will not be stored.

This fixes GitHub issue #987.

It does not need any backport.
2020-12-24 17:18:00 +01:00
Dragan Dosen
9eea56009d REGTESTS: add tests for the xxh3 converter 2020-12-23 06:39:21 +01:00
Amaury Denoyelle
39ff8c519c REGTESTS: complete http-check test
Add a new check for a pseudo-websocket handshake, specifying the
Connection header to verify if it is properly handled by http-check send
directive. Also check that default http/1.1 checks have the header
Connection: close.
2020-12-22 14:22:44 +01:00
Ilya Shipitsin
f38a01884a CLEANUP: assorted typo fixes in the code and comments
This is 13n iteration of typo fixes
2020-12-21 11:24:48 +01:00
Amaury Denoyelle
78ea5c99a9 REGTESTS: add regtest for http-request set-timeout
This test compares the timeout value for requests using the sample
fetches in accordance with the application of set-timeout rules.
2020-12-11 12:01:07 +01:00