16345 Commits

Author SHA1 Message Date
William Lallemand
01e2be84d7 BUG/MINOR: httpclient/lua: don't pop the lua stack when getting headers
hlua_httpclient_table_to_hdrs() does a lua_pop(L, 1) at the end of the
function, this is supposed to be done in the caller and it is already be
done in hlua_httpclient_send().

This call has the consequence of poping the next parameter of the
httpclient, ignoring it.

This patch fixes the issue by removing the lua_pop(L, 1).

Must be backported in 2.5.
2022-01-14 20:51:31 +01:00
William Lallemand
bad9c8cac4 BUG/MINOR: httpclient: set default Accept and User-Agent headers
Some servers require at least an Accept and a User-Agent header in the
request. This patch sets some default value.

Must be backported in 2.5.
2022-01-14 20:46:21 +01:00
William Lallemand
e1e045f4d7 BUG/MINOR: httpclient: don't send an empty body
Forbid the httpclient to send an empty chunked client when there is no
data to send. It does happen when doing a simple GET too.

Must be backported in 2.5.
2022-01-14 18:10:43 +01:00
Ilya Shipitsin
b9e3fb7315 CI: github actions: use cache for OpenTracing
this caches OpenTracing libs between builds, should save couple of minutes
for each build.
2022-01-13 13:37:34 +01:00
Christopher Faulet
28e7ba8688 BUG/MEDIUM: htx: Adjust length to add DATA block in an empty HTX buffer
htx_add_data() is able to partially consume data. However there is a bug
when the HTX buffer is empty.  The data length is not properly
adjusted. Thus, if it exceeds the HTX buffer size, no block is added. To fix
the issue, the length is now adjusted first.

This patch must be backported as far as 2.0.
2022-01-13 09:34:22 +01:00
Frédéric Lécaille
b80b20c6ff MINOR: quic: Do not wakeup the I/O handler before the mux is started
If we wakeup the I/O handler before the mux is started, it is possible
it has enough time to parse the ClientHello TLS message and update the
mux transport parameters, leading to a crash.
So, we initialize ->qcc quic_conn struct member at the very last time,
when the mux if fully initialized. The condition to wakeup the I/O handler
from lstnr_rcv_pkt() is: xprt context and mux both initialized.
Note that if the xprt context is initialized, it implies its tasklet is
initialized. So, we do not check anymore this latter condition.
2022-01-12 18:08:29 +01:00
Frédéric Lécaille
bec186dde5 MINOR: quic: As server, skip 0-RTT packet number space
This is true only when we are building packets. A QUIC server never
sends 0-RTT packets. So let't skip the associated TLS encryption level.
2022-01-12 18:08:29 +01:00
Willy Tarreau
39fd546d4b MINOR: pools: enable pools with DEBUG_FAIL_ALLOC as well
During 2.4-dev, fault injection was enabled for cached pools with commit
207c09509 ("MINOR: pools: move the fault injector to __pool_alloc()"),
except that the condition for CONFIG_HAP_POOLS still depended on
DEBUG_FAIL_ALLOC not being set, which limits the usability to cases
where the define is set by hand. Let's remove it from the equation as
this is not a constraint anymore. While a bit old, there's no need to
backport this as it's only used during development.
2022-01-12 17:31:01 +01:00
Willy Tarreau
3b990fe0be BUG/MEDIUM: connection: properly leave stopping list on error
The stopping-list management introduced by commit d3a88c1c3 ("MEDIUM:
connection: close front idling connection on soft-stop") missed two
error paths in the H1 and H2 muxes. The effect is that if a stream
or HPACK table couldn't be allocated for these incoming connections,
we would leave with the connection freed still attached to the
stopping_list and it would never leave it, resulting in use-after-free
hence either a crash or a data corruption.

This is marked as medium as it only happens under extreme memory pressure
or when playing with tune.fail-alloc. Other stability issues remain in
such a case so that abnormal behaviors cannot be explained by this bug
alone.

This must be backported to 2.4.
2022-01-12 17:31:01 +01:00
Amaury Denoyelle
9ab2fb3921 MINOR: quic: free xprt tasklet on its thread
Free the ssl_sock_ctx tasklet in quic_close() instead of
quic_conn_drop(). This ensures that the tasklet is destroyed safely by
the same thread.

This has no impact as the free operation was previously conducted with
care and should not be responsible of any crash.
2022-01-12 15:21:27 +01:00
Amaury Denoyelle
b76ae69513 MEDIUM: quic: implement Retry emission
Implement the emission of Retry packets. These packets are emitted in
response to Initial from clients without token. The token from the Retry
packet contains the ODCID from the Initial packet.

By default, Retry packet emission is disabled and the handshake can
continue without address validation. To enable Retry, a new bind option
has been defined named "quic-force-retry". If set, the handshake must be
conducted only after receiving a token in the Initial packet.
2022-01-12 11:08:48 +01:00
Amaury Denoyelle
c3b6f4d484 MINOR: quic: define retry_source_connection_id TP
Define a new QUIC transport parameter retry_source_connection_id. This
parameter is set only by server, after issuing a Retry packet.
2022-01-12 11:08:48 +01:00
Amaury Denoyelle
5ff1c9778c MEDIUM: quic: implement Initial token parsing
Implement the parsing of token from Initial packets. It is expected that
the token contains a CID which is the DCID from the Initial packet
received from the client without token which triggers a Retry packet.
This CID is then used for transport parameters.

Note that at the moment Retry packet emission is not implemented. This
will be achieved in a following commit.
2022-01-12 11:08:48 +01:00
Amaury Denoyelle
6efec292ef MINOR: quic: implement Retry TLS AEAD tag generation
Implement a new QUIC TLS related function
quic_tls_generate_retry_integrity_tag(). This function can be used to
calculate the AEAD tag of a Retry packet.
2022-01-12 11:08:48 +01:00
Amaury Denoyelle
c8b4ce4a47 MINOR: quic: add config parse source file
Create a new dedicated source file for QUIC related options parsing on
the bind line.
2022-01-12 11:08:48 +01:00
Amaury Denoyelle
ce340fe4a7 MINOR: quic: fix return of quic_dgram_read
It is expected that quic_dgram_read() returns the total number of bytes
read. Fix the return value when the read has been successful. This bug
has no impact as in the end the return value is not checked by the
caller.
2022-01-12 11:08:48 +01:00
Frédéric Lécaille
1aa57d32bb MINOR: quic: Do not dereference ->conn quic_conn struct member
->conn quic_conn struct member is a connection struct object which may be
released from several places. With this patch we do our best to stop dereferencing
this member as much as we can.
2022-01-12 09:49:49 +01:00
Remi Tricot-Le Breton
aab8d255bc REGTESTS: ssl: Fix ssl_errors regtest with OpenSSL 1.0.2
This test was broken with OpenSSL 1.0.2 after commit a996763619d
(BUG/MINOR: ssl: Store client SNI in SSL context in case of ClientHello
error) because it expected the default TLS version to be 1.3 in some
cases (when it can't be the case with OpenSSL 1.0.2).
2022-01-11 20:02:37 +01:00
Frédéric Lécaille
ba85acdc70 MINOR: quid: Add traces quic_close() and quic_conn_io_cb()
This is to have an idea of possible remaining issues regarding the
connection terminations.
2022-01-11 16:56:04 +01:00
Frédéric Lécaille
81cd3c8eed MINOR: quic: Wrong CRYPTO frame concatenation
This commit was not correct:
	 "MINOR: quic: Only one CRYPTO frame by encryption level"
Indeed, when receiving CRYPTO data from TLS stack for a packet number space,
there are rare cases where there is already other frames than CRYPTO data frames
in the packet number space, especially for 01RTT packet number space.  This is
very often with quant as client.
2022-01-11 16:12:31 +01:00
Frédéric Lécaille
2fe8b3be20 MINOR: quic: Flag the connection as being attached to a listener
We do not rely on connection objects to know if we are a listener or not.
2022-01-11 16:12:31 +01:00
Frédéric Lécaille
19cd46e6e5 MINOR: quic: Reset ->conn quic_conn struct member when calling qc_release()
There may be remaining locations where ->conn quic_conn struct member
is used. So let's reset this.
Add a trace to have an idead when this connection is released.
2022-01-11 16:12:31 +01:00
Frédéric Lécaille
5f7f118b31 MINOR: quic: Remaining TRACEs with connection as firt arg
This is a quic_conn struct which is expected by TRACE_()* macros
2022-01-11 16:12:31 +01:00
David CARLIER
bb10dad5a8 BUILD: cpuset: fix build issue on macos introduced by previous change
The build on macos was broken by recent commit df91cbd58 ("MINOR: cpuset:
switch to sched_setaffinity for FreeBSD 14 and above."), let's move the
variable declaration inside the ifdef.
2022-01-11 15:09:49 +01:00
Ilya Shipitsin
65eab587a2 CI: github actions: clean default step conditions
step condition "if: ${{ !failure() }}" was added in 2ef4c7c84363f5a9b80a2093df1370514319db28
during my experiments. As Tim Dsterhus mentioned, that condition is default and may be omitted.
2022-01-11 15:00:55 +01:00
Willy Tarreau
b64ef3e3f8 DOC: internals: document the pools architecture and API
The purpose here is to explain how memory pools work, what their
architecture is depending on the build options (4 possible combinations),
and how the various build options affect their behavior.

Two pool-specific macros that were previously documented in initcalls
were moved to pools.txt.
2022-01-11 14:51:41 +01:00
Christopher Faulet
b4eca0e908 BUG/MAJOR: mux-h1: Don't decrement .curr_len for unsent data
A regression was introduced by commit 140f1a58 ("BUG/MEDIUM: mux-h1: Fix
splicing by properly detecting end of message"). To detect end of the
outgoing message, when the content-length is announced, we count amount of
data already sent. But only data really sent must be counted.

If the output buffer is full, we can fail to send data (fully or
partially). In this case, we must take care to only count sent
data. Otherwise we may think too much data were sent and an internal error
may be erroneously reported.

This patch should fix issues #1510 and #1511. It must be backported as far
as 2.4.
2022-01-11 09:15:13 +01:00
Remi Tricot-Le Breton
a996763619 BUG/MINOR: ssl: Store client SNI in SSL context in case of ClientHello error
If an error is raised during the ClientHello callback on the server side
(ssl_sock_switchctx_cbk), the servername callback won't be called and
the client's SNI will not be saved in the SSL context. But since we use
the SSL_get_servername function to return this SNI in the ssl_fc_sni
sample fetch, that means that in case of error, such as an SNI mismatch
with a frontend having the strict-sni option enabled, the sample fetch
would not work (making strict-sni related errors hard to debug).

This patch fixes that by storing the SNI as an ex_data in the SSL
context in case the ClientHello callback returns an error. This way the
sample fetch can fallback to getting the SNI this way. It will still
first call the SSL_get_servername function first since it is the proper
way of getting a client's SNI when the handshake succeeded.

In order to avoid memory allocations are runtime into this highly used
runtime function, a new memory pool was created to store those client
SNIs. Its entry size is set to 256 bytes since SNIs can't be longer than
255 characters.

This fixes GitHub #1484.

It can be backported in 2.5.
2022-01-10 16:31:22 +01:00
William Lallemand
f82afbb9cd BUG/MEDIUM: mworker: don't use _getsocks in wait mode
Since version 2.5 the master is automatically re-executed in wait-mode
when the config is successfully loaded, puting corner cases of the wait
mode in plain sight.

When using the -x argument and with the right timing, the master will
try to get the FDs again in wait mode even through it's not needed
anymore, which will harm the worker by removing its listeners.

However, if it fails, (and it's suppose to, sometimes), the
master will exit with EXIT_FAILURE because it does not have the
MODE_MWORKER flag, but only the MODE_MWORKER_WAIT flag. With the
consequence of killing the workers.

This patch fixes the issue by restricting the use of _getsocks to some
modes.

This patch must be backported in every version supported, even through
the impact should me more harmless in version prior to 2.5.
2022-01-07 18:44:27 +01:00
Frédéric Lécaille
99942d6f4c MINOR: quic: Non-optimal use of a TX buffer
When full, after having reset the writer index, let's reuse the TX buffer in
any case.
2022-01-07 17:58:26 +01:00
Frédéric Lécaille
f010f0aaf2 MINOR: quic: Missing retransmission from qc_prep_fast_retrans()
In fact we must look for the first packet with some ack-elicting frame to
in the packet number space tree to retransmit from. Obviously there
may be already retransmit packets which are not deemed as lost and
still present in the packet number space tree for TX packets.
2022-01-07 17:58:26 +01:00
Frédéric Lécaille
d4ecf94827 MINOR: quic: Only one CRYPTO frame by encryption level
When receiving CRYPTO data from the TLS stack, concatenate the CRYPTO data
to the first allocated CRYPTO frame if present. This reduces by one the number
of handshake packets built for a connection with a standard size certificate.
2022-01-07 17:58:26 +01:00
Willy Tarreau
790169fe69 BUILD: makefile: add -Wno-atomic-alignment to work around clang abusive warning
As reported in github issue #1502, clang, when building for i386, will
try to use CMPXCHG8B-based loops for 64-bit atomic operations, and emits
warnings for all 64-bit operands that are not 64-bit aligned, an alignment
that is *not* required by the ABI, that the compiler itself does not
enforce, and that the intel SDM clearly says is not required on this
32-bit platform for this operation. But this is likely an excessive
outcome of the same code being used in 64-bit for CMPXCHG16B which does
require proper alignment. Firefox already gave up on this one 3 years
ago, let's not waste our time arguing and just shut up the warning
instead. It might hide some real bugs in the future but till now
experience showed that overall it's unlikely.

This should be backported to all maintained branches that use 64-bit
atomic ops (e.g. for counters).

Thanks to Brad Smith for reporting it and confirming that shutting the
warning addresses it.
2022-01-07 14:58:48 +01:00
Ilya Shipitsin
37d3e38130 CLEANUP: assorted typo fixes in the code and comments
This is 30th iteration of typo fixes
2022-01-07 14:42:54 +01:00
Ilya Shipitsin
6569de2b88 CI: refactor spelling check
let us switch to codespell github actions instead of invocation from cmdline.
also, "ifset,thrid,strack,ba,chck,hel,unx,mor" added to whitelist, those are
variable names and special terms widely used in HAProxy
2022-01-07 14:42:33 +01:00
David CARLIER
df91cbd584 MINOR: cpuset: switch to sched_setaffinity for FreeBSD 14 and above.
Following up previous update on cpuset-t.h. Ultimately, at some point
 the cpuset_setaffinity code path could be removed.
2022-01-07 06:53:51 +01:00
William Dauchy
a9dd901143 MINOR: proxy: add option idle-close-on-response
Avoid closing idle connections if a soft stop is in progress.

By default, idle connections will be closed during a soft stop. In some
environments, a client talking to the proxy may have prepared some idle
connections in order to send requests later. If there is no proper retry
on write errors, this can result in errors while haproxy is reloading.
Even though a proper implementation should retry on connection/write
errors, this option was introduced to support back compat with haproxy <
v2.4. Indeed before v2.4, we were waiting for a last request to be able
to add a "connection: close" header and advice the client to close the
connection.

In a real life example, this behavior was seen in AWS using the ALB in
front of a haproxy. The end result was ALB sending 502 during haproxy
reloads.
This patch was tested on haproxy v2.4, with a regular reload on the
process, and a constant trend of requests coming in. Before the patch,
we see regular 502 returned to the client; when activating the option,
the 502 disappear.

This patch should help fixing github issue #1506.
In order to unblock some v2.3 to v2.4 migraton, this patch should be
backported up to v2.4 branch.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
[wt: minor edits to the doc to mention other options to care about]
Signed-off-by: Willy Tarreau <w@1wt.eu>
2022-01-06 09:09:51 +01:00
Frédéric Lécaille
6b6631593f MINOR: quic: Re-arm the PTO timer upon datagram receipt
When block by the anti-amplification limit, this is the responsability of the
client to unblock it sending new datagrams. On the server side, even if not
well parsed, such datagrams must trigger the PTO timer arming.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
078634d126 MINOR: quic: PTO timer too often reset
It must be reset when the anti-amplication was reached but only if the
peer address was not validated.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
41a076087b MINOR: quic: Flag asap the connection having reached the anti-amplification limit
The best location to flag the connection is just after having built the packet
which reached the anti-amplication limit.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
de6f7c503e MINOR: quic: Prepare Handshake packets asap after completed handshake
Switch back to QUIC_HS_ST_SERVER_HANDSHAKE state after a completed handshake
if acks must be send.
Also ensure we build post handshake frames only one time without using prev_st
variable and ensure we discard the Handshake packet number space only one time.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
917a7dbdc7 MINOR: quic: Do not drop secret key but drop the CRYPTO data
We need to be able to decrypt late Handshake packets after the TLS secret
keys have been discarded. If not the peer send Handshake packet which have
not been acknowledged. But for such packets, we discard the CRYPTO data.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
ee2b8b377f MINOR: quic: Improve qc_prep_pkts() flexibility
We want to be able to choose the encryption levels to be used
by qc_prep_pkts() outside of it.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
f7ef97698a MINOR: quic: Comment fix.
When we drop a packet with unknown length, this is the entire datagram which
must be skipped.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
a56054e438 MINOR: quic: Probe several packet number space upon timer expiration
When the loss detection timer expires, we SHOULD include new data in
our probing packets (RFC 9002 par 6.2.4. Sending Probe Packets).
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
db6a4727cf MINOR: quic: Probe Initial packet number space more often
Especially when the PTO expires for Handshake packet number space and when
Initial packets are still flying (for QUIC servers).
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
3bb457c4ba MINOR: quic: Speeding up Handshake Completion
According to RFC 9002 par. 6.2.3. when receving duplicate Initial CRYPTO
data a server may a packet containing non unacknowledged before the PTO
expiry.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
63556772cc MINOR: quic: qc_prep_pkts() code moving
Move the switch default case code out of the switch to improve the readibily.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
5732062cd2 MINOR: quic: Useless test in qc_prep_pkts()
These tests were there to initiate PTO probing but they are not correct.
Furthermore they may break the PTO probing process and lead to useless packet
building.
2022-01-04 17:30:00 +01:00
Frédéric Lécaille
dd51da599e MINOR: quic: Wrong packet number space trace in qc_prep_pkts()
It was always the first packet number space information which was dumped.
2022-01-04 17:30:00 +01:00