18927 Commits

Author SHA1 Message Date
Aurelien DARRAGON
81b7c9518c MINOR: check: use atomic for s->consecutive_errors
Properly use atomic operations when dealing with s->consecutive_errors as
we're using it out of server's lock.

Race is negligible, no backport needed.
2022-12-07 17:04:08 +01:00
Aurelien DARRAGON
7d541a91ec BUG/MINOR: checks: restore legacy on-error fastinter behavior
With previous commit, 9e080bf ("BUG/MINOR: checks: make sure fastinter is used
even on forced transitions"), on-error mark-down|sudden-death|fail-check are
now working as expected.

However, on-error fastinter remains broken because srv_getinter(), used in
the above commit to check the expiration date, won't return fastinter interval
if server health is maxed out (which is the case with on-error fastinter mode).

To fix this, we introduce a check flag named CHK_ST_FASTINTER.
This flag is set when on-error is triggered. This way we can force
srv_getinter() to return fastinter interval whenever the flag is set.
The flag is automatically cleared as soon as the new check task expiry is
recalculated in process_chk_conn().
This restores original behavior prior to d114f4a ("MEDIUM: checks: spread the
checks load over random threads").

It must be backported to 2.7 along with the aforementioned commits.
2022-12-07 17:03:55 +01:00
William Lallemand
e57b702e2b BUG/MEDIUM: mworker: create the mcli_reload socketpairs in case of upgrade
In ticket #1956, it was reported that an upgrade from 2.6 to 2.7 via a
reload would stop the master process.

When upgrading the binary, the new process is considered reexec and does
not try to creates the socketpair for the mcli_reload listener, then
tries to bind on -1 since the socket doesn't exit. The failure provokes
an exit() of the master.

This patch fixes the issue by trying to create the mcli_reload sockets
only when they don't exist, instead of creating them at first start.
This way we also avoid possible fd leak since we always try to use the
existing FDs first.

Must be backported in 2.7.
2022-12-07 15:30:52 +01:00
William Lallemand
035058e8bf BUG/MEDIUM: mworker: fix segv in early failure of mworker mode with peers
During an early failure of the mworker mode, the
mworker_cleanlisteners() function is called and tries to cleanup the
peers, however the peers are in a semi-initialized state and will use
NULL pointers.

The fix check the variable before trying to use them.

Bug revealed in issue #1956.

Could be backported as far as 2.0.
2022-12-07 15:27:36 +01:00
William Lallemand
40db4ae8bb MINOR: mworker: display an alert upon a wait-mode exit
When the mworker wait mode fails it does an exit, but there is no
error message which says it exits.

Add a message which specify that the error is non-recoverable.

Could be backported in 2.7 and possibly earlier branch.
2022-12-07 15:07:53 +01:00
Ilya Shipitsin
5fa29b8a74 CLEANUP: assorted typo fixes in the code and comments
This is 34th iteration of typo fixes
2022-12-07 09:08:18 +01:00
Willy Tarreau
9e080bf375 BUG/MINOR: checks: make sure fastinter is used even on forced transitions
Aurlien also found that while previous commit a56798ea4 ("BUG/MEDIUM:
checks: do not reschedule a possibly running task on state change")
addressed one specific case where the check's task had to be woken up
quickly, but it's not always sufficient as the check will not be
considered as expired regarding the fastinter yet.

Let's make sure we do consider this specific case to update the timer
based on the new state if the new value is shorter. This particularly
means that even if the timer is not expired yet during a wakeup when
nothing is in progress, we need to check if applying the currently
effective interval right now to the current date would expire earlier
than what is programmed, then the timer needs to be updated. I.e.
make sure we never miss fastinter during a state transition before
the end of the current period.

The approach is not pretty, but it forces to repass via the existing
block dedicated to updating the timer if the current one is expired
and the updated one would appear earlier.

This must be backported to 2.7 along with the commit above.
2022-12-06 18:48:22 +01:00
Willy Tarreau
a56798ea4d BUG/MEDIUM: checks: do not reschedule a possibly running task on state change
Aurlien found an issue introduced in 2.7-dev8 with commit d114f4a68
("MEDIUM: checks: spread the checks load over random threads"), but which
in fact has deeper roots.

When a server's state is changed via __health_adjust(), if a fastinter
setting is set, the task gets rescheduled to run at the new date. The
way it's done is not thread safe, as nothing prevents another thread
where the task is already running from also updating the expire field
in parallel. But since such events are quite rare, this statistically
never happens. However, with the commit above, the tasks are no longer
required to go to the shared wait queue and are no longer marked as
shared between multiple threads. It's just that *any* thread may run
them at a time without implying that all of them are allowed to modify
them. And this change is sufficient to trigger the BUG_ON() condition
in the scheduler that detects the inconsistency between a task queued
in one thread and being manipulated in parallel by another one:

  FATAL: bug condition "task->tid != tid" matched at
  include/haproxy/task.h:670
    call trace(13):
    | 0x55f61cf520c9 [c6 04 25 01 00 00 00 00]: main-0x2ee7
    | 0x55f61d0646e8 [8b 45 08 a8 40 0f 85 65]: back_handle_st_cer+0x78/0x4d7
    | 0x55f61cff3e72 [41 0f b6 4f 01 e9 c8 df]: process_stream+0x2252/0x364f
    | 0x55f61d0d2fab [48 89 c3 48 85 db 74 75]: run_tasks_from_lists+0x34b/0x8c4
    | 0x55f61d0d38ad [29 44 24 18 8b 54 24 18]: process_runnable_tasks+0x37d/0x6c6
    | 0x55f61d0a22fa [83 3d 0b 63 1e 00 01 0f]: run_poll_loop+0x13a/0x536
    | 0x55f61d0a28c9 [48 8b 1d f0 46 19 00 48]: main+0x14d919
    | 0x55f61cf56dfe [31 c0 e8 eb 93 1b 00 31]: main+0x1e4e/0x2d5d

At first glance it looked like it could be addressed in the scheduler
only, but in fact the problem clearly is at the application level, since
some shared fields are manipulated without protection. At minima, the
task's expiry ought to be touched only under the server's lock. While
it's arguable that the scheduler could make such updates easier, changing
it alone will not be sufficient here.

Looking at the sequencing closer, it becomes obvious that we do not need
this task_schedule() at all: a simple task_wakeup() is sufficient for the
callee to update its timers. Indeed, the process_chk_con() function already
deals with spurious wakeups, and already uses srv_getinter() to calculate
the next wakeup date based on the current state. So here, instead of
having to queue the task from __health_adjust() to anticipate a new check,
we can simply wake the task up and let it decide when it needs to run
next. This is much cleaner as the expiry calculation remains performed at
a single place, from the task itself, as it should be, and it fixes the
problem above.

This should be backported to 2.7, but not to older versions where the
risks of breakage are higher than the chance to fix something that
ever happened.
2022-12-06 14:14:41 +01:00
Aurelien DARRAGON
22f82f81e5 MINOR: server/event_hdl: add support for SERVER_UP and SERVER_DOWN events
We're using srv_update_status() as the only event source or UP/DOWN server
events in an attempt to simplify the support for these 2 events.

It seems srv_update_status() is the common path for server state changes anyway

Tested with server state updated from various sources:
  - the cli
  - server-state file (maybe we could disable this or at least don't publish
  in global event queue in the future if it ends in slower startup for setups
  relying on huge server state files)
  - dns records (ie: srv template)
  (again, could be fined tuned to only publish in server specific subscriber
  list and no longer in global subscription list if mass dns update tend to
  slow down srv_update_status())
  - normal checks and observe checks (HCHK_STATUS_HANA)
  (same as above, if checks related state update storms are expected)
  - lua scripts
  - html stats page (admin mode)
2022-12-06 10:22:07 +01:00
Aurelien DARRAGON
129ecf441f MINOR: server/event_hdl: add support for SERVER_ADD and SERVER_DEL events
Basic support for ADD and DEL server events are added through this commit:
	SERVER_ADD is published on dynamic server addition through cli.
	SERVER_DEL is published on dynamic server deletion through cli.

This work depends on:
	"MINOR: event_hdl: add event handler base api"
	"MINOR: server: add srv->rid (revision id) value"
2022-12-06 10:22:07 +01:00
Aurelien DARRAGON
745ce8e8ad MINOR: stats: add server revision id support
Make use of the new srv->rid value in stats.

Stat is referred as ST_F_SRID, it is now used in stats_fill_sv_stats
function in order to be included in csv and json stats dumps.

Moreover, "rid: $value" will be displayed next to server puid
in html stats page if "stats show-legend" is specified in the stats frontend.
(mouse hovering tooltip)

Depends on the following commit:
	"MINOR: server: add srv->rid (revision id) value"
2022-12-06 10:22:06 +01:00
Aurelien DARRAGON
61e3894dfe MINOR: server: add srv->rid (revision id) value
With current design, we could not distinguish between
previously existing deleted server and a new server reusing
the deleted server name/id.

This can cause some confusion when auditing stats/events/logs,
because the new server will look similar to the old
one.

To address this, we're adding a new value in server structure: rid

rid (revision id) value is an unsigned 32bits value that is set upon
server creation. Value is derived from a global counter that starts
at 0 and is incremented each time one or multiple server deletions are
followed by a server addition (meaning that old name/id reuse could occur).

Thanks to this revision id, it is now easy to tell whether the server
we're looking at is the same as before or if it has been deleted and
re-added in the meantime.
(combining server name/id + server revision id yields a process-wide unique
identifier)
2022-12-06 10:22:06 +01:00
Christopher Faulet
7f59d68fe2 BUG/MEDIIM: stconn: Flush output data before forwarding close to write side
In process_stream(), we wait to have an empty output channel to forward a
close to the write side (a shutw). However, at the stream-connector level,
when a close is detected on one side and we don't want to keep half-close
connections, the shutw is unconditionally forwarded to the write side. This
typically happens on server side.

At first glance, this bug may truncate messages. But depending on the muxes
and the stream states, the bug may be more visible. On recent versions
(2.8-dev and 2.7) and on 2.2 and 2.0, the stream may be freezed, waiting for
the client timeout, if the client mux is unable to forward data because the
client is too slow _AND_ the response channel is not empty _AND_ the server
closes its connection _AND_ the server mux has forwarded all data to the
upper layer _AND_ the client decides to send some data and to close its
connection. On 2.6 and 2.4, it is worst. Instead of a freeze, the client mux
is woken up in loop.

Of course, conditions are pretty hard to meet. Especially because it is highly
time dependent. For what it's worth, I reproduce it with tcploop on client and
server sides and a basic HTTP configuration for HAProxy:

  * client: tcploop -v 8889 C S:"GET / HTTP/1.1\r\nConnection: upgrade\r\n\r\n" P5000 S:"1234567890" K
  * server: tcploop -v 8000 L A R S:"HTTP/1.1 101 ok\r\nConnection: upgrade\r\n\r\n" P2000 S2660000 F R

On 2.8-dev, without this patch, the stream is freezed and when the client
connection timed out, client data are truncated and '--cL' is reported in
logs. With the patch, the client data are forwarded to the server and the
connection is closed. A '--CD' is reported in logs.

It is an old bug. It was probably introduced with the multiplexers. To fix
it, in stconn (Formerly the stream-interface), we must wait all output data
be flushed before forwarding close to write side.

This patch must be backported as far as 2.2 and must be evaluated for 2.0.
2022-12-05 11:24:24 +01:00
Amaury Denoyelle
30fc27750d BUG/MINOR: quic: fix fd leak on startup check quic-conn owned socket
A startup check is done for first QUIC listener to detect if quic-conn
owned socket is supported by the system. This is done by creating a
dummy socket reusing the listener address. This socket must be closed as
soon as the check is done.

The socket condition is invalid as it excludes zero which is a valid
file-descriptor value. Fix this bug by adjusting this condition.

In theory, this bug could prevent the usage of quic-conn owned socket as
startup check would report a false error. Also, the file-descriptor
would leak as it is not closed. In practice, this cannot happen when
startup check is done after a 'quic4/quic6' listener is instantiated as
file-descriptor are allocated in ascending order by the system.

This should fix github issue #1954.

quic-conn owned socket implementation is scheduled for backport on 2.7.
This commit must be backported with it, more specifically to fix the
following patch :
  75839a44e7e904bd1e332b58bd579e03b6d106f0
  MINOR: quic: startup detect for quic-conn owned socket support
2022-12-05 10:45:20 +01:00
William Lallemand
151dbbe778 BUG/MINOR: ssl: initialize WolfSSL before parsing
The wolfSSL library need to be initialized before parsing the
configuration which uses some SSL functions.

To be backported in 2.6.
2022-12-02 17:17:43 +01:00
William Lallemand
44c80ce5b3 BUG/MINOR: ssl: initialize SSL error before parsing
The SSL error initialization need to be done before the configuration
parsing, because it uses the SSL.

Need to be backported to 2.6.
2022-12-02 17:10:11 +01:00
Amaury Denoyelle
e30f378236 MINOR: quic: activate socket per conn by default
Activate QUIC connection socket to achieve the best performance. The
previous behavior can be reverted by tune.quic.socket-owner
configuration option.

This change is part of quic-conn owned socket implementation.

Contrary to its siblings patches, I suggest to not backport it to 2.7.
This should ensure that stable releases behavior is perserved. If a user
faces issues with QUIC performance on 2.7, he can nonetheless change the
default configuration.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
d3083c9df9 MINOR: quic: reconnect quic-conn socket on address migration
UDP addresses may change over time for a QUIC connection. When using
quic-conn owned socket, we have to detect address change to break the
bind/connect association on the socket.

For the moment, on change detected, QUIC connection socket is closed and
a new one is opened. In the future, we may improve this by trying to
keep the original socket and reexecute only bind/connect syscalls.

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
b2bd83972b MEDIUM: quic: requeue datagrams received on wrong socket
There is a small race condition when QUIC connection socket is
instantiated between the bind() and connect() system calls. This means
that the first datagram read on the sockets may belong to another
connection.

To detect this rare case, we compare the DCID for each QUIC datagram
read on the QUIC socket. If it does not match the connection CID, the
datagram is requeue using quic_receiver_buf to be able to handle it on
the correct thread.

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
b7ce79c814 MINOR: mux-quic: rename duplicate function names
qc_rcv_buf and qc_snd_buf are names used for static functions in both
quic-sock and quic-mux. To remove this ambiguity, slightly modify names
used in MUX code.

In the future, we should properly define a unique prefix for all QUIC
MUX functions to avoid such problem in the future.

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
7c9fdd9c3a MEDIUM: quic: move receive out of FD handler to quic-conn io-cb
This change is the second part for reception on QUIC connection socket.
All operations inside the FD handler has been delayed to quic-conn
tasklet via the new function qc_rcv_buf().

With this change, buffer management on reception has been simplified. It
is now possible to use a local buffer inside qc_rcv_buf() instead of
quic_receiver_buf().

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
5b41486b7f MEDIUM: quic: use quic-conn socket for reception
Try to use the quic-conn socket for reception if it is allocated. For
this, the socket is inserted in the fdtab. This will call the new
handler quic_conn_io_cb() which is responsible to process the recv()
system call. It will reuse datagram dispatch for simplicity. However,
this is guaranteed to be called on the quic-conn thread, so it will be
more efficient to use a dedicated buffer. This will be implemented in
another commit.

This patch should improve performance by reducing contention on the
receiver socket. However, more gain can be obtained when the datagram
dispatch operation will be skipped.

Older quic_sock_fd_iocb() is renamed to quic_lstnr_sock_fd_iocb() to
emphasize its usage for the receiver socket.

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
dc0dcb394b MINOR: quic: use connection socket for emission
If quic-conn has a dedicated socket, use it for sending over the
listener socket. This should improve performance by reducing contention
over the shared listener socket.

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
40909dfec5 MINOR: quic: allocate a socket per quic-conn
Allocate quic-conn owned socket if possible. This requires that this is
activated in haproxy configuration. Also, this is done only if local
address is known so it depends on the support of IP_PKTINFO.

For the moment this socket is not used. This causes QUIC support to be
broken as received datagram are not read. This commit will be completed
by a following patch to support recv operation on the newly allocated
socket.

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
511ddd5785 MINOR: quic: define config option for socket per conn
Define global configuration option "tune.quic.socket-owner". This option
can be used to activate or not socket per QUIC connection mode. The
default value is "listener" which disable this feature. It can be
activated with the option "connection".

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
8d46acdfcb MINOR: quic: test IP_PKTINFO support for quic-conn owned socket
Extend the startup platform detection support test for quic-conn owned
socket. It is required to be able to retrieve destination address on a
recvfrom() system call so check if IP_PKTINFO or IP_RECVDSTADDR flags
are supported.

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
75839a44e7 MINOR: quic: startup detect for quic-conn owned socket support
To be able to use individual sockets for QUIC connections, we rely on
the OS network stack which must support UDP sockets binding on the same
local address.

Add a detection code for this feature executed on startup. When the
first QUIC listener socket is binded, a test socket is created and
binded on the same address. If the bind call fails, we consider that
it's impossible to use individual socket for QUIC connections.

A new global option GTUNE_QUIC_SOCK_PER_CONN is defined. If startup
detect fails, this value is resetted from global options. For the
moment, there is no code to activate the option : this will be in a
follow-up patch with the introduction of a new configuration option.

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
eb6be98a65 MINOR: quic: ignore address migration during handshake
QUIC protocol support address migration which allows to maintain the
connection even if client has changed its network address. This is done
through address migration.

RFC 9000 stipulates that address migration is forbidden before handshake
has been completed. Add a check for this : drop silently every datagram
if client network address has changed until handshake completion.

This commit is one of the first steps towards QUIC connection migration
support.

This should be backported up to 2.7.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
eec0b3c1bd MINOR: quic: detect connection migration
Detect connection migration attempted by the client. This is done by
comparing addresses stored in quic-conn with src/dest addresses of the
UDP datagram.

A new function qc_handle_conn_migration() has been added. For the
moment, no operation is conducted and the function will be completed
during connection migration implementation. The only notable things is
the increment of a new counter "quic_conn_migration_done".

This should be backported up to 2.7.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
21e611dc89 MINOR: tools: add port for ipcmp as optional criteria
Complete ipcmp() function with a new argument <check_port>. If this
argument is true, the function will compare port values besides IP
addresses and return true only if both are identical.

This commit will simplify QUIC connection migration detection. As such,
it should be backported to 2.7.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
8687b63c69 MINOR: quic: extract datagram parsing code
Extract individual datagram parsing code outside of datagrams list loop
in quic_lstnr_dghdlr(). This is moved in a new function named
quic_dgram_parse().

To complete this change, quic_lstnr_dghdlr() has been moved into
quic_sock source file : it belongs to QUIC socket lower layer and is
directly called by quic_sock_fd_iocb().

This commit will ease implementation of quic-conn owned socket.
New function quic_dgram_parse() will be easily usable after a receive
operation done on quic-conn IO-cb.

This should be backported up to 2.7.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
3f474e64c8 MINOR: quic: complete traces in qc_rx_pkt_handle()
Add missing ENTER trace for qc_rx_pkt_handle() function. LEAVE traces
are already present.

This should be backported up to 2.7.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
518c98f150 MINOR: quic: remove qc from quic_rx_packet
quic_rx_packet struct had a reference to the quic_conn instance. This is
useless as qc instance is always passed through function argument. In
fact, pkt.qc is used only in qc_pkt_decrypt() on key update, even though
qc is also passed as argument.

Simplify this by removing qc field from quic_rx_packet structure
definition. Also clean up qc_pkt_decrypt() documentation and interface
to align it with other quic-conn related functions.

This should be backported up to 2.7.
2022-12-02 14:45:43 +01:00
William Lallemand
52ddd99940 MEDIUM: ssl: rename the struct "cert_key_and_chain" to "ckch_data"
Rename the structure "cert_key_and_chain" to "ckch_data" in order to
avoid confusion with the store whcih often called "ckchs".

The "cert_key_and_chain *ckch" were renamed "ckch_data *data", so we now
have store->data instead of ckchs->ckch.

Marked medium because it changes the API.
2022-12-02 11:48:30 +01:00
Aurelien DARRAGON
d06b9c8b99 DOC/MINOR: api: add documentation for event_hdl feature
This is an initial work for the dedicated
event handler API internal documentation.

The file is located at doc/internals/api/event_hdl.txt

event_hdl feature has been introduced with:
	MINOR: event_hdl: add event handler base api
2022-12-02 09:40:52 +01:00
Aurelien DARRAGON
68e692da02 MINOR: event_hdl: add event handler base api
Adding base code to provide subscribe/publish API for internal
events processing.

event_hdl provides two complementary APIs, both are implemented
in src/event_hdl.c and include/haproxy/event_hdl{-t.h,.h}:

	One API targeting developers that want to register event handlers
	that will be notified on specific events.
	(SUBSCRIBE)

	One API targeting developers that want to notify registered handlers
	about an event.
	(PUBLISH)

This feature is being considered to address the following scenarios:
	- mailers code refactoring (getting rid of deprecated
	tcp-check ruleset implementation)
	- server events from lua code (registering user defined
	lua function that is executed with relevant data when a
	server is dynamically added/removed or on server state change)
	- providing a stable and easy to use API for upcoming
	developments that rely on specific events to perform actions.
	(e.g: ressource cleanup when a server is deleted from haproxy)

At this time though, we don't have much use cases in mind in addition to
server events handling, but the API is aimed at being multipurpose
so that new event families, with their own particularities, can be
easily implemented afterwards (and hopefully) without requiring breaking
changes to the API.

Moreover, you should know that the API was not designed to cope well
with high rate event publishing.
Mostly because publishing means iterating over unsorted subscriber list.
So it won't scale well as subscriber list increases, but it is intended in
order to keep the code simple and versatile.

Instead, it is assumed that events implemented using this API
should be periodic events, and that events related to critical
io/networking processing should be handled using
dedicated facilities anyway.
(After all, this is meant to be a general purpose event API)

Apart from being easily extensible, one of the main goals of this API is
to make subscriber code as simple and safe as possible.

This is done by offering multiple event handling modes:
	- SYNC mode:
		publishing code directly
		leverages handler code (callback function)
		and handler code has a direct access to "live" event data
		(pointers mostly, alongside with lock hints/context
		so that accessing data pointers can be done properly)
	- normal ASYNC mode:
		handler is executed in a backward compatible way with sync mode,
		so that it is easy to switch from and to SYNC/ASYNC mode.
		Only here the handler has access to "offline" event data, and
		not "live" data (ptrs) so that data consistency is guaranteed.
		By offline, you should understand "snapshot" of relevant data
		at the time of the event, so that the handler can consume it
		later (even if associated ressource is not valid anymore)
	- advanced ASYNC mode
		same as normal ASYNC mode, but here handler is not a function
		that is executed with event data passed as argument: handler is a
		user defined tasklet that is notified when event occurs.
		The tasklet may consume pending events and associated data
		through its own message queue.

ASYNC mode should be considered first if you don't rely on live event
data and you wan't to make sure that your code has the lowest impact
possible on publisher code. (ie: you don't want to break stuff)

Internal API documentation will follow:
	You will find more details about the notions we roughly approached here.
2022-12-02 09:40:52 +01:00
scientiamobile
6d6787ba7c LICENSE: wurfl: clarify the dummy library license.
This clarifies that LGPL is also permitted for the wurfl.h dummy file.
Should be backported where relevant.

Signed-off-by: Luca Passani <luca.passani@scientiamobile.com>
2022-12-02 05:32:46 +01:00
Willy Tarreau
b59e3f6045 MINOR: debug: add a balance of alloc - free at the end of the memstats dump
When digging into suspected memory leaks, it's cumbersome to count the
number of allocations and free calls. Here we're adding a summary at the
end of the sum of allocs minus the sum of frees, excluding realloc since
we can't know how much it releases upon each call. This means that when
doing many realloc+free the count may be negative but in practice there
are very few reallocs so that's not a problem. Also the size/call is signed
and corresponds to the average size allocated (e.g. leaked) per call.

It seems to work reasonably well for now:

  > debug dev memstats match buf
  quic_conn.c:2978       P_FREE  size:   1239547904  calls:     75656  size/call:  16384 buffer
  quic_conn.c:2960      P_ALLOC  size:   1239547904  calls:     75656  size/call:  16384 buffer
  mux_quic.c:393        P_ALLOC  size:   9112780800  calls:    556200  size/call:  16384 buffer
  mux_quic.c:383        P_ALLOC  size:  17783193600  calls:   1085400  size/call:  16384 buffer
  mux_quic.c:159         P_FREE  size:   8935833600  calls:    545400  size/call:  16384 buffer
  mux_quic.c:142         P_FREE  size:   9112780800  calls:    556200  size/call:  16384 buffer
  h3.c:776              P_ALLOC  size:   8935833600  calls:    545400  size/call:  16384 buffer
  quic_stream.c:166      P_FREE  size:    975241216  calls:     59524  size/call:  16384 buffer
  quic_stream.c:127      P_FREE  size:   7960592384  calls:    485876  size/call:  16384 buffer
  stream.c:772           P_FREE  size:      8798208  calls:       537  size/call:  16384 buffer
  stream.c:768           P_FREE  size:      2424832  calls:       148  size/call:  16384 buffer
  stream.c:751          P_ALLOC  size:   8852062208  calls:    540287  size/call:  16384 buffer
  stream.c:641           P_FREE  size:   8849162240  calls:    540110  size/call:  16384 buffer
  stream.c:640           P_FREE  size:   8847360000  calls:    540000  size/call:  16384 buffer
  channel.h:850         P_ALLOC  size:      2441216  calls:       149  size/call:  16384 buffer
  channel.h:850         P_ALLOC  size:      5914624  calls:       361  size/call:  16384 buffer
  dynbuf.c:55            P_FREE  size:        32768  calls:         2  size/call:  16384 buffer
  Total                 BALANCE  size:            0  calls:   5606906  size/call:      0 (excl. realloc)

Let's see how useful this becomes over time.
2022-12-01 16:12:21 +01:00
Willy Tarreau
e57fbed3c4 MINOR: debug: support pool filtering on "debug dev memstats"
Sometimes when debugging it's convenient to be able to focus only on
certain pools. Just like we did for "show pools", let's add a filter
based on a prefix on "debug dev memstats match <prefix>".
2022-12-01 16:12:21 +01:00
Dragan Dosen
a9800a0f58 MEDIUM: 51d: add support for 51Degrees V4 with Hash algorithm
This patch also adds a set of new global options:

- 51degrees-use-performance-graph { on | off }
- 51degrees-use-predictive-graph { on | off }
- 51degrees-drift <number>
- 51degrees-difference <number>
- 51degrees-allow-unmatched { on | off }

To build using the latest 51Degrees V4 engine with Hash algorithm, set
USE_51DEGREES_V4=1.

Other supported build options are 51DEGREES_INC, 51DEGREES_LIB and
51DEGREES_SRC which needs to be set to the directory that contains
headers and C files. For example:

make TARGET=<target> USE_51DEGREES_V4=1 51DEGREES_SRC='51D_REPO_PATH'/src
2022-12-01 16:12:21 +01:00
Willy Tarreau
eaded987ee [RELEASE] Released version 2.8-dev0
Released version 2.8-dev0 with the following main changes :
    - MINOR: version: mention that it's development again
2022-12-01 15:25:34 +01:00
Willy Tarreau
989c55dc2f MINOR: version: mention that it's development again
This essentially reverts d705b85a4a43e9be5c99796e2b83197b42d6918e.
2022-12-01 15:24:10 +01:00
Willy Tarreau
437fd289f2 [RELEASE] Released version 2.7.0
Released version 2.7.0 with the following main changes :
    - MINOR: ssl: forgotten newline in error messages on ca-file
    - BUG/MINOR: ssl: shut the ca-file errors emitted during httpclient init
    - DOC: config: provide some configuration hints for "http-reuse"
    - DOC: config: refer to section about quoting in the "add_item" converter
    - DOC: halog: explain how to use -ac and -ad in the help message
    - DOC: config: clarify the fact that SNI should not be used in HTTP scenarios
    - DOC: config: mention that a single monitor-uri rule is supported
    - DOC: config: explain how default matching method for ACL works
    - DOC: config: clarify the fact that "retries" is not just for connections
    - BUILD: halog: fix missing double-quote at end of help line
    - DOC: config: clarify the -m dir and -m dom pattern matching methods
    - MINOR: activity: report uptime in "show activity"
    - REORG: activity/cli: move the "show activity" handler to activity.c
    - DEV: poll: add support for epoll
    - DEV: tcploop: centralize the polling code into wait_for_fd()
    - DEV: tcploop: add support for POLLRDHUP when supported
    - DEV: tcploop: do not report an error on POLLERR
    - DEV: tcploop: add optional support for epoll
    - SCRIPTS: announce-release: add a link to the data plane API
    - CLEANUP: stick-table: fill alignment holes in the stktable struct
    - MINOR: stick-table: store a per-table hash seed and use it
    - MINOR: stick-table: show the shard number in each entry's "show table" output
    - CLEANUP: ncbuf: remove ncb_blk args by value
    - CLEANUP: ncbuf: inline small functions
    - CLEANUP: ncbuf: use standard BUG_ON with DEBUG_STRICT
    - BUG/MINOR: quic: Endless loop during retransmissions
    - MINOR: mux-h2: add the expire task and its expiration date in "show fd"
    - BUG/MINOR: peers: always initialize the stksess shard value
    - REGTESTS: fix peers-related regtests regarding "show table"
    - BUG/MEDIUM: mux-h1: Close client H1C on EOS when there is no output data
    - MINOR: stick-table: change the API of the function used to calculate the shard
    - CLEANUP: peers: factor out the key len calculation in received updates
    - BUG/MINOR: peers: always update the stksess shard number on incoming updates
    - CLEANUP: assorted typo fixes in the code and comments
    - MINOR: mux-h1: add the expire task and its expiration date in "show fd"
    - MINOR: debug: improve error handling on the memstats command parser
    - BUILD: quic: allow build with USE_QUIC and USE_OPENSSL_WOLFSSL
    - CLEANUP: anon: clarify the help message on "debug dev hash"
    - MINOR: debug: relax access restrictions on "debug dev hash" and "memstats"
    - SCRIPTS: run-regtests: add a version check
    - MINOR: version: mention that it's stable now
2022-12-01 15:16:46 +01:00
Willy Tarreau
d705b85a4a MINOR: version: mention that it's stable now
This version will be maintained up to around Q1 2024. The INSTALL file
also mentions it.
2022-12-01 15:15:24 +01:00
Willy Tarreau
9d5e11682c SCRIPTS: run-regtests: add a version check
It happens from time to time while switching between branches and/or
updating after someone else's changes that regtests are run by accident
on the wrong binary, typically the one the tests were run on during
development and not with the latest adaptations. And obviously it's
when this happens that we break the CI. There are various causes to
this but they all come down to humans context-switching a lot, and
there's no real fix for this that doesn't add even more burden hence
increases the overhead. However we can help the human detect such
mistakes very easily.

This change here will compare the version of the haproxy binary to
the version of the tree, and will emit a warning in the regtest output
if they do not match, regardless of the outcome of the test. This is
sufficient in case of failures because these are quickly glanced over,
and is sufficient as well in case of accidental success because the
warning is the last message. E.g:

  ########################## Starting vtest ##########################
  Testing with haproxy version: 2.7-dev10-cfcdbc-38
  Warning: version does not match the current tree (2.7-dev10-111c78-39)
  0 tests failed, 0 tests skipped, 182 tests passed

This should not affect builds made out of a git tree because the version
is retrieved using "make version", or exactly the same way as it's passd
to the haproxy binary. We just need to know what "make" command to run,
so $MAKE is used primarily, falling back to "make" then to "gmake". In
case all of these fail, we just ignore the version check. This should be
sufficient to catch human mistakes without affecting the CI.
2022-11-30 18:44:33 +01:00
Willy Tarreau
111c78329e MINOR: debug: relax access restrictions on "debug dev hash" and "memstats"
These two have absolutely zero impact on the process and do not need to
be restricted to the expert mode. The first one calculates a string hash
that can be used by anyone when checking a dump; the second one may be
used by anyone tracking a memory leak, and is cumbersome to use due to
the "expert-mode on" that needs to be prepended. In addition this gives
bad habits to users and needlessly taints the process. So let's drop
this restriction for these two commands.
2022-11-30 17:58:00 +01:00
Willy Tarreau
50dd7e95c8 CLEANUP: anon: clarify the help message on "debug dev hash"
This command is used to hash a section name using the current anon key,
it was brought in 2.7 by commit 54966dffd ("MINOR: anon: store the
anonymizing key in the CLI's appctx"). However the help message only
says "return msg hashed" which is misleading because if anon mode is
not enabled, it returns the string as-is. Let's just mention this
condition in the help message, and also fix the alphabetical ordering
and alignment on the line.
2022-11-30 17:58:00 +01:00
Stefan Eissing
b82296c10e BUILD: quic: allow build with USE_QUIC and USE_OPENSSL_WOLFSSL
WolfSSL does not implement the TLS1_3_CK_AES_128_CCM_SHA256 cipher as
well as the SSL_ERROR_WANT_ASYNC, SSL_ERROR_WANT_ASYNC_JOB and
SSL_ERROR_WANT_CLIENT_HELLO_CB error codes.

This patch disables them for WolfSSL.

Signed-off-by: William Lallemand <wlallemand@haproxy.org>
2022-11-30 17:38:27 +01:00
Willy Tarreau
334d091b75 MINOR: debug: improve error handling on the memstats command parser
"debug dev memstats" supports various options but silently ignores the
unknown ones. Let's make sure it returns indications about what it
expects, as the help message is quite limited otherwise.
2022-11-30 17:24:29 +01:00
Christopher Faulet
38f61351c3 MINOR: mux-h1: add the expire task and its expiration date in "show fd"
Just like for the H2 multiplexer, info about the H1 connection task is now
displayed in "show fd" output. The task pointer is displayed and, if not
null, its expiration date.

It may be useful to backport it.
2022-11-30 14:49:56 +01:00