19083 Commits

Author SHA1 Message Date
Christopher Faulet
a92480462c MINOR: http-rules: Add missing actions in http-after-response ruleset
This patch adds the support of following actions in the http-after-response
ruleset:

  * set-map, del-map and del-acl
  * set-log-level
  * sc-inc-gpc, sc-inc-gpc0 and set-inc-gpc1
  * sc-inc-gpt and sc-set-gpt0

This patch should solve the issue #1980.
2023-01-05 11:23:59 +01:00
Christopher Faulet
d9d36b8b6b DOC: config: fix alphabetical ordering of http-after-response rules
The 'capture' action must be placed after the 'allow' action.

This patch could be backported as far as 2.5.
2023-01-05 10:25:32 +01:00
Christopher Faulet
61aded057d BUG/MAJOR: buf: Fix copy of wrapping output data when a buffer is realigned
There is a bug in b_slow_realign() function when wrapping output data are
copied in the swap buffer. block1 and block2 sizes are inverted. Thus blocks
with a wrong size are copied. It leads to data mixin if the first block is
in reality larger than the second one or to a copy of data outside the
buffer is the first block is smaller than the second one.

The bug was introduced when the buffer API was refactored in 1.9. It was
found by a code review and seems never to have been triggered in almost 5
years. However, we cannot exclude it is responsible of some unresolved bugs.

This patch should fix issue #1978. It must be backported as far as 2.0.
2023-01-05 09:34:49 +01:00
Christopher Faulet
31850b470a BUG/MINOR: http-fetch: Only fill txn status during prefetch if not already set
When an HTTP sample fetch is evaluated, a prefetch is performed to check the
channel contains a valid HTTP message. If the HTTP analysis was not already
started, some info are filled.

It may be an issue when an error is returned before the response analysis
and when http-after-response rules are used because the original HTTP txn
status may be crushed. For instance, with the following configuration:

  listen l1
    log global
    mode http
    bind :8000

    log-format ST=%ST
    http-after-response set-status 400
    #http-after-response set-var(res.foo) status

A "ST=503" is reported in the log messages, independantly on the first
http-after-response rule. The same must happen if the second rule is
uncommented. However, for now, a "ST=400" is logged.

To fix the bug, during the prefetch, the HTTP txn status is only set if it
is undefined (-1). This way, we are sure the original one is never lost.

This patch should be backported as far as 2.2.
2023-01-05 09:33:23 +01:00
Tim Duesterhus
18cd4746e5 REGTESTS: Remove tests with REQUIRE_VERSION_BELOW=1.9
HAProxy 2.0 is the lowest supported version, thus this never matches.
2023-01-05 09:11:38 +01:00
Tim Duesterhus
7aff1bf6b9 REGTESTS: Remove REQUIRE_VERSION=2.0 from all tests
HAProxy 2.0 is the lowest supported version, thus this always matches.

see 1b095cac9468d0c3eeb157e9b1a2947487bd3c83
2023-01-05 09:11:38 +01:00
Tim Duesterhus
2a5fb62ad1 REGTESTS: Remove REQUIRE_VERSION=1.9 from all tests
HAProxy 2.0 is the lowest supported version, thus this always matches.

see 1b095cac9468d0c3eeb157e9b1a2947487bd3c83
2023-01-05 09:11:38 +01:00
Sbastien Gross
537b9e7f36 MINOR: config: add environment variables for default log format
This patch provides a convenient way to override the default TCP, HTTP
and HTTP log formats. Instead of having a look into the documentation
to figure out what is the appropriate default log format three new
environment variables can be used: HAPROXY_TCP_LOG_FMT,
HAPROXY_HTTP_LOG_FMT and HAPROXY_HTTPS_LOG_FMT. Their content are
substituted verbatim.

These variables are set before parsing the configuration and are unset
just after all configuration files are successful parsed.

Example:

    # Instead of writing this long log-format line...
    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC \
                %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r \
                lr=last_rule_file:last_rule_line"

    # ..the HAPROXY_HTTP_LOG_FMT can be used to provide the default
    # http log-format string
    log-format "${HAPROXY_HTTP_LOG_FMT} lr=last_rule_file:last_rule_line"

Please note that nothing prevents users to unset the variables or
override their content in a global section.

Signed-off-by: Sbastien Gross <sgross@haproxy.com>
2023-01-04 08:23:43 +01:00
Tim Duesterhus
081091187e CI: Reformat matrix.py using black
The initial version of matrix.py was formatted using `black` [1], but with all
the later changes, the formatting diverged quite a bit. This patch reformats
the script using black, fixing the indentation of some statements and
shortening overlong lines.

[1] https://github.com/psf/black
2023-01-03 16:28:34 +01:00
Tim Duesterhus
e327e41430 CI: Explicitly check environment variable against None in matrix.py
For consistency with `GITHUB_OUTPUT` at the bottom.
2023-01-03 16:28:34 +01:00
Tim Duesterhus
96e03e1a89 CI: Unify the GITHUB_TOKEN name across matrix.py and vtest.yml
This makes naming a little clearer in matrix.py, because the name matches the
name of the actual secret.
2023-01-03 16:28:34 +01:00
Tim Duesterhus
fd7931e7f3 CI: Use proper if blocks instead of conditional expressions in matrix.py
For complex expressions, such as the ones modified, the condition expression is
much less readable, especially with the actual condition in the middle of the
"then" and "else" part.
2023-01-03 16:28:34 +01:00
Tim Duesterhus
fcc21be922 CI: Add in-memory cache for the latest OpenSSL/LibreSSL
These functions were previously called once per compiler. Add the `lru_cache`
decorator to only perform one HTTP request each.
2023-01-03 16:28:34 +01:00
Tim Duesterhus
3c9b6f6394 CI: Improve headline in matrix.py
Since 4a04cd35ae89bf6a3bb7620f7a49804de3240ac4 (CI: github: split ssl lib
selection based on git branch) the branch, instead of the workflow type is
passed. The headline should reflect that.
2023-01-03 16:28:34 +01:00
Willy Tarreau
20391519c3 BUG/MINOR: stick-table: report the correct action name in error message
sc-inc-gpc() learned to use arrays in 2.5 with commit 4d7ada8f9 ("MEDIUM:
stick-table: add the new arrays of gpc and gpc_rate"), but the error
message says "sc-set-gpc" instead of "sc-inc-gpc". Let's fix this to
avoid confusion.

This can be backported to 2.5.
2023-01-02 17:35:50 +01:00
Remi Tricot-Le Breton
c389b04bc5 BUG/MINOR: ssl: Missing goto in error path in ocsp update code
When converting an OCSP request's information into base64, the return
value of a2base64 is checked but processing is not interrupted when it
returns a negative value, which was caught by coverity.

This patch fixes GitHub issue #1974.
It does not need to be backported.
2023-01-02 15:21:57 +01:00
Willy Tarreau
c57fb3be75 MINOR: cfgparse-ssl: avoid a possible crash on OOM in ssl_bind_parse_npn()
Upon out of memory condition at boot, we could possibly crash when
parsing the "npn" bind line keyword since it's used unchecked. There's
no real need to backport this though it will not hurt.
2023-01-02 09:51:35 +01:00
Willy Tarreau
b5662519df BUG/MINOR: debug: don't mask the TH_FL_STUCK flag before dumping threads
Commit f0c86ddfe ("BUG/MEDIUM: debug: fix parallel thread dumps again")
added a clearing of the TH_FL_STUCK flag before dumping threads in case
of parallel dumps, but that was in part a sort of workaround for some
remains of the commit that introduced the flag in 2.0 before the watchdog
existed, and which would set it after dumping a thread: e6a02fa65 ("MINOR:
threads: add a "stuck" flag to the thread_info struct"), and in part an
attempt to avoid that a thread waiting for too long during the dump would
get the flag set. But that is not possible, a thread waiting for being
dumped has the harmless bit set and doesn't get the stuck bit. What happens
in fact is that issuing "show threads" in fast loops ends up causing some
threads to keep their STUCK bit that was set at the end of "show threads",
and confuses the output.

The problem with doing this is that the flag is cleared before the thread
is dumped, and since this flag is used to decide whether to show a backtrace
or not, we don't get backtraces anymore of stuck threads since the commit
above in 2.7.

This patch just removes the two points where the flag was cleared by the
commit above. It should be backported to 2.7.
2023-01-02 09:51:35 +01:00
Willy Tarreau
23078c8f2f BUILD: makefile: also list per-option settings in 'make opts'
Thanks to the generic naming of the build options, it's now relatively
easy to enumerate all _CFLAGS and _LDFLAGS for defined USE_* options.
That was added to the first line of 'make opts', but is only listed for
enabled options, non-empty variables or cmd-line defined variables.
2022-12-23 17:33:53 +01:00
Willy Tarreau
6e70a3986c BUILD: makefile: only consider settings from enabled options
Due to the previous SSL exception we coudln't restrict the collected
CFLAGS/LDFLAGS to those of enabled options, so all of them were
considered if set. The problem is that it would prevent simply
disabling a build option without unsetting its xxx_CFLAGS or _LDFLAGS
values if those had incompatible values (e.g. -lfoo).

Now that only existing options are listed in collect_opts_flags, we
can safely check that the option is set and only consider its settings
in this case. Thus OT_LDFLAGS will not be used if USE_OT is not set
for example.
2022-12-23 17:01:55 +01:00
Willy Tarreau
6a2cd33509 BUILD: makefile: remove the special case of the SSL option
By creating USE_SSL and enabling it when USE_OPENSSL is set, we can
get rid of the special case that was made with it regarding cflags
collect and when resetting options. The option doesn't need to be
manually set, though in the future it might prove useful if other
non-openssl API are supported.
2022-12-23 16:53:35 +01:00
Willy Tarreau
2b8d0978f3 BUILD: makefile: make all OpenSSL variants use the same settings
It's getting complicated to configure includes and lib dirs for
OpenSSL API variants such as WolfSSL, because some settings are
common and others are specific but carry a prefix that doesn't
match the USE_* rule scheme.

This patch simplifies everything by considering that all SSL libs
will use SSL_INC, SSL_LIB, SSL_CFLAGS and SSL_LDFLAGS. That's much
more convenient. This works thanks to the settings collector which
explicitly checks the SSL_* settings. When USE_OPENSSL_WOLFSSL is
set, then USE_OPENSSL is implied, so that there's no need to
duplicate maintenance effort.
2022-12-23 16:53:35 +01:00
Willy Tarreau
323b50b0f1 BUILD: makefile: support WURFL_CFLAGS and WURFL_LDFLAGS
The CFLAGS and LDFLAGS appended by USE_WURFL can now be overridden
using WURFL_CFLAGS and WURFL_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
3f2803e5cb BUILD: makefile: refactor support for 51DEGREES v3/v4
In order to simplify maintenance and long-term evolutions, now the
feature remains enabled by setting USE_51DEGREES=1 and the version
is set in 51DEGREES_VER (3 or 4 are supported only). The default
version remains 3. All 51DEGREES flags are shared between both
versions and only use the "51DEGREES_" prefix.

The related CFLAGS and LDFLAGS can now be overridden using
51DEGREES_CFLAGS and 51DEGREES_LDFLAGS, both of which are automatically
collected into the respective OPTIONS_*. The USE_51DEGREES_V4 option is
now removed, and the doc was updated.
2022-12-23 16:53:35 +01:00
Willy Tarreau
6985e2a9a6 BUILD: makefile: support PCRE[2]_CFLAGS and PCRE[2]_LDFLAGS
The CFLAGS and LDFLAGS appended by USE_PCRE/USE_PCRE2 can now be
overridden using PCRE_CFLAGS/PCRE2_CFLAGS and PCRE_LDFLAGS/PCRE2_LDFLAGS.

Its worth noting that PCRE2_LDFLAGS did already exist and was preset from
the pkgconfig output then complemented with -lpcre2-posix, and only then
the -L and optional -Wl,-Bstatic were appended when adding them to the
resulting global LDFLAGS. A search on the net did not reveal any use of
PCRE2_LDFLAGS in any public build scripts, and for consistency sake it's
important to make sure that we can now finally override the -L settings
like we're able to do with every other build option. Thus the meaning of
this variable changed to include all the related ldflags (-L and -Wl).

These flags are now automatically collected into OPTIONS_*.
2022-12-23 16:53:35 +01:00
Willy Tarreau
202295436c BUILD: makefile: support DEVICEATLAS_CFLAGS and DEVICEATLAS_LDFLAGS
The CFLAGS and LDFLAGS appended by USE_DEVICEATLAS can now be overridden
using DEVICEATLAS_CFLAGS and DEVICEATLAS_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
b2ef365de2 BUILD: makefile: support LUA_CFLAGS and LUA_LDFLAGS
The CFLAGS and LDFLAGS appended by USE_LUA can now be overridden using
LUA_CFLAGS and LUA_LDFLAGS. Note that if these flags are forced, they
have to contain the optional -DHLUA_PREPEND_PATH= since this is added
to CFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
7f48b6f1a4 BUILD: makefile: support WOLFSSL_CFLAGS and WOLFSSL_LDFLAGS
The CFLAGS and LDFLAGS appended by USE_WOLFSSL can now be overridden
using WOLFSSL_CFLAGS and WOLFSSL_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
e609ac1bbe BUILD: makefile: support OPENSSL_CFLAGS and OPENSSL_LDFLAGS
The CFLAGS and LDFLAGS appended by USE_OPENSSL can now be overridden
using OPENSSL_CFLAGS and OPENSSL_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
a88cc63d09 BUILD: makefile: support ENGINE_CFLAGS
The CFLAGS appended by USE_ENGINE can now be overridden using
ENGINE_CFLAGS. These would have been better located inside the
OPENSSL stuff but it's a bit too late now.
2022-12-23 16:53:35 +01:00
Willy Tarreau
d957924e68 BUILD: makefile: support ZLIB_CFLAGS and ZLIB_LDFLAGS
The CFLAGS and LDFLAGS appended by USE_ZLIB can now be overridden using
ZLIB_CFLAGS and ZLIB_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
f852d31145 BUILD: makefile: support SYSTEMD_LDFLAGS
The LDFLAGS appended by USE_SYSTEMD can now be overridden using
SYSTEMD_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
eb12fcc40a BUILD: makefile: support BACKTRACE_LDFLAGS
The LDFLAGS appended by USE_BACKTRACE can now be overridden using
BACKTRACE_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
3b66494d25 BUILD: makefile: support THREAD_LDFLAGS
The LDFLAGS appended by USE_THREAD can now be overridden using
THREAD_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
84e3283be3 BUILD: makefile: support RT_LDFLAGS
The LDFLAGS appended by USE_RT can now be overridden using
RT_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
2b5c9f7b4f BUILD: makefile: support LIBCRYPT_LDFLAGS
The LDFLAGS appended by USE_LIBCRYPT can now be overridden using
LIBCRYPT_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
c108f37c2a BUILD: makefile: rework 51D to split v3/v4
There are multiple options for 51DEGREES, v3/v4, threading or not,
pattern/trie for v3, vhash for v4, use of libatomic, etc. While the
current rules deal with all of that correctly, it's too difficult to
focus on one version because the two are interleaved for every single
option. Let's just split them into two independent blocks. This removes
some if/endif, and makes the lecture much more straightforward.
2022-12-23 16:53:35 +01:00
Willy Tarreau
f6ceb0ec4e CLEANUP: makefile: properly indent ifeq/ifneq conditional blocks
Some conditional blocks have become out of control over time and are
totally unreadble. It took 15 minutes to figure what "endif" matched
what "if" in the PCRE one for example, and DA and 51D use multiple
levels as well that are not easy to sort out.

Let's reindent the whole thing. Most places that were already indented
used 2 spaces per level, so here we're keeping that principle. It was
just not done on the two last ones that are used to define some rules
because we don't want spaces before rule names. A few had the opening
condition indicated on the endif line.

It would be desirable that over time this more maintainable layout is
preserved.
2022-12-23 16:53:35 +01:00
Willy Tarreau
8ea58f5c76 BUILD: makefile: properly report USE_PCRE/USE_PCRE2 in features
The PCRE/PCRE2 CFLAGS forcefully add -DUSE_PCRE or -DUSE_PCRE2 because
we want that USE_STATIC_PCRE or USE_PCRE_JIT implicitly enables them.
However, doing it this way is incorrect because the option is not visible
in BUILD_FEATURES, and for example, some regtests depending on such
features (such as map_redirect.vtc) would be skipped if only the static
or jit versions are enabled.

The correct way to do this is to always set USE_PCRE feature for such
variants instead of adding the define.

This could almost be backported but would require to backport other
makefile patches and likely only has effects on the reg-tests at the
moment, so it's probably not worth the hassle.
2022-12-23 16:53:35 +01:00
Willy Tarreau
f297b18eac BUILD: makefile: add an implicit USE_MATH variable for -lm
Lua and 51d make use of -lm, which would be better served by having its
own option than being passed in the LDFLAGS. It also simplifies linking
against a static version of libm. The option uses its own LDFLAGS which
are automatically collected into OPTIONS_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
b16d9b5816 BUILD: makefile: never force -latomic, set USE_LIBATOMIC instead
Two places, 51Dv4 and AIX7.2, used to forcefully add -latomic to the
ldflags (and via different variables). This must not be done because
it depends on compiler, arch, etc. USE_LIBATOMIC=implicit is much
better: it allows the user to forcefully disable it if undesired.
The LIBATOMIC_LDFLAGS are set to -latomic and automatically added
to OPTIONS_LDFLAGS.

It will make this dependency appear in haproxy -vv but that's not
and issue and it may even sometimes help when troubleshooting.
2022-12-23 16:53:35 +01:00
Willy Tarreau
447247aa8d BUILD: makefile: do not restrict Lua's prepend path to empty LUA_LIB_NAME
The HLUA_PREPEND_PATH and HLUA_PREPEND_CPATH settings were only applied
when LUA_LIB_NAME was empty, otherwise they were silently ignored. Let's
take them out of that conditional block as this makes no sense to enforce
such a restriction (the main reason in fact is that this whole block is
unreadable).

Also take this opportunity to unfold the last two imbricated tests of
LUA_LIB_NAME and put comments around certain blocks to know what "endif"
matches what "if".
2022-12-23 16:53:35 +01:00
Willy Tarreau
2a3a882fe2 BUILD: makefile: make sure LUA_INC and LUA_LIB are always initialized
While LUA_INC is sometimes set in the makefile (only when LUA_LIB_NAME
is not set), LUA_LIB is never pre-initialized and faces the risk of
being accidently inherited from the environment. Let's make sure both
are properly reset first when not explicitly set. For this we always
set LUA_INC based on the autodetection if it's not set, and always
pre-initialize LUA_LIB to empty. This also helps make that block
slightly less difficult to understand.
2022-12-23 16:53:35 +01:00
Willy Tarreau
682b78b4e6 BUILD: makefile: reference libdl only once
There used to be special cases where USE_DL was only for the SSL library,
then for Lua, then was used globally, but each of them kept their own copy
of -ldl. When building on a system supporting libdl, with SSL and Lua
enabled, no less than 3 -ldl are found on the linker's command line.

What matters is only that it's close to the end, so let's remove the old
specific ones and move the global one to the end. The option now uses its
own DL_LDFLAGS that is automatically collected into OPTIONS_LDFLAGS.
2022-12-23 16:53:35 +01:00
Willy Tarreau
914a668077 BUILD: makefile: make sure to also ignore SSL_INC when using wolfssl
I got a build error when adding USE_OPENSSL_WOLFSSL to my make command
line because SSL_INC was still set and caused some conflicting headers
to be included first. There's already an exclusion test for the wolfssl
variant used for SSL_LIB, make it also cover SSL_INC to avoid this.

This may be backported to 2.7 to ease testing of wolfssl.
2022-12-23 16:53:35 +01:00
Willy Tarreau
4f1890e882 BUILD: makefile: clean the wolfssl include and lib generation rules
The default include paths for wolfssl didn't match the explicit pattern
one. This was causing some confusion about what to look for, complexifying
the rules and making /usr/local/include to be automatically included if a
path was not set.

Let's just proceed as we usually do, i.e. pass -I only when a path is
specified, so that it works similarly to openssl. Let's also simplify
the LDFLAG rule at the same time.

This may be backported to 2.7 to ease testing of wolfssl.
2022-12-23 16:53:35 +01:00
Willy Tarreau
bc2f3934f9 BUILD: makefile: ensure that all USE_* handlers appear before CFLAGS are used
It happens that a few "if USE_foo" were placed too low in the makefile,
and would mostly work by luck thanks to not using variables that were
already referenced before. The opentracing include is even trickier
because it extends OPTIONS_CFLAGS that was last read a few lines before
being included, but it only works because COPTS is defined as a macro and
not a variable, so it will be evaluated later. At least now it doesn't
touch OPTIONS_* anymore and since it's cleanly arranged, it will work by
default via the flags collector.

Let's just move these late USE_* handlers upper and place a visible
delimiter after them reminding not to add any after.
2022-12-23 16:53:35 +01:00
Willy Tarreau
ea3e67f891 BUILD: makefile: start to automatically collect CFLAGS/LDFLAGS
Now OPTIONS_CFLAGS and OPTIONS_LDFLAGS don't need to be set anymore
for options USE_xxx that set xxx_CFLAGS or xxx_LDFLAGS. These ones
will be automatically connected.

The only entry for now that was ready for this was PCRE2, so it was
adjusted so as not to append to OPTIONS_LDFLAGS anymore. More will
come later.
2022-12-23 16:53:35 +01:00
Willy Tarreau
8fa2f49f24 BUILD: makefile: add a function to collect all options' CFLAGS/LDFLAGS
The new function collect_opts_flags now scans all USE_* options defined
in use_opts and appends the corresponding *_CFLAGS and *_LDFLAGS to
OPTIONS_{C,LD}FLAGS respectively. This will be useful to get rid of all
the individual concatenations to these variables.
2022-12-23 16:53:35 +01:00
Willy Tarreau
b14e89e322 BUILD: makefile: initialize all build options' variables at once
A lot of _SRC, _INC, _LIB etc variables are set and expected to be
initialized to an empty string by default. However, an in-depth
review of all of them showed that WOLFSSL_{INC,LIB}, SSL_{INC,LIB},
LUA_{INC,LIB}, and maybe others were not always initialized and could
sometimes leak from the environment and as such cause strange build
issues when running from cascaded scripts that had exported them.

The approach taken here consists in iterating over all USE_* options
and unsetting any _SRC, _INC, _LIB, _CFLAGS and _LDFLAGS that follows
the same name. For the few variable names options that don't exactly
match the build option (SSL & WOLFSSL), these ones are specifically
added to the list. The few that were explicitly cleared in their own
sections were just removed since not needed anymore. Note that an
"undefine" command appeared in GNU make 3.82 but since we support
older ones we can only initialize the variables to an empty string
here. It's not a problem in practice.

We're now certain that these variables are empty wherever they are
used, and that it is possible to just append to them, or use them
as-is.
2022-12-23 16:53:35 +01:00