14775 Commits

Author SHA1 Message Date
Willy Tarreau
e8ceea1345 BUILD: auth: include missing list.h
list_for_each_entry() requires list.h but used to inherit it by accident
through global.h and mworker-t.h. Let's explicitly add it.
2021-05-08 12:29:51 +02:00
Willy Tarreau
7f673c2cde BUILD: wdt: include signal-t.h
WDT_SIG is used there, thus signal-t.h is required. Currently it's
retrieved by accident through global.h.
2021-05-08 12:29:01 +02:00
Willy Tarreau
29c460bc07 REORG: threads: move all_thread_mask() to thread.h
It was declared in global.h, forcing plenty of source files to include
it only for this while it's only based on definitions from thread.h.
2021-05-08 12:26:10 +02:00
Willy Tarreau
cfc4f24d80 REORG: vars: move the "proc" scope variables out of the global struct
The presence of this field causes a long dependency chain because almost
everyone includes global-t.h, and vars include sample_data which include
some system includes as well as HTTP parts.

There is absolutely no reason for having the process-wide variables in
the global struct, let's just move them into vars.c and vars.h. This
reduces from ~190k to ~170k the preprocessed output of version.c.
2021-05-08 12:11:29 +02:00
Willy Tarreau
9eec7e206e MINOR: config: mark tune.fd.edge-triggered as experimental
This one is stated as experimental in the doc but could still be used
by accidental copy-paste. Let's mark it with KWF_EXPERIMENTAL so that
users have to opt-in to use it.
2021-05-08 11:06:32 +02:00
Willy Tarreau
c5977728b3 MINOR: stats: make "show info" able to report rates as floats when asked
Now "show info float" will also report SSL rates, connection rates and
key reuse ratios as floats. This can be convenient at very low rates.

Note that the SSL reuse ratio which used to commonly oscillate between
0 and 1 under load is now more often above zero with small values. It
indicates that for better stability we shouldn't be comparing a key rate
with a connection rate but instead we should measure the reuse rate at
its source.
2021-05-08 10:52:12 +02:00
Willy Tarreau
e8abc3293f MINOR: stats: report uptime and start time as floats with subsecond resolution
When "show info float" is used, the uptime and start time will be reported
with subsecond resolution (microsecond actually since timeval is used).
2021-05-08 10:52:12 +02:00
Willy Tarreau
d37e26eaa6 MINOR: stats: use tv_remain() to precisely compute the uptime
We'll have to support reporting sub-second uptimes, so let's use the
appropriate function which will automatically adjust the tv_usec field.
In addition to this, it will also report a more accurate uptime thanks
to considering the sub-second part in the result.
2021-05-08 10:52:12 +02:00
Willy Tarreau
2745620240 MINOR: stats: support an optional "float" option to "show info"
This will allow some fields to be produced with a higher accuracy when
the requester indicates being able to parse floats. Rates and times are
among the elements which can make sense.
2021-05-08 10:52:12 +02:00
Willy Tarreau
0b26b3866c MINOR: stats: pass the appctx flags to stats_fill_info()
Currently the stats filling function knows nothing about the caller's
needs, so let's pass the STAT_* flags so that it can adapt to the
requester's constraints.
2021-05-08 10:52:12 +02:00
Willy Tarreau
6004fb7681 MINOR: stats: add the HTML conversion for float types
For the prometheus exporter, a new float type was added for the fields
and its conversion was added everywhere except for the HTML output.
Now that we have F2H() we can implement it for consistency.
2021-05-08 10:48:17 +02:00
Willy Tarreau
065ba3186e MINOR: stats: avoid excessive padding of float values with trailing zeroes
When emitting stats, we don't need to have 6 zeroes after the decimal point
for each value, so let's trim floating point numbers to the longest needed
only.
2021-05-08 10:48:17 +02:00
Willy Tarreau
aa33f20e27 MINOR: freq_ctr: add new functions to report float measurements
For stats reporting it can be convenient to report floats at low rates
instead of discrete integers. We do have quite some precision since we
currently divide counters by number of milliseconds, so we can usually
add 3 digits after the decimal point.
2021-05-08 10:48:17 +02:00
Willy Tarreau
ae03d26eea MINOR: tools: add a float-to-ascii conversion function
We already had ultoa_r() and friends but nothing to emit inline floats.
This is now done with ftoa_r() and F2A/F2H. Note that the latter both use
the itoa_str[] as temporary storage and that the HTML format currently is
the exact same as the ASCII one. The trailing zeroes are always timmed so
these outputs are usable in user-visible output.
2021-05-08 10:48:17 +02:00
Willy Tarreau
56d1d8dab0 MINOR: tools: implement trimming of floating point numbers
When using "%f" to print a float, it automatically gets 6 digits after
the decimal point and there's no way to automatically adjust to the
required ones by dropping trailing zeroes. This function does exactly
this and automatically drops the decimal point if all digits after it
were zeroes. This will make numbers more friendly in stats and makes
outputs shorter (e.g. JSON where everything is just a "number").

The function is designed to be easy to use with snprint() and chunks:

  snprintf:
    flt_trim(buf, 0, snprintf(buf, sizeof(buf), "%f", x));

  chunk_printf:
    out->data = flt_trim(out->area, 0, chunk_printf(out, "%f", x));

  chunk_appendf:
    size_t prev_data = out->data;
    out->data = flt_trim(out->area, prev_data, chunk_appendf(out, "%f", x));
2021-05-08 10:42:11 +02:00
Willy Tarreau
589722ed2c DOC: management: mention that some fields may be emitted as floats
Floats in stats output are currently not emitted but are technically
possible, so let's mention that.
2021-05-08 07:48:07 +02:00
Willy Tarreau
a1169b6231 MINOR: sample: improve error reporting on missing arg to strcmp() converter
Calling the strcmp() converter with no argument yields this strange error:

  [ALERT]    (31439) : parsing [test.cfg:3] : error detected in frontend 'f' while parsing 'http-request redirect' rule : failed to parse sample expression <src,strcmp]> : invalid args in converter 'strcmp' : failed to register variable name ''.

This is because the vars name check tries to see if it can create such a
variable having an empty name. Let's at least make a special case of the
missing argument. Now we can read a more explicit:

  [ALERT]    (31655) : parsing [test.cfg:3] : error detected in frontend 'f' while parsing 'http-request redirect' rule : failed to parse sample expression <src,strcmp]> : invalid args in converter 'strcmp' : missing variable name.

This was done for secure_strcmp() as well.
2021-05-08 06:55:25 +02:00
Remi Tricot-Le Breton
02bd68431b DOC: ssl: Add information about crl-file option
When using the crl-file option with multiple Certificate Authority
levels in the CA chain, there must be one CRL per CA or the verify
function on the backend side will raise an "unagle to get certificate
CRL" error (error code 3).

This was required by GitHub issue #1201.
2021-05-07 18:31:11 +02:00
Willy Tarreau
90f2c7f41a ADDONS: make addons/ discoverable by git via .gitignore
When the contrib/ parts were moved to addons/, no rule was added in
.gitignore to make addons/ searchable for new files, let's fix it.
2021-05-07 16:48:14 +02:00
Remi Tricot-Le Breton
7c980dffad DOC: ssl: Extra files loading now works for backends too
When implementing the server side certificate hot update, the ckch
mechanism was used on the backend side in order to mimic the frontend
certificate management and to enable server line certificate update via
the CLI (see GitHub issue #427). As an unexpected side effect, we now
also look for ssl extra files (cert.pem.key, cert.pem.ocsp ...) for the
backend side.
This patch updates the documentation accordingly.

This answers to GitHub issue #845.
2021-05-07 15:37:12 +02:00
Amaury Denoyelle
24abb0cdc1 BUG/MINOR: server: do not report diag for peer servers with null weight
Only check servers attached to a proxy with PR_CAP_LB.

This does not need to be backported as the diag message was added in the
current 2.4-dev branch.
2021-05-07 15:20:54 +02:00
Amaury Denoyelle
b979f59871 MINOR: proxy: define PR_CAP_LB
Add a new proxy capability for proxy with load-balancing capabilities.
This help to differentiate listen/frontend/backend with special proxies
such as peer proxies.
2021-05-07 15:12:20 +02:00
Amaury Denoyelle
86c1d0fddb BUILD: fix usage of ha_alert without format string
The compilation is failing due to no format string used in ha_alert.
This does not need to be backported.
2021-05-07 15:07:21 +02:00
Amaury Denoyelle
a9e639afe2 MINOR: http_act: mark normalize-uri as experimental
normalize-uri http rule is marked as experimental, so it cannot be
activated without the global 'expose-experimental-directives'. The
associated vtc is updated to be able to use it.
2021-05-07 14:35:02 +02:00
Amaury Denoyelle
5dfdf3e5b0 MINOR: stats: report tainted on show info
Add a new info field ST_F_TAINTED to dump tainted status at the end of
the 'show info' output.
2021-05-07 14:35:02 +02:00
Amaury Denoyelle
f492992065 MINOR: cli: set tainted when using CLI expert/experimental mode
Mark the process as tainted as soon as a command command only accessible
in expert or experimental mode is executed.
2021-05-07 14:35:02 +02:00
Amaury Denoyelle
0351773534 MINOR: action: implement experimental actions
Support experimental actions. It is mandatory to use
'expose-experimental-directives' before to be able to use them.

If such action is present in the config file, the tainted status of the
process is updated. Another tainted status is set when an experimental
action is executed.
2021-05-07 14:35:02 +02:00
Amaury Denoyelle
e4a617c931 MINOR: action: replace match_pfx by a keyword flags field
Define a new keyword flag KWF_MATCH_PREFIX. This is used to replace the
match_pfx field of action struct.

This has the benefit to have more explicit action declaration, and now
it is possible to quickly implement experimental actions.
2021-05-07 14:35:01 +02:00
Amaury Denoyelle
d2e53cd47e MINOR: cfgparse: implement experimental config keywords
Add a new flag to mark a keyword as experimental. An experimental
keyword cannot be used if the global 'expose-experimental-directives' is
not present first.

Only keywords parsed through a standard cfg_keywords lists in
global/proxies section will be automatically detected if declared
experimental. To support a keyword outside of these lists,
check_kw_experimental must be called manually during its parsing.

If an experimental keyword is present in the config, the tainted flag is
updated.

For the moment, no keyword is marked as experimental.
2021-05-07 14:34:41 +02:00
Amaury Denoyelle
fae9edf470 MINOR: cfgparse: add a new field flags in cfg_keyword
This field will be used to add various mechanism to config parsing.
Currently no flag value is implemented. The following commit will
implement experimental keywords.
2021-05-07 14:12:27 +02:00
Amaury Denoyelle
484454d906 MINOR: global: define tainted flag
Add a global flag named 'tainted'. Its purpose is to report various
status about experimental features used for the current process
lifetime.

By default it is initialized to 0. It can be set/retrieve by a couple of
new functions mark_tainted()/get_tainted(). Once a flag is set, it
cannot be resetted.

Currently, no tainted status is implemented, it will be the subject of
the following commits.
2021-05-07 14:12:27 +02:00
Christopher Faulet
ea86083718 BUG/MINOR: checks: Reschedule check on observe mode only if fastinter is set
On observe mode, if a server is marked as DOWN, the server's health-check is
rescheduled using the fastinter timeout if the new expiration date is newer
that the current one. But this must only be performed if the fastinter
timeout is defined.

Internally, tick_is_lt() function only checks the date and does not perform any
verification on the provided args. Thus, we must take care of it. However, it is
possible to disable the server health-check by setting its task expiration date
to TICK_ETERNITY.

This patch must be backported as far as 2.2. It is related to
2021-05-07 12:10:30 +02:00
Christopher Faulet
92017a3215 BUG/MINOR: checks: Handle synchronous connect when a tcpcheck is started
A connection may be synchronously established. In the tcpcheck context, it
may be a problem if several connections come one after another. In this
case, there is no event to close the very first connection before starting
the next one. The checks is thus blocked and timed out, a L7 timeout error
is reported.

To fix the bug, when a tcpcheck is started, we immediately evaluate its
state. Most of time, nothing is performed and we must wait. But it is thus
possible to handle the result of a successfull connection.

This patch should fix the issue #1234. It must be backported as far as 2.2.
2021-05-07 12:00:56 +02:00
Christopher Faulet
30aa0da532 BUG/MINOR: stream: Reset stream final state and si error type on L7 retry
Thanks to a previous fix, the stream error mask is now cleared on L7
retry. But the stream final state (SF_FINST_*) and the stream-interface
error type must also be reset to properly restart a new connection and be
sure to not inherit errors from the previous connection attempt.

In addition, SF_ADDR_SET flag is not systematically removed.
stream_choose_redispatch() already takes care to unset it if necessary. When
the connection is not redispatch, the server address can be preserved.

This patch must be backported as far as 2.0.
2021-05-07 12:00:56 +02:00
Willy Tarreau
b205bfdab7 CLEANUP: cli/tree-wide: properly re-align the CLI commands' help messages
There were 102 CLI commands whose help were zig-zagging all along the dump
making them unreadable. This patch realigns all these messages so that the
command now uses up to 40 characters before the delimiting colon. About a
third of the commands did not correctly list their arguments which were
added after the first version, so they were all updated. Some abuses of
the term "id" were fixed to use a more explanatory term. The
"set ssl ocsp-response" command was not listed because it lacked a help
message, this was fixed as well. The deprecated enable/disable commands
for agent/health/server were prominently written as deprecated. Whenever
possible, clearer explanations were provided.
2021-05-07 11:51:26 +02:00
Willy Tarreau
7190b987ab MINOR: config: add a new message directive: .diag
This one works just like .notice/.warning/.alert except that it prints
the message at level "DIAG" only when haproxy runs in diagnostic mode
(-dD). This can be convenient for example to pass a few hints to help
locate certain config parts or to leave messages about certain temporary
workarounds.

Example:

  .diag "WTA/2021-05-07: $.LINE: replace 'redirect' with 'return' after final switch to 2.4"
         http-request redirect location /goaway if ABUSE
2021-05-07 09:06:40 +02:00
Willy Tarreau
9f903af510 MEDIUM: log: slightly refine the output format of alerts/warnings/etc
For about 20 years we've been emitting cryptic messages on warnings and
alerts, that nobody knows how to parse:

  [NOTICE] 126/080118 (3115) : haproxy version is 2.4-dev18-0b7c78-49
  [NOTICE] 126/080118 (3115) : path to executable is ./haproxy
  [WARNING] 126/080119 (3115) : Server default/srv1 is DOWN via static/srv1. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  [ALERT] 126/080119 (3115) : backend 'default' has no server available!

Hint: the first 3-digit number is the day of year, and the 6 digits
after it represent the time of day in format HHMMSS, then the pid in
parenthesis. These are not quite user-friendly and such cryptic into
are not useful at all.

This patch slightly adjusts the output by performing these minimal changes:
  - removing the date/time, as they were added very early when haproxy
    was meant to be used in foreground as a debugging tool, and they're
    provided in more details in logs nowadays ;

  - better aligning the fields by padding the severity tag to 10 chars.
    The diag output was renamed to "DIAG" only.

Now the output provides this:

  [NOTICE]   (4563) : haproxy version is 2.4-dev18-75a428-51
  [NOTICE]   (4563) : path to executable is ./haproxy
  [WARNING]  (4563) : Server default/srv1 is DOWN via static/srv1. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
  [ALERT]    (4563) : backend 'default' has no server available!

The useless space before the colon was kept so as not to confuse any
possible output parser.

The few entries in the doc referring to this format were adjusted to
reflect the new one.

The change was tagged "MEDIUM" as it may have visible consequences on
home-grown monitoring tools, though it is extremely unlikely due to the
limited extent of these changes.
2021-05-07 08:55:11 +02:00
Willy Tarreau
75a4284bab BUG/MINOR: stream: properly clear the previous error mask on L7 retries
The cleanup of the previous error was incorrect on L7 retries, it would
OR two values while they're part of an enum, leaving some bits set.
Depending on the errors it was possible to occasionally see an internal
error ("I" flag) being logged.

This should be backported as far as 2.0, though the do_l7_retry() function
in in proto_htx.c in older versions.
2021-05-07 08:22:16 +02:00
Willy Tarreau
2639e2edc2 BUG/MINOR: activity: use the new pointer to calculate the new size in realloc()
When memory profiling is enabled, realloc() can occasionally get the area
size wrong due to the wrong pointer being used to check the new size. When
the old area gets unmapped in the operation, this may even result in a
crash. There's no impact without memory profiling though.

No backport is needed as this is exclusively 2.4-dev.
2021-05-07 08:01:35 +02:00
Willy Tarreau
0b7c78aa05 MINOR: config: add predicates "version_atleast" and "version_before" to cond blocks
These predicates respectively verify that the current version is at least
a given version or is before a specific one. The syntax is exactly the one
reported by "haproxy -v", though each component is optional, so both "1.5"
and "2.4-dev18-88910-48" are supported. Missing components equal zero, and
"dev" is below "pre" or "rc", which are both inferior to no such mention
(i.e. they are negative). Thus "2.4-dev18" is older than "2.4-rc1" which
is older than "2.4".
2021-05-06 17:04:45 +02:00
Willy Tarreau
58ca706e16 MINOR: config: add predicate "feature" to detect certain built-in features
The "feature(name)" predicate will return true if <name> corresponds to
a name listed after a '+' in the features list, that is it was enabled at
build time with USE_<name>=1. Typical use cases will include OPENSSL, LUA
and LINUX_SPLICE. But maybe it will also be convenient to use with optional
addons such as PROMEX and the device detection modules to help keeping the
same configs across various deployments.
2021-05-06 17:02:36 +02:00
Willy Tarreau
6492e87b0e MINOR: config: add predicates "streq()" and "strneq()" to conditional expressions
"streq(str1,str2)" will return true if the two strings match while
"strneq(str1,str2)" will return true only if they differ. This is
convenient to match an environment variable against a predefined value.
2021-05-06 17:02:36 +02:00
Willy Tarreau
42ed14b529 MINOR: config: add predicate "defined()" to conditional expression blocks
"defined(name)" will return true if <name> is a defined environment variable
otherwise false, regardless of its contents.
2021-05-06 17:02:36 +02:00
Willy Tarreau
732525fae7 MINOR: config: make cfg_eval_condition() support predicates with arguments
Now we can look up a list of known predicates and pre-parse their
arguments. For now the list is empty. The code needed to be arranged with
a common exit point to release all arguments because there's no default
argument freeing function (it likely only used to exist in the deinit
code). Since we only support simple arguments for now it's no big deal,
only a 2-liner loop.
2021-05-06 17:02:36 +02:00
Willy Tarreau
299bd1c3ae MINOR: config: improve .if condition error reporting
Let's return the position of the first unparsable character on error,
so that instead of just saying "unparsable conditional expression blah"
we can have:

  [ALERT] 125/150618 (13995) : parsing [test-conds2.cfg:1]: unparsable conditional expression '12/blah' in '.if' at position 1:
    .if 12/blah
        ^
This is important because conditions will be made from environment
variables or later from more complex expressions where the error will
not always be easy to locate.
2021-05-06 17:02:36 +02:00
Willy Tarreau
a43dfda4e1 MINOR: global: add version comparison functions
The new function split_version() converts a parsable haproxy version to
an array of integers. The function compare_current_version() compares an
arbitrary version to the current one. These two functions were written
by Thierry Fournier in 2013, and are still usable as-is. They will be
used to write config language predicates.
2021-05-06 17:02:36 +02:00
Willy Tarreau
f0d3b732fb MINOR: global: export the build features string list
Till now it was only presented in the version output but could not be
consulted outside of haproxy.c, let's export it as a variable, and set
it to an empty string if not defined.
2021-05-06 17:02:36 +02:00
Willy Tarreau
3e293a9135 MINOR: arg: improve the error message on missing closing parenthesis
When the closing brace is missing after an argument (acl, ...), the
error may report something like "expected ')' before ''". Let's just
drop "before ''" when the final word is empty to make the message a
bit clearer.
2021-05-06 17:02:36 +02:00
Willy Tarreau
7541056aa0 BUILD: activity: do not include malloc.h
It doesn't exist on MacOS and broke the build. We don't need it as it's
already included by compat.h when relevant. No backport is needed.
2021-05-06 11:38:41 +02:00
Willy Tarreau
a46f1af2b1 MINOR: config: support some pseudo-variables for file/line/section
The new pseudo-variables ".FILE", ".LINE" and ".SECTION" will be resolved
on the fly by the config parser and will respectively retrieve the current
configuration file name, the current line number and the current section
being parsed. This may help emit logs, errors, and debugging information
(e.g. which rule matched).

The '.' in the first char was reserved for such pseudo-variables and no
other variable is permitted. This will allow to add support for new ones
in the future if they prove to be useful (e.g. randoms/uuid for secret
keying or automatic naming of configuration objects).
2021-05-06 10:36:38 +02:00