1999 Commits

Author SHA1 Message Date
Willy Tarreau
8a022d5049 MINOR: config: add a new "default-path" global directive
By default haproxy loads all files designated by a relative path from the
location the process is started in. In some circumstances it might be
desirable to force all relative paths to start from a different location
just as if the process was started from such locations. This is what this
directive is made for. Technically it will perform a temporary chdir() to
the designated location while processing each configuration file, and will
return to the original directory after processing each file. It takes an
argument indicating the policy to use when loading files whose path does
not start with a slash ('/').

A few options are offered, "current" (the default), "config" (files
relative to config file's dir), "parent" (files relative to config file's
parent dir), and "origin" with an absolute path.

This should address issue #1198.
2021-04-28 11:30:13 +02:00
Alex
59c5335dbe DOC: general: fix example in set-timeout
The alternative arguments are always in curly brackets, let's fix it for
set-timeout.
The Example in set-timeout does not have the one of the required argument.

This commit makes the PR https://github.com/cbonte/haproxy-dconv/pull/34
obsolete.
2021-04-28 11:08:35 +02:00
Alex
bf1bd5a2a8 DOC: general: fix white spaces for HTML converter
The HTML converter expects some formats to recognize if a keyword is a
keyword.
2021-04-26 10:43:51 +02:00
Ilya Shipitsin
b2be9a1ea9 CLEANUP: assorted typo fixes in the code and comments
This is 22nd iteration of typo fixes
2021-04-26 10:42:58 +02:00
Tim Duesterhus
2e4a18e04a MINOR: uri_normalizer: Add a percent-decode-unreserved normalizer
This normalizer decodes percent encoded characters within the RFC 3986
unreserved set.

See GitHub Issue #714.
2021-04-23 19:43:45 +02:00
Tim Duesterhus
d6d33deaea DOC: Add RFC references for the path-strip-dot(dot)? normalizers
This is RFC 3986#6.2.2.3.
2021-04-23 19:43:45 +02:00
Tim Duesterhus
c315efda60 DOC: Fix RFC reference for the percent-to-uppercase normalizer
The section is 6.2.2.1, not 6.2.21 (missing dot).
2021-04-23 19:43:45 +02:00
Tim Duesterhus
7a95f41fd6 DOC: Fix indentation for path-strip-dot normalizer
The long explanation should be indented two additional spaces.
2021-04-23 19:43:45 +02:00
Willy Tarreau
bfd19d68f0 [RELEASE] Released version 2.4-dev17
Released version 2.4-dev17 with the following main changes :
    - MINOIR: mux-pt/trace: Register a new trace source with its events
    - BUG/MINOR: mux-pt: Fix a possible UAF because of traces in mux_pt_io_cb
    - CI: travis: Drastically clean up .travis.yml
    - CLEANUP: pattern: make all pattern tables read-only
    - MINOR: trace: replace the trace() inline function with an equivalent macro
    - MINOR: initcall: uniformize the section names between MacOS and other unixes
    - CLEANUP: initcall: rename HA_SECTION to HA_INIT_SECTION
    - MINOR: compiler: add macros to declare section names
    - CLEANUP: initcall: rely on HA_SECTION_* instead of defining its own
    - MINOR: global: declare a read_mostly section
    - MINOR: fd: move a few read-mostly variables to their own section
    - MINOR: epoll: move epoll_fd to read_mostly
    - MINOR: kqueue: move kqueue_fd to read_mostly
    - MINOR: pool: move pool declarations to read_mostly
    - MINOR: threads: mark all_threads_mask as read_mostly
    - MINOR: server: move idle_conn_task to read_mostly
    - MINOR: protocol: move __protocol_by_family to read_mostly
    - MINOR: pattern: make the pat_lru_seed read_mostly
    - MINOR: trace: make trace sources read_mostly
    - MINOR: freq_ctr: add a generic function to report the total value
    - MEDIUM: freq_ctr: make read_freq_ctr_period() use freq_ctr_total()
    - MEDIUM: freq_ctr: reimplement freq_ctr_remain_period() from freq_ctr_total()
    - MINOR: freq_ctr: add the missing next_event_delay_period()
    - MINOR: freq_ctr: unify freq_ctr and freq_ctr_period into freq_ctr
    - MEDIUM: freq_ctr: replace the per-second counters with the generic ones
    - MINOR: freq_ctr: add cpu_relax in the rotation loop of update_freq_ctr_period()
    - MINOR: freq_ctr: simplify and improve the update function
    - CLEANUP: time: remove the now unused ms_left_scaled
    - MINOR: time: move the time initialization out of tv_update_date()
    - MINOR: time: remove useless variable copies in tv_update_date()
    - MINOR: time: change the global timeval and the the global tick at once
    - MEDIUM: time: make the clock offset global and no per-thread
    - MINOR: atomic: reimplement the relaxed version of x86 BTS/BTR
    - MINOR: trace: Add the checks as a possible trace source
    - MINOIR: checks/trace: Register a new trace source with its events
    - MINOR: hlua: Add function to release a lua function
    - BUG/MINOR: hlua: Fix memory leaks on error path when registering a task
    - BUG/MINOR: hlua: Fix memory leaks on error path when registering a converter
    - BUG/MINOR: hlua: Fix memory leaks on error path when registering a fetch
    - BUG/MINOR: hlua: Fix memory leaks on error path when parsing a lua action
    - BUG/MINOR: hlua: Fix memory leaks on error path when registering an action
    - BUG/MINOR: hlua: Fix memory leaks on error path when registering a service
    - BUG/MINOR: hlua: Fix memory leaks on error path when registering a cli keyword
    - BUG/MINOR: cfgparse/proxy: Fix some leaks during proxy section parsing
    - BUG/MINOR: listener: Handle allocation error when allocating a new bind_conf
    - BUG/MINOR: cfgparse/proxy: Hande allocation errors during proxy section parsing
    - MINOR: cfgparse/proxy: Group alloc error handling during proxy section parsing
    - DOC: internals: update the SSL architecture schema
    - BUG/MEDIUM: sample: Fix adjusting size in field converter
    - MINOR: sample: add ub64dec and ub64enc converters
    - CLEANUP: sample: align samples list in sample.c
    - MINOR: ist: Add `istclear(struct ist*)`
    - CI: cirrus: install "pcre" package
    - MINOR: opentracing: correct calculation of the number of arguments in the args[]
    - MINOR: opentracing: transfer of context names without prefix
    - MINOR: sample: converter: Add mjson library.
    - MINOR: sample: converter: Add json_query converter
    - CI: travis-ci: enable weekly graviton2 builds
    - DOC: ssl: Certificate hot update only works on fronted certificates
    - DOC: ssl: Certificate hot update works on server certificates
    - BUG/MEDIUM: threads: Ignore current thread to end its harmless period
    - MINOR: threads: Only consider running threads to end a thread harmeless period
    - BUG/MINOR: checks: Set missing id to the dummy checks frontend
    - MINOR: logs: Add support of checks as session origin to format lf strings
    - BUG/MINOR: connection: Fix fc_http_major and bc_http_major for TCP connections
    - MINOR: connection: Make bc_http_major compatible with tcp-checks
    - BUG/MINOR: ssl-samples: Fix ssl_bc_* samples when called from a health-check
    - BUG/MINOR: http-fetch: Make method smp safe if headers were already forwarded
    - MINOR: tcp_samples: Add samples to get src/dst info of the backend connection
    - MINOR: tcp_samples: Be able to call bc_src/bc_dst from the health-checks
    - BUG/MINOR: http_htx: Remove BUG_ON() from http_get_stline() function
    - BUG/MINOR: logs: Report the true number of retries if there was no connection
    - BUILD: makefile: Redirect stderr to /dev/null when probing options
    - MINOR: uri_normalizer: Add uri_normalizer module
    - MINOR: uri_normalizer: Add `enum uri_normalizer_err`
    - MINOR: uri_normalizer: Add `http-request normalize-uri`
    - MINOR: uri_normalizer: Add a `merge-slashes` normalizer to http-request normalize-uri
    - MINOR: uri_normalizer: Add a `dotdot` normalizer to http-request normalize-uri
    - MINOR: uri_normalizer: Add support for supressing leading `../` for dotdot normalizer
    - MINOR: uri_normalizer: Add a `sort-query` normalizer
    - MINOR: uri_normalizer: Add a `percent-upper` normalizer
    - MEDIUM: http_act: Rename uri-normalizers
    - DOC: Add introduction to http-request normalize-uri
    - DOC: Note that URI normalization is experimental
    - BUG/MINOR: pools: maintain consistent ->allocated count on alloc failures
    - BUG/MINOR: pools/buffers: make sure to always reserve the required buffers
    - MINOR: pools: drop the unused static history of artificially failed allocs
    - CLEANUP: pools: remove unused arguments to pool_evict_from_cache()
    - MEDIUM: pools: move the cache into the pool header
    - MINOR: pool: remove the size field from pool_cache_head
    - MINOR: pools: rename CONFIG_HAP_LOCAL_POOLS to CONFIG_HAP_POOLS
    - MINOR: pools: enable the fault injector in all allocation modes
    - MINOR: pools: make the basic pool_refill_alloc()/pool_free() update needed_avg
    - MEDIUM: pools: unify pool_refill_alloc() across all models
    - CLEANUP: pools: re-merge pool_refill_alloc() and __pool_refill_alloc()
    - MINOR: pools: call pool_alloc_nocache() out of the pool's lock
    - CLEANUP: pools: move the lock to the only __pool_get_first() that needs it
    - CLEANUP: pools: rename __pool_get_first() to pool_get_from_shared_cache()
    - CLEANUP: pools: rename pool_*_{from,to}_cache() to *_local_cache()
    - CLEANUP: pools: rename __pool_free() to pool_put_to_shared_cache()
    - MINOR: tools: add statistical_prng_range() to get a random number over a range
    - MINOR: pools: use cheaper randoms for fault injections
    - MINOR: pools: move the fault injector to __pool_alloc()
    - MINOR: pools: split the OS-based allocator in two
    - MINOR: pools: always use atomic ops to maintain counters
    - MINOR: pools: move pool_free_area() out of the lock in the locked version
    - MINOR: pools: factor the release code into pool_put_to_os()
    - MEDIUM: pools: make CONFIG_HAP_POOLS control both local and shared pools
    - MINOR: pools: create unified pool_{get_from,put_to}_cache()
    - MINOR: pools: evict excess objects using pool_evict_from_local_cache()
    - MEDIUM: pools: make pool_put_to_cache() always call pool_put_to_local_cache()
    - CLEANUP: pools: make the local cache allocator fall back to the shared cache
    - CLEANUP: pools: merge pool_{get_from,put_to}_local_caches with generic ones
    - CLEANUP: pools: uninline pool_put_to_cache()
    - CLEANUP: pools: declare dummy pool functions to remove some ifdefs
    - BUILD: pools: fix build with DEBUG_FAIL_ALLOC
    - BUG/MINOR: server: make srv_alloc_lb() allocate lb_nodes for consistent hash
    - CONTRIB: mod_defender: import the minimal number of includes
    - CONTRIB: mod_defender: make the code build with the embedded includes
    - CONTRIB: modsecurity: import the minimal number of includes
    - CONTRIB: modsecurity: make the code build with the embedded includes
    - CLEANUP: sample: Improve local variables in sample_conv_json_query
    - CLEANUP: sample: Explicitly handle all possible enum values from mjson
    - CLEANUP: sample: Use explicit return for successful `json_query`s
    - CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion
    - CONTRIB: move spoa_example out of the tree
    - BUG/MINOR: server: free srv.lb_nodes in free_server
    - BUG/MINOR: logs: free logsrv.conf.file on exit
    - BUG/MEDIUM: server: ensure thread-safety of server runtime creation
    - MINOR: server: add log on dynamic server creation
    - MINOR: server: implement delete server cli command
    - CONTRIB: move spoa_server out of the tree
    - CONTRIB: move modsecurity out of the tree
    - BUG/MINOR: server: fix potential null gcc error in delete server
    - BUG/MAJOR: mux-h2: Properly detect too large frames when decoding headers
    - BUG/MEDIUM: mux-h2: Fix dfl calculation when merging CONTINUATION frames
    - BUG/MINOR: uri_normalizer: Use delim parameter when building the sorted query in uri_normalizer_query_sort
    - CLEANUP: uri_normalizer: Remove trailing whitespace
    - MINOR: uri_normalizer: Add a `strip-dot` normalizer
    - CONTRIB: move mod_defender out of the tree
    - CLEANUP: contrib: remove the last references to the now dead contrib/ directory
    - BUG/MEDIUM: config: fix cpu-map notation with both process and threads
    - MINOR: config: add a diag for invalid cpu-map statement
    - BUG/MINOR: mworker/init: don't reset nb_oldpids in non-mworker cases
    - BUG/MINOR: mworker: don't use oldpids[] anymore for reload
    - BUILD: makefile: fix the "make clean" target on strict bourne shells
    - IMPORT: slz: import slz into the tree
    - BUILD: compression: switch SLZ from out-of-tree to in-tree
    - CI: github: do not build libslz any more
    - CLEANUP: compression: remove calls to SLZ init functions
    - BUG/MEDIUM: mux-h2: Properly handle shutdowns when received with data
    - MINOR: cpuset: define a platform-independent cpuset type
    - MINOR: cfgparse: use hap_cpuset for parse_cpu_set
    - MEDIUM: config: use platform independent type hap_cpuset for cpu-map
    - MINOR: thread: implement the detection of forced cpu affinity
    - MINOR: cfgparse: support the comma separator on parse_cpu_set
    - MEDIUM: cfgparse: detect numa and set affinity if needed
    - MINOR: global: add option to disable numa detection
    - BUG/MINOR: haproxy: fix compilation on macOS
    - BUG/MINOR: cpuset: fix compilation on platform without cpu affinity
    - MINOR: time: avoid unneeded updates to now_offset
    - MINOR: time: avoid overwriting the same values of global_now
    - CLEANUP: time: use __tv_to_ms() in tv_update_date() instead of open-coding
    - MINOR: time: avoid u64 needlessly expensive computations for the 32-bit now_ms
    - BUG/MINOR: peers: remove useless table check if initial resync is finished
    - BUG/MEDIUM: peers: re-work connection to new process during reload.
    - BUG/MEDIUM: peers: re-work refcnt on table to protect against flush
    - BUG/MEDIUM: config: fix missing initialization in numa_detect_topology()
2021-04-23 19:11:10 +02:00
Amaury Denoyelle
0f50cb9c73 MINOR: global: add option to disable numa detection
Render numa detection optional with a global configuration statement
'no numa-cpu-mapping'. This can be used if the applied affinity of the
algorithm is not optimal. Also complete the documentation with this new
keyword.
2021-04-23 16:06:49 +02:00
Amaury Denoyelle
982fb53390 MEDIUM: config: use platform independent type hap_cpuset for cpu-map
Use the platform independent type hap_cpuset for the cpu-map statement
parsing. This allow to address CPU index greater than LONGBITS.

Update the documentation to reflect the removal of this limit except for
platforms without cpu_set_t type or equivalent.
2021-04-23 16:06:49 +02:00
Maximilian Mader
ff3bb8b609 MINOR: uri_normalizer: Add a strip-dot normalizer
This normalizer removes "/./" segments from the path component.
Usually the dot refers to the current directory which renders those segments redundant.

See GitHub Issue #714.
2021-04-21 12:15:14 +02:00
Amaury Denoyelle
e558043e13 MINOR: server: implement delete server cli command
Implement a new CLI command 'del server'. It can be used to removed a
dynamically added server. Only servers in maintenance mode can be
removed, and without pending/active/idle connection on it.

Add a new reg-test for this feature. The scenario of the reg-test need
to first add a dynamic server. It is then deleted and a client is used
to ensure that the server is non joinable.

The management doc is updated with the new command 'del server'.
2021-04-21 11:00:31 +02:00
Willy Tarreau
8695199aa8 CONTRIB: move spoa_example out of the tree
As previously mentioned SPOA code has nothing to do in the haproxy core
since they're not dependent on haproxy's version. This one was moved to
its own repository here with complete history:

     https://github.com/haproxy/spoa-example
2021-04-21 09:39:06 +02:00
Willy Tarreau
2b71810cb3 CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion
The current "ADD" vs "ADDQ" is confusing because when thinking in terms
of appending at the end of a list, "ADD" naturally comes to mind, but
here it does the opposite, it inserts. Several times already it's been
incorrectly used where ADDQ was expected, the latest of which was a
fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain
why we queue the stream at the head of the server list").

Let's use more explicit (but slightly longer) names now:

   LIST_ADD        ->       LIST_INSERT
   LIST_ADDQ       ->       LIST_APPEND
   LIST_ADDED      ->       LIST_INLIST
   LIST_DEL        ->       LIST_DELETE

The same is true for MT_LISTs, including their "TRY" variant.
LIST_DEL_INIT keeps its short name to encourage to use it instead of the
lazier LIST_DELETE which is often less safe.

The change is large (~674 non-comment entries) but is mechanical enough
to remain safe. No permutation was performed, so any out-of-tree code
can easily map older names to new ones.

The list doc was updated.
2021-04-21 09:20:17 +02:00
Tim Duesterhus
2963fd34f8 DOC: Note that URI normalization is experimental
Add a paragraph to the URI normalization documentation that URI normalization
is currently considered to be experimental.
2021-04-19 09:05:57 +02:00
Tim Duesterhus
b918a4a564 DOC: Add introduction to http-request normalize-uri
This patch adds an introduction to the http-request normalize-uri section,
explaining what to expect from the normalizers and possible issues that might
arise when not being careful.
2021-04-19 09:05:57 +02:00
Tim Duesterhus
5be6ab269e MEDIUM: http_act: Rename uri-normalizers
This patch renames all existing uri-normalizers into a more consistent naming
scheme:

1. The part of the URI that is being touched.
2. The modification being performed as an explicit verb.
2021-04-19 09:05:57 +02:00
Tim Duesterhus
a407193376 MINOR: uri_normalizer: Add a percent-upper normalizer
This normalizer uppercases the hexadecimal characters used in percent-encoding.

See GitHub Issue #714.
2021-04-19 09:05:57 +02:00
Tim Duesterhus
d7b89be30a MINOR: uri_normalizer: Add a sort-query normalizer
This normalizer sorts the `&` delimited query parameters by parameter name.

See GitHub Issue #714.
2021-04-19 09:05:57 +02:00
Tim Duesterhus
560e1a6352 MINOR: uri_normalizer: Add support for supressing leading ../ for dotdot normalizer
This adds an option to supress `../` at the start of the resulting path.
2021-04-19 09:05:57 +02:00
Tim Duesterhus
9982fc2bbd MINOR: uri_normalizer: Add a dotdot normalizer to http-request normalize-uri
This normalizer merges `../` path segments with the predecing segment, removing
both the preceding segment and the `../`.

Empty segments do not receive special treatment. The `merge-slashes` normalizer
should be executed first.

See GitHub Issue #714.
2021-04-19 09:05:57 +02:00
Tim Duesterhus
d371e99d1c MINOR: uri_normalizer: Add a merge-slashes normalizer to http-request normalize-uri
This normalizer merges adjacent slashes into a single slash, thus removing
empty path segments.

See GitHub Issue #714.
2021-04-19 09:05:57 +02:00
Christopher Faulet
7d081f02a4 MINOR: tcp_samples: Add samples to get src/dst info of the backend connection
This patch adds 4 new sample fetches to get the source and the destination
info (ip address and port) of the backend connection :

 * bc_dst      : Returns the destination address of the backend connection
 * bc_dst_port : Returns the destination port of the backend connection
 * bc_src      : Returns the source address of the backend connection
 * bc_src_port : Returns the source port of the backend connection

The configuration manual was updated accordingly.
2021-04-19 08:31:05 +02:00
Remi Tricot-Le Breton
b5f0fac765 DOC: ssl: Certificate hot update works on server certificates
The CLI's "set ssl cert" command also works on backend certificates
(see GitHub issue #427).

It does not need to be backported.
2021-04-15 17:52:08 +02:00
Remi Tricot-Le Breton
3445909a63 DOC: ssl: Certificate hot update only works on fronted certificates
The CLI's "set ssl cert" command only works on frontend certificates but
the documentation did not specify this limitations yet.

This patch can be backported to all stable branches.
2021-04-15 17:52:08 +02:00
Alex
51c8ad45ce MINOR: sample: converter: Add json_query converter
With the json_query can a JSON value be extacted from a header
or body of the request and saved to a variable.

This converter makes it possible to handle some JSON workload
to route requests to different backends.
2021-04-15 17:07:03 +02:00
Moemen MHEDHBI
92f7d43c5d MINOR: sample: add ub64dec and ub64enc converters
ub64dec and ub64enc are the base64url equivalent of b64dec and base64
converters. base64url encoding is the "URL and Filename Safe Alphabet"
variant of base64 encoding. It is also used in in JWT (JSON Web Token)
standard.
RFC1421 mention in base64.c file is deprecated so it was replaced with
RFC4648 to which existing converters, base64/b64dec, still apply.

Example:
  HAProxy:
    http-request return content-type text/plain lf-string %[req.hdr(Authorization),word(2,.),ub64dec]
  Client:
    Token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiZm9vIiwia2V5IjoiY2hhZTZBaFhhaTZlIn0.5VsVj7mdxVvo1wP5c0dVHnr-S_khnIdFkThqvwukmdg
    $ curl -H "Authorization: Bearer ${TOKEN}" http://haproxy.local
    {"user":"foo","key":"chae6AhXai6e"}
2021-04-13 17:28:13 +02:00
Remi Tricot-Le Breton
59846b6773 DOC: internals: update the SSL architecture schema
This commit adds the new fields added to the ckch_inst structure in
order to manage the backend certificate hot update (GitHub #427) and the
bug of the default certificate update (GitHub #1143).
2021-04-13 11:34:44 +02:00
Willy Tarreau
86512dd152 [RELEASE] Released version 2.4-dev16
Released version 2.4-dev16 with the following main changes :
    - CLEANUP: dev/flags: remove useless test in the stdin number parser
    - MINOR: No longer rely on deprecated sample fetches for predefined ACLs
    - MINOR: acl: Add HTTP_2.0 predefined macro
    - BUG/MINOR: hlua: Detect end of request when reading data for an HTTP applet
    - BUG/MINOR: tools: fix parsing "us" unit for timers
    - MINOR: server/bind: add support of new prefixes for addresses.
    - MINOR: log: register config file and line number on log servers.
    - MEDIUM: log: support tcp or stream addresses on log lines.
    - BUG/MEDIUM: log: fix config parse error logging on stdout/stderr or any raw fd
    - CLEANUP: fd: remove FD_POLL_DATA and FD_POLL_STICKY
    - MEDIUM: fd: prepare FD_POLL_* to move to bits 8-15
    - MEDIUM: fd: merge fdtab[].ev and state for FD_EV_* and FD_POLL_* into state
    - MINOR: fd: move .linger_risk into fdtab[].state
    - MINOR: fd: move .cloned into fdtab[].state
    - MINOR: fd: move .initialized into fdtab[].state
    - MINOR: fd: move .et_possible into fdtab[].state
    - MINOR: fd: move .exported into fdtab[].state
    - MINOR: fd: implement an exclusive syscall bit to remove the ugly "log" lock
    - MINOR: cli/show-fd: slightly reorganize the FD status flags
    - MINOR: atomic/arm64: detect and use builtins for the double-word CAS
    - CLEANUP: atomic: add an explicit _FETCH variant for add/sub/and/or
    - CLEANUP: atomic: make all standard add/or/and/sub operations return void
    - CLEANUP: atomic: add a fetch-and-xxx variant for common operations
    - CLEANUP: atomic: add HA_ATOMIC_INC/DEC for unit increments
    - CLEANUP: atomic/tree-wide: replace single increments/decrements with inc/dec
    - CLEANUP: atomic: use the __atomic variant of BTS/BTR on modern compilers
    - MINOR: atomic: implement native BTS/BTR for x86
    - MINOR: ist: Add `istappend(struct ist, char)`
    - MINOR: ist: Add `istshift(struct ist*)`
    - MINOR: ist: Add `istsplit(struct ist*, char)`
    - BUG/MAJOR: fd: switch temp values to uint in fd_stop_both()
    - MINOR: opentracing: register config file and line number on log servers
    - MEDIUM: resolvers: add support of tcp address on nameserver line.
    - MINOR: ist: Rename istappend() to __istappend()
    - CLEANUP: htx: Make http_get_stline take a `const struct`
    - CLEANUP: ist: Remove unused `count` argument from `ist2str*`
    - CLEANUP: Remove useless malloc() casts
2021-04-09 17:10:39 +02:00
Emeric Brun
c8f3e45c6a MEDIUM: resolvers: add support of tcp address on nameserver line.
This patch re-works configuration parsing, it removes the "server"
lines from "resolvers" sections introduced in commit 56fc5d9eb:
MEDIUM: resolvers: add supports of TCP nameservers in resolvers.

It also extends the nameserver lines to support stream server
addresses such as:

resolvers
  nameserver localhost tcp@127.0.0.1:53

Doing so, a part of nameserver's init code was factorized in
function 'parse_resolvers' and removed from 'post_parse_resolvers'.
2021-04-08 14:20:40 +02:00
Emeric Brun
94aab06e24 MEDIUM: log: support tcp or stream addresses on log lines.
An explicit stream address prefix such as "tcp6@" "tcp4@"
"stream+ipv6@" "stream+ipv4@" or "stream+unix@" will
allocate an implicit ring buffer with a forward server
targeting the given address.

This is usefull to simply send logs to a log server in tcp
and It doesn't need to declare a ring section in configuration.
2021-04-07 09:18:34 +02:00
Emeric Brun
ce325c4360 MINOR: server/bind: add support of new prefixes for addresses.
Since the internal function str2sa_range is used to addresses
for different objects ('server', 'bind' but also 'log' or
'nameserver') we notice that some combinations are missing.

"ip@" is introduced to authorize the prefix "dgram+ip@" or
"stream+ip@" which dectects automatically IP version but
specify dgram or stream.

"tcp@" was introduced and is an alias for "stream+ip@".
"tcp6" and "tcp4" are now aliases for "stream+ipv6@" and
"stream+ipv4@".

"uxst@" and "uxdg@" are now aliases for "stream+unix@" and
"dgram+unix@".

This patch also adds a complete section in documentation to
describe adresses and their prefixes.
2021-04-07 09:18:32 +02:00
Christopher Faulet
8043e831d1 MINOR: acl: Add HTTP_2.0 predefined macro
HTTP_2.0 predefined macro returns true for HTTP/2 requests. HTTP/2 doen't
convey a version information, so this macro may seem a bit strange. But for
compatiblity reasons, internally, the "HTTP/2.0" version is set. Thus, it is
handy to rely on it to differenciate HTTP/1 and HTTP/2 requests.
2021-04-06 07:31:51 +02:00
Christopher Faulet
779184e35e MINOR: No longer rely on deprecated sample fetches for predefined ACLs
Some predefined ACLs were still based on deprecated sample fetches, like
req_proto_http or req_ver. Now, they use non-deprecated sample fetches. In
addition, the usage lines in the configuration manual have been updated to
be more explicit.
2021-04-05 17:21:05 +02:00
Willy Tarreau
59fa1d1414 [RELEASE] Released version 2.4-dev15
Released version 2.4-dev15 with the following main changes :
    - BUG/MINOR: payload: Wait for more data if buffer is empty in payload/payload_lv
    - BUG/MINOR: stats: Apply proper styles in HTML status page.
    - BUG/MEDIUM: time: make sure to always initialize the global tick
    - BUG/MINOR: tcp: fix silent-drop workaround for IPv6
    - BUILD: tcp: use IPPROTO_IPV6 instead of SOL_IPV6 on FreeBSD/MacOS
    - CLEANUP: socket: replace SOL_IP/IPV6/TCP with IPPROTO_IP/IPV6/TCP
    - BUG/MINOR: http_fetch: make hdr_ip() resistant to empty fields
    - BUG/MINOR: mux-h2: Don't emit log twice if an error occurred on the preface
    - MINOR: stream: Don't trigger errors on destructive HTTP upgrades
    - MINOR: frontend: Create HTTP txn for HTX streams
    - MINOR: stream: Be sure to set HTTP analysers when creating an HTX stream
    - BUG/MINOR: stream: Properly handle TCP>H1>H2 upgrades in http_wait_for_request
    - BUG/MINOR: config: Add warning for http-after-response rules in TCP mode
    - MINOR: muxes: Add a flag to notify a mux does not support any upgrade
    - MINOR: mux-h1: Don't perform implicit HTTP/2 upgrade if not supported by mux
    - MINOR: mux-pt: Don't perform implicit HTTP upgrade if not supported by mux
    - MEDIUM: mux-h1: Expose h1 in the list of supported mux protocols
    - MEDIUM: mux-pt: Expose passthrough in the list of supported mux protocols
    - MINOR: muxes: Show muxes flags when the mux list is displayed
    - DOC: config: Improve documentation about proto/check-proto keywords
    - MINOR: stream: Use stream type instead of proxy mode when appropriate
    - MINOR: filters/http-ana: Decide to filter HTTP headers in HTTP analysers
    - MINOR: http-ana: Simplify creation/destruction of HTTP transactions
    - MINOR: stream: Handle stream HTTP upgrade in a dedicated function
    - MEDIUM: Add tcp-request switch-mode action to perform HTTP upgrade
    - MINOR: config/proxy: Don't warn for HTTP rules in TCP if 'switch-mode http' set
    - MINOR: config/proxy: Warn if a TCP proxy without backend is upgradable to HTTP
    - DOC: config: Add documentation about TCP to HTTP upgrades
    - REGTESTS: Add script to tests TCP to HTTP upgrades
    - BUG/MINOR: payload/htx: Ingore L6 sample fetches for HTX streams/checks
    - MINOR: htx: Make internal.strm.is_htx an internal sample fetch
    - MINOR: action: Use a generic function to check validity of an action rule list
    - MINOR: payload/config: Warn if a L6 sample fetch is used from an HTTP proxy
    - MEDIUM: http-rules: Add wait-for-body action on request and response side
    - REGTESTS: Add script to tests the wait-for-body HTTP action
    - BUG/MINOR: http-fetch: Fix test on message state to capture the version
    - CLEANUP: vars: always pre-initialize smp in vars_parse_cli_get_var()
    - MINOR: global: define diagnostic mode of execution
    - MINOR: cfgparse: diag for multiple nbthread statements
    - MINOR: server: diag for 0 weight server
    - MINOR: diag: create cfgdiag module
    - MINOR: diag: diag if servers use the same cookie value
    - MINOR: config: diag if global section after non-global
    - TESTS: slightly reorganize the code in the tests/ directory
    - TESTS: move tests/*.cfg to tests/config
    - REGTESTS: ssl: "set ssl cert" and multi-certificates bundle
    - REGTESTS: ssl: mark set_ssl_cert_bundle.vtc as broken
    - CONTRIB: halog: fix issue with array of type char
    - CONTRIB: tcploop: add a shutr command
    - CONTRIB: debug: add the show-fd-to-flags script
    - CONTRIB: debug: split poll from flags
    - CONTRIB: move some dev-specific tools to dev/
    - BUILD: makefile: always build the flags utility
    - DEV: flags: replace the unneeded makefile with a README
    - BUILD: makefile: integrate the hpack tools
    - CONTRIB: merge ip6range with iprange
    - CONTRIB: move some admin-related sub-projects to admin/
    - CONTRIB: move halog to admin/
    - ADMIN: halog: automatically enable USE_MEMCHR on the right glibc version
    - BUILD: makefile: build halog with the correct flags
    - BUILD: makefile: add a "USE_PROMEX" variable to ease building prometheus-exporter
    - CONTRIB: move prometheus-exporter to addons/promex
    - DOC: add a few words about USE_* and the addons directory
    - CONTRIB: move 51Degrees to addons/51degrees
    - CONTRIB: move src/da.c and contrib/deviceatlas to addons/deviceatlas
    - CONTRIB: move src/wurfl.c and contrib/wurfl to addons/wurfl
    - CONTRIB: move contrib/opentracing to addons/ot
    - BUG/MINOR: opentracing: initialization after establishing daemon mode
    - DOC: clarify that compression works for HTTP/2
2021-04-02 19:16:32 +02:00
Julien Pivotto
ff80c82877 DOC: clarify that compression works for HTTP/2
This patch clarifies that compression also works with HTTP/2. I have
picked the wording "HTTP/1.1 or above" because it is already used
elsewhere in the documentation.

I have tested that compression indeed works in HTTP/2.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-04-02 19:15:25 +02:00
Willy Tarreau
a63d1a0863 CONTRIB: move contrib/opentracing to addons/ot
This one is the last optional module to build with haproxy, so let's move
it to addons/. It was renamed to "ot" as it was the only one whose USE_*
option did not match the directory name, now this is consistent.

Few changes were required, only the Makefile, and doc were adjusted, as
the directory was already self-contained and relocatable.
2021-04-02 17:48:42 +02:00
Willy Tarreau
57610c694e CONTRIB: move src/wurfl.c and contrib/wurfl to addons/wurfl
Both the source file and the dummy library are now at the same place.
Maybe the build howto could be moved there as well to make things even
cleaner.

The Makefile, MAINTAINERS, doc, and vtest matrix were updated.
2021-04-02 17:48:42 +02:00
Willy Tarreau
f8d9ec57f0 CONTRIB: move src/da.c and contrib/deviceatlas to addons/deviceatlas
Both the source file and the dummy library are now at the same place.
Maybe the build howto could be moved there as well to make things even
cleaner.

The Makefile, MAINTAINERS, doc, github build matrix, coverity checks
and travis CI's build were updated.
2021-04-02 17:48:42 +02:00
Willy Tarreau
977209d1d8 CONTRIB: move 51Degrees to addons/51degrees
Now it's much cleaner, both 51d.c and the dummy library live together and
are easier to spot and maintain. The build howto probably ought to be moved
there as well. Makefile, docs and MAINTAINERS were updated, as well as
the github CI's build matrix, travis CI's, and coverity checks.
2021-04-02 17:48:42 +02:00
Amaury Denoyelle
7b01a8dbdd MINOR: global: define diagnostic mode of execution
Define MODE_DIAG which is used to run haproxy in diagnostic mode. This
mode is used to output extra warnings about possible configuration
blunder or sub-optimal usage. It can be activated with argument '-dD'.

A new output function ha_diag_warning is implemented reserved for
diagnostic output. It serves to standardize the format of diagnostic
messages.

A macro HA_DIAG_WARN_COND is also available to automatically check if
diagnostic mode is on before executing the diagnostic check.
2021-04-01 18:03:37 +02:00
Christopher Faulet
021a8e4d7b MEDIUM: http-rules: Add wait-for-body action on request and response side
Historically, an option was added to wait for the request payload (option
http-buffer-request). This option has 2 drawbacks. First, it is an ON/OFF
option for the whole proxy. It cannot be enabled on demand depending on the
message. Then, as its name suggests, it only works on the request side. The
only option to wait for the response payload was to write a dedicated
filter. While it is an acceptable solution for complex applications, it is a
bit overkill to simply match strings in the body.

To make everyone happy, this patch adds a dedicated HTTP action to wait for
the message payload, for the request or the response depending it is used in
an http-request or an http-response ruleset. The time to wait is
configurable and, optionally, the minimum payload size to have before stop
to wait.

Both the http action and the old http analyzer rely on the same internal
function.
2021-04-01 16:27:40 +02:00
Christopher Faulet
a434a00864 BUG/MINOR: payload/htx: Ingore L6 sample fetches for HTX streams/checks
Use a L6 sample fetch on an HTX streams or a HTX health-check is meaningless
because data are not raw but structured. So now, these sample fetches fail
when called from an HTTP proxy. In addition, a warning has been added in the
configuration manual, at the begining of the L6 sample fetches section.

Note that req.len and res.len samples return the HTX data size instead of
failing. It is not accurate because it does not reflect the buffer size nor
the raw data length. But we keep it for backward compatibility purpose.
However it remains a bit strange to use it on an HTTP proxy.

This patch may be backported to all versions supporting the HTX, i.e as far
as 2.0. But the part about the health-checks is only valid for the 2.2 and
upper.
2021-04-01 15:31:55 +02:00
Christopher Faulet
4d37e53dfc DOC: config: Add documentation about TCP to HTTP upgrades
This patch adds explanation about chaining a TCP frontend to an HTTP
backend. It also explain how the HTTP upgrades work in this context. A note
has also been added in "Fetching HTTP samples" section to warning about HTTP
content processing in TCP.
2021-04-01 13:24:40 +02:00
Christopher Faulet
ae863c62e3 MEDIUM: Add tcp-request switch-mode action to perform HTTP upgrade
It is now possible to perform HTTP upgrades on a TCP stream from the
frontend side. To do so, a tcp-request content rule must be defined with the
switch-mode action, specifying the mode (for now, only http is supported)
and optionnaly the proto (h1 or h2).

This way it could be possible to set HTTP directives on a TCP frontend which
will only be evaluated if an upgrade is performed. This new way to perform
HTTP upgrades should replace progressively the old way, consisting to route
the request to an HTTP backend. And it should be also a good start to remove
all HTTP processing from tcp-request content rules.

This action is terminal, it stops the ruleset evaluation. It is only
available on proxy with the frontend capability.

The configuration manual has been updated accordingly.
2021-04-01 13:17:19 +02:00
Christopher Faulet
982e17dcf5 DOC: config: Improve documentation about proto/check-proto keywords
This patch adds a description about information provided by "haproxy -vv"
command regarding the available protocols. The description is adapted
depending the context (bind line, server line or health-check).
2021-04-01 11:06:48 +02:00
Willy Tarreau
af6d88b359 [RELEASE] Released version 2.4-dev14
Released version 2.4-dev14 with the following main changes :
    - MEDIUM: quic: Fix build.
    - MEDIUM: quic: Fix build.
    - CI: codespell: whitelist "Dragan Dosen"
    - CLEANUP: assorted typo fixes in the code and comments
    - CI: github actions: update LibreSSL to 3.2.5
    - REGTESTS: revert workaround for a crash with recent libressl on http-reuse sni
    - CLEANUP: mark defproxy as const on parse tune.fail-alloc
    - REGTESTS: remove unneeded experimental-mode in cli add server test
    - REGTESTS: wait for proper return of enable server in cli add server test
    - MINOR: compression: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: spoe: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: fcgi-app: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: cache: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: ssl: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: opentracing: use pool_alloc(), not pool_alloc_dirty()
    - MINOR: dynbuf: make b_alloc() always check if the buffer is allocated
    - CLEANUP: compression: do not test for buffer before calling b_alloc()
    - CLEANUP: l7-retries: do not test the buffer before calling b_alloc()
    - MINOR: channel: simplify the channel's buffer allocation
    - MEDIUM: dynbuf: remove last usages of b_alloc_margin()
    - CLEANUP: dynbuf: remove b_alloc_margin()
    - CLEANUP: dynbuf: remove the unused b_alloc_fast() function
    - CLEANUP: pools: remove the unused pool_get_first() function
    - MINOR: pools: make the pool allocator support a few flags
    - MINOR: pools: add pool_zalloc() to return a zeroed area
    - CLEANUP: connection: use pool_zalloc() in conn_alloc_hash_node()
    - CLEANUP: filters: use pool_zalloc() in flt_stream_add_filter()
    - CLEANUP: spoe: use pool_zalloc() instead of pool_alloc+memset
    - CLEANUP: frontend: use pool_zalloc() in frontend_accept()
    - CLEANUP: mailers: use pool_zalloc() in enqueue_one_email_alert()
    - CLEANUP: resolvers: use pool_zalloc() in resolv_link_resolution()
    - CLEANUP: ssl: use pool_zalloc() in ssl_init_keylog()
    - CLEANUP: tcpcheck: use pool_zalloc() instead of pool_alloc+memset
    - CLEANUP: quic: use pool_zalloc() instead of pool_alloc+memset
    - MINOR: time: also provide a global, monotonic global_now_ms timer
    - BUG/MEDIUM: freq_ctr/threads: use the global_now_ms variable
    - MINOR: tools: introduce new option PA_O_DEFAULT_DGRAM on str2sa_range.
    - BUILD: tools: fix build error with new PA_O_DEFAULT_DGRAM
    - BUG/MINOR: ssl: Prevent disk access when using "add ssl crt-list"
    - CLEANUP: ssl: remove unused definitions
    - BUILD: ssl: guard ecdh functions with SSL_CTX_set_tmp_ecdh macro
    - MINOR: lua: Slightly improve function dumping the lua traceback
    - BUG/MEDIUM: debug/lua: Use internal hlua function to dump the lua traceback
    - BUG/MEDIUM: lua: Always init the lua stack before referencing the context
    - MINOR: fd: make fd_clr_running() return the remaining running mask
    - MINOR: fd: remove the unneeded running bit from fd_insert()
    - BUG/MEDIUM: fd: do not wait on FD removal in fd_delete()
    - CLEANUP: fd: remove unused fd_set_running_excl()
    - CLEANUP: fd: slightly simplify up _fd_delete_orphan()
    - BUG/MEDIUM: fd: Take the fd_mig_lock when closing if no DWCAS is available.
    - BUG/MEDIUM: release lock on idle conn killing on reached pool high count
    - BUG/MEDIUM: thread: Fix a deadlock if an isolated thread is marked as harmless
    - MINOR: tools: make url2ipv4 return the exact number of bytes parsed
    - BUG/MINOR: http_fetch: make hdr_ip() reject trailing characters
    - BUG/MEDIUM: mux-h1: make h1_shutw_conn() idempotent
    - BUG/MINOR: ssl: Fix update of default certificate
    - BUG/MINOR: ssl: Prevent removal of crt-list line if the instance is a default one
    - BUILD: ssl: introduce fine guard for ssl random extraction functions
    - REORG: global: move initcall register code in a dedicated file
    - REORG: global: move free acl/action in their related source files
    - REORG: split proxy allocation functions
    - MINOR: proxy: implement a free_proxy function
    - MINOR: proxy: define cap PR_CAP_LUA
    - MINOR: lua: properly allocate the lua Socket proxy
    - MINOR: lua: properly allocate the lua Socket servers
    - MINOR: vars: make get_vars() allow the session to be null
    - MINOR: vars: make the var() sample fetch keyword depend on nothing
    - CLEANUP: sample: remove duplicate "stopping" sample fetch keyword
    - MINOR: sample: make smp_resolve_args() return an allocate error message
    - MINOR: sample: add a new SMP_SRC_CONST sample capability
    - MINOR: sample: mark the truly constant sample fetch keywords as such
    - MINOR: sample: add a new CFG_PARSER context for samples
    - MINOR: action: add a new ACT_F_CFG_PARSER origin designation
    - MEDIUM: vars: add support for a "set-var" global directive
    - REGTESTS: add a basic reg-test for some "set-var" commands
    - MINOR: sample: add a new CLI_PARSER context for samples
    - MINOR: action: add a new ACT_F_CLI_PARSER origin designation
    - MINOR: vars/cli: add a "get var" CLI command to retrieve global variables
    - MEDIUM: cli: add a new experimental "set var" command
    - MINOR: compat: add short aliases for a few very commonly used types
    - BUILD: ssl: use EVP_CIPH_GCM_MODE macro instead of HA_OPENSSL_VERSION
    - MEDIUM: backend: use a trylock to grab a connection on high FD counts as well
2021-03-27 09:42:09 +01:00
Willy Tarreau
b8bd1ee893 MEDIUM: cli: add a new experimental "set var" command
set var <name> <expression>
  Allows to set or overwrite the process-wide variable 'name' with the result
  of expression <expression>. Only process-wide variables may be used, so the
  name must begin with 'proc.' otherwise no variable will be set. The
  <expression> may only involve "internal" sample fetch keywords and converters
  even though the most likely useful ones will be str('something') or int().
  Note that the command line parser doesn't know about quotes, so any space in
  the expression must be preceeded by a backslash. This command requires levels
  "operator" or "admin". This command is only supported on a CLI connection
  running in experimental mode (see "experimental-mode on").

Just like for "set-var" in the global section, the command uses a temporary
dummy proxy to create a temporary "set-var(name)" rule to assign the value.

The reg test was updated to verify that an updated global variable is properly
reflected in subsequent HTTP responses.
2021-03-26 16:57:43 +01:00
Willy Tarreau
c35eb38f1d MINOR: vars/cli: add a "get var" CLI command to retrieve global variables
Process-wide variables can now be displayed from the CLI using "get var"
followed by the variable name. They must all start with "proc." otherwise
they will not be found. The output is very similar to the one of the
debug converter, with a type and value being reported for the embedded
sample.

This command is limited to clients with the level "operator" or higher,
since it can possibly expose traffic-related data.
2021-03-26 16:52:13 +01:00