Commit Graph

505 Commits

Author SHA1 Message Date
Thayne McCombs
8f0cc5c4ba CLEANUP: Fix spelling errors in comments
This is from the output of codespell. It's done at once over a bunch
of files and only affects comments, so there is nothing user-visible.
No backport needed.
2021-01-08 14:56:32 +01:00
William Dauchy
3bad3d5ee6 BUILD: Makefile: exclude broken tests by default
it could be sometimes a bit confusing to have tests which are known to
be broken executed in the default `make reg-tests` command, especially
for not frequent contributors which are not necessarily aware of all our
quirks.

without this patch, this test is failing on my side:
  #    top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (2.228) exit=2
  1 tests failed, 0 tests skipped, 107 tests passed

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-08 14:43:27 +01:00
Tim Duesterhus
1ab76dd956 DOC: Improve the message printed when running make w/o TARGET
Rephrase the message to no longer talk about something that "is no longer
supported", but about what actually *is* supported.

Adjustments include:

- Removal of rare targets to make it easier to find the proper one.
- Reformatting to be easier to read (more newlines)
- Explanation of common non-default feature flags.
2021-01-05 19:32:48 +01:00
Willy Tarreau
782e9be1b3 Revert "BUILD: Makefile: disable -Warray-bounds until it's fixed in gcc 11"
This reverts commit 5e8c35da1b.

While the issue is being discussed with gcc folks, a reasonable workaround
could be found for the tls_keys_ref list usage which doesn't significantly
complicate the code. Since it was the only place triggering the warning and
I don't feel very comfortable leaving this one disabled for too long, let's
re-enable it right now. This definitely closes issue #1010.
2021-01-05 11:17:28 +01:00
Willy Tarreau
5e8c35da1b BUILD: Makefile: disable -Warray-bounds until it's fixed in gcc 11
Ilya reported in issue #1010 that gcc 11 (still in development phase 3)
sees out-of-bounds accesses to tlskeys_reference. After a deep analysis
it turns out that the warning is erroneous and Tim could set up a
reproducer that was further simplified to file a gcc bug report (98503).
In order not to break builds on this version, let's temporarily disable
the warning, since no acceptable workaround could be defined. Once gcc
gets it right, the warning should be re-enabled as it is likely to catch
real errors (it's the first false positive here for us even though
roughly one hundred issues were reported around this one alone).
2021-01-03 20:27:13 +01:00
Frdric Lcaille
b83b0105e8 MINOR: quic: Enable the compilation of QUIC modules.
Adds the QUIC C object files to the list to compile to support QUIC protocol.
USE_QUIC compilation option must be used for that.
2020-12-23 11:57:26 +01:00
Dragan Dosen
6f7cc11e6d MEDIUM: xxhash: use the XXH_INLINE_ALL macro to inline all functions
This way we make all xxhash functions inline, with implementations being
directly included within xxhash.h.

Makefile is updated as well, since we don't need to compile and link
xxhash.o anymore.

Inlining should improve performance on small data inputs.
2020-12-23 06:39:21 +01:00
Miroslav Zagorac
70230c6484 CONTRIB: opentracing: add the OpenTracing filter
This commit adds the OpenTracing filter (hereinafter we will use the
abbreviated name 'the OT filter') to the contrib tree.

The OT filter adds native support for using distributed tracing in HAProxy.
This is enabled by sending an OpenTracing compliant request to one of the
supported tracers; such as Datadog, Jaeger, Lightstep and Zipkin tracers.
Please note: tracers are not listed by any preference, but alphabetically.

The OT filter is a standard HAProxy filter, so what applies to others also
applies to this one (of course, by that I mean what is described in the
documentation, more precisely in the doc/internals/filters.txt file).

The OT filter activation is done explicitly by specifying it in the HAProxy
configuration.  If this is not done, the OT filter in no way participates
in the work of HAProxy.

As for the impact on HAProxy speed, this is documented with several tests
located in the test directory, and the result is found in the README-speed-*
files.  In short, the speed of operation depends on the way it is used and
the complexity of the configuration, from an almost immeasurable impact to
a significant deceleration (5x and more).  I think that in some normal use
the speed of HAProxy with the filter on will be quite satisfactory with a
slowdown of less than 4%.

The OT filter allows intensive use of ACLs, which can be defined anywhere in
the configuration.  Thus, it is possible to use the filter only for those
connections that are of interest to us.

More detailed documentation related to the operation, configuration and use
of the filter can be found in the contrib/opentracing directory.

To make the OpenTracing filter easier to configure and compile, several
entries have been added to the Makefile.  When running the make utility,
it is possible to use several new arguments:

  USE_OT=1     : enable the OpenTracing filter
  OT_DEBUG=1   : compile the OpenTracing filter in debug mode
  OT_INC=path  : force the include path to libopentracing-c-wrapper
  OT_LIB=path  : force the lib path to libopentracing-c-wrapper
  OT_RUNPATH=1 : add libopentracing-c-wrapper RUNPATH to haproxy executable

If USE_OT is set, then an additional Makefile from the contrib/opentracing
directory is included in the compilation process.
2020-12-16 15:49:53 +01:00
Willy Tarreau
da867d8d68 BUILD: Makefile: have "make clean" destroy .o/.a/.s in contrib subdirs as well
Now that we sometimes link some contrib subparts directly into the
haproxy binary, it's becoming a real problem that they're not cleaned
on make clean.  Some of the tools there are useful as .so or pure
binaries and we don't want to remove them, but anything intermediary
susceptible to be linked into haproxy should be clenaed. This is what
this patch does for 3 levels of subdirs into contrib/, without touching
the rest. It should be sufficient for the vast majority of use cases.
2020-12-16 14:19:44 +01:00
Tim Duesterhus
c8d19702f4 BUILD: Show the value of DEBUG= in haproxy -vv
Previously this was not visible after building.
2020-11-21 18:27:33 +01:00
Tim Duesterhus
81e948e051 BUILD: Make DEBUG part of .build_opts
This forces a recompilation if the value of DEBUG= changes.
2020-11-21 18:27:33 +01:00
Matthieu Guegan
496374e592 BUILD: makefile: enable crypt(3) for OpenBSD
Allow OpenBSD to support encrypted passwords in Userlists.

OpenBSD's crypt(3) function is provided directly by libc and does not
require -lcrypt.
Signed-off-by: Matthieu Guegan <matthieu.guegan@deindeal.ch>
2020-11-21 05:45:05 +01:00
Baptiste Assmann
e279ca6bbe MINOR: sample: Add converts to parses MQTT messages
This patch implements a couple of converters to validate and extract data from a
MQTT (Message Queuing Telemetry Transport) message. The validation consists of a
few checks as well as "packet size" validation. The extraction can get any field
from the variable header and the payload.

This is limited to CONNECT and CONNACK packet types only. All other messages are
considered as invalid. It is not a problem for now because only the first packet
on each side can be parsed (CONNECT for the client and CONNACK for the server).

MQTT 3.1.1 and 5.0 are supported.

Reviewed and Fixed by Christopher Faulet <cfaulet@haproxy.com>
2020-11-05 19:27:03 +01:00
Baptiste Assmann
e138dda1e0 MINOR: sample: Add converters to parse FIX messages
This patch implements a couple of converters to validate and extract tag value
from a FIX (Financial Information eXchange) message. The validation consists in
a few checks such as mandatory fields and checksum computation. The extraction
can get any tag value based on a tag string or tag id.

This patch requires the istend() function. Thus it depends on "MINOR: ist: Add
istend() function to return a pointer to the end of the string".

Reviewed and Fixed by Christopher Faulet <cfaulet@haproxy.com>
2020-11-05 19:26:30 +01:00
Willy Tarreau
2404e37838 BUILD: makefile: usual reorder of objects for faster builds
Reordered the objets by reverse build times made the total build time
go down from 17.7s to 17.2s at -O2 using make -j8 on my PC, and from
~3.2 to ~2.7s on the build farm.
2020-11-05 16:46:24 +01:00
Willy Tarreau
5c643f37d0 BUILD: makefile: add entries to build common debugging tools
A few tools in contrib/ such as halog, flags, poll and tcploop are
occasionally useful at least to developers, and some of them such as
halog or flags can occasionally break due to some changes in the include
files. As reported in issue #907, their inability to inherit the global
build options also causes some warnings related to some specificities
of the main include files. Let's just add entries in the main makefile
to build them.
2020-10-22 05:17:08 +02:00
Brad Smith
ad5afbafea BUILD: makefile: Enable getaddrinfo() on OS/X
Enable getaddrinfo() on OS/X.
2020-10-10 10:09:29 +02:00
Brad Smith
0fdfe4179e BUILD: makefile: Update feature flags for NetBSD
This updates the feature flags for NetBSD.

NetBSD 8 adds support for accept4().

Enable getaddrinfo().
2020-10-09 09:53:56 +02:00
Willy Tarreau
4a539f343a BUG/MINOR: makefile: fix a tiny typo in the target list
Previous commit 382001b46 ("BUILD: Add a DragonFlyBSD target") introduced
a tiny typo in the target list ("iopenbs" vs "openbsd"). This will have to
be backported if that patch is backported.
2020-10-09 05:58:40 +02:00
Brad Smith
382001b46b BUILD: Add a DragonFlyBSD target
Add a target for DragonFlyBSD 4.3 and above.
2020-10-08 20:54:18 +02:00
Brad Smith
7c503bb459 BUILD: makefile: Enable closefrom() support on Solaris
Solaris 9 (released 2002) added support for closefrom().

I bumped the version in the comment to 10 as the default feature
flags already has event ports enabled which were introduced in
Solaris 10.
2020-10-02 08:32:33 +02:00
Willy Tarreau
ac32b4b98c BUILD: makefile: add an EXTRAVERSION variable to ease local naming
Sometimes it's desirable to append local version naming to packages,
and currently it can only be done using SUBVERS which is already set
by default to the git commit ID and patch count since last known tag,
making the addition a bit complicated.

Let's just add a new EXTRAVERSION field that is empty by default, and
systematically appended verbatim to the version string everywhere. This
way it becomes trivial to append some local strings, such as:

   make TARGET=foo EXTRAVERSION=+$(quilt applied|wc -l)
   -> 2.3-dev5-5018aa-15+1

or :

   make TARGET=foo EXTRAVERSION=-$(date +%F)
   -> 2.3-dev5-5018aa-15-20200110

Let's be careful not to add double quotes (used as the string delimiter)
nor spaces (which can confuse version parsers on the output). The extra
version is also used to name a tarball. It's always pre-initialized to an
empty string so that it's not accidently inherited from the environment.
It's not reported in "make version" to avoid fooling tools (it would be
pointless anyway).

As a side effect it also becomes possible to force VERSION and SUBVERS
to an empty string and use EXTRAVERSION alone to force a specific version
(could possibly be useful when bisecting from patch queues outside of Git
for example).
2020-10-01 04:13:36 +02:00
Brad Smith
5018aacae5 BUILD: makefile: Fix building with closefrom() support enabled
I noticed the USE_CLOSEFROM define was not being passed along like the rest
during the build.

Looking around I see this was broken with the following two commits and related
series..

BUILD: Makefile: also report disabled options in the BUILD_OPTIONS variable
http://git.haproxy.org/?p=haproxy.git;a=commit;h=05fd82da76d1bbc8d65d63ab246bda7cbcf8481a

BUILD: pass all "USE_*" variables as -DUSE_* to the compiler
http://git.haproxy.org/?p=haproxy.git;a=commit;h=824cd00d3bda8f7f6d4c30baf77ba6c19ab47811

Looks like this should be back ported to 2.0, 2.1 and 2.2.
2020-09-30 16:06:51 +02:00
Brad Smith
9d85eb02d5 BUILD: makefile: Update feature flags for FreeBSD
This updates the feature flags for FreeBSD.

FreeBSD 10 adds support for accept4().

Enable getaddrinfo().

From the FreeBSD port / package.
2020-09-29 11:11:04 +02:00
Brad Smith
90c9d78447 BUILD: makefile: Update feature flags for OpenBSD
Update the OpenBSD target features being enabled.

I updated the list of features after noticing
"BUILD: makefile: disable threads by default on OpenBSD".

The Makefile utilizing gcc(1) by default resulted in utilizing
our legacy and obsolete compiler (GCC 4.2.1) instead of the
proper system compiler (Clang), which does support TLS. With
"BUILD: makefile: change default value of CC from gcc to cc"
that is resolved.
2020-09-27 21:48:18 +02:00
Brad Smith
94636436df BUILD: makefile: change default value of CC from gcc to cc
Change the default value of CC from gcc to cc to be more appropriate
for modern day mix of compilers. On GCC based OS's cc -> gcc. On Clang
based OS's cc -> clang. FreeBSD / OpenBSD have switched to Clang and
this corrects building with the proper compiler on OS's using Clang
as the default compiler. This especially matters for the necessity for
TLS on OpenBSD. I would expect this affects OpenMandriva and other
Linux OS's using Clang as well.
2020-09-15 07:32:06 +02:00
Ilya Shipitsin
4a034f2212 BUILD: introduce possibility to define ABORT_NOW() conditionally
code analysis tools recognize abort() better, so let us introduce
such possibility
2020-09-12 13:11:27 +02:00
Willy Tarreau
0d06df6448 MINOR: sock: introduce sock_inet and sock_unix
These files will regroup everything specific to AF_INET, AF_INET6 and
AF_UNIX socket definitions and address management. Some code there might
be agnostic to the socket type and could later move to af_xxxx.c but for
now we only support regular sockets so no need to go too far.

The files are quite poor at this step, they only contain the address
comparison function for each address family.
2020-08-28 18:51:36 +02:00
Willy Tarreau
18b7df7a2b REORG: sock: start to move some generic socket code to sock.c
The new file sock.c will contain generic code for standard sockets
relying on file descriptors. We currently have way too much duplication
between proto_uxst, proto_tcp, proto_sockpair and proto_udp.

For now only get_src, get_dst and sock_create_server_socket were moved,
and are used where appropriate.
2020-08-28 18:51:36 +02:00
Willy Tarreau
1318034317 REORG: unix: move UNIX bind/server keywords from proto_uxst.c to cfgparse-unix.c
Let's finish the cleanup and get rid of all bind and server keywords
parsers from proto_uxst.c. They're now moved to cfgparse-unix.c. Now
proto_uxst.c is clean and only contains code related to binding and
connecting.
2020-08-28 18:51:36 +02:00
Willy Tarreau
de70ca5dfd REORG: tcp: move TCP bind/server keywords from proto_tcp.c to cfgparse-tcp.c
Let's continue the cleanup and get rid of all bind and server keywords
parsers from proto_tcp.c. They're now moved to cfgparse-tcp.c, just as
was done for ssl before 2.2 release. Nothing has changed beyond this.
Now proto_tcp.c is clean and only contains code related to binding and
connecting.
2020-08-28 18:51:36 +02:00
Willy Tarreau
8987e7a8c9 REORG: tcp: move TCP sample fetches from proto_tcp.c to tcp_sample.c
Let's continue the cleanup and get rid of all sample fetch functions
from proto_tcp.c. They're now moved to tcp_sample.c, just as was done
for ssl before 2.2 release. Nothing has changed beyond this.
2020-08-28 18:51:36 +02:00
Willy Tarreau
aeae66cf22 REORG: tcp: move TCP actions from proto_tcp.c to tcp_act.c
The file proto_tcp.c has become a real mess because it still contains
tons of definitions that have nothing to do with the TCP protocol setup.
This commit moves the ruleset actions "set-src-port", "set-dst-port",
"set-src", "set-dst", and "silent-drop" to a new file "tcp_act.c".
Nothing has changed beyond this.
2020-08-28 18:51:36 +02:00
Willy Tarreau
b36dcc66ed BUILD: makefile: don't disable -Wstringop-overflow anymore
This basically reverts commit c4e6460f6 ("MINOR: build: Disable
-Wstringop-overflow.") which is no more needed after previous one.
2020-08-11 10:31:18 +02:00
Ilya Shipitsin
f21023e1cf BUILD: Makefile: require SSL_LIB, SSL_INC to be explicitly set
The SSL_INC and SSL_LIB variables were not initialized in the Makefile,
so they could be accidently inherited from the environment. We require
that any makefile variable is explicitly set on the command line so they
must be initialized.

Note that the Travis scripts used to rely only on these variables to be
exported, so it was adjusted as well.
2020-08-05 11:37:32 +02:00
Emeric Brun
3835c0dcb5 MEDIUM: udp: adds minimal proto udp support for message listeners.
This patch introduce proto_udp.c targeting a further support of
log forwarding feature.

This code was originally produced by Frederic Lecaille working on
QUIC support and only minimal requirements for syslog support
have been merged.
2020-07-15 17:50:12 +02:00
Willy Tarreau
4c442b08b3 CLEANUP: makefile: update the outdated list of DEBUG_xxx options
A few options didn't exist anymore (FSM, HASH) and quite a few ones were
added since last update (MEM_STATS, DONT_SHARE_POOLS, NO_LOCKLESS_POOLS,
NO_LOCAL_POOLS, FAIL_ALLOC, STRICT_NOCRASH, HPACK.
2020-07-04 12:43:46 +02:00
Willy Tarreau
45e83e8c56 BUILD: makefile: disable threads by default on OpenBSD
As reported by Ilya in issue #725, building with threads on OpenBSD
is broken with gcc:

  include/haproxy/tinfo.h:30: error: thread-local storage not supported for this target

Better stay safe and disable it. Clang seems to support (or emulate)
thread-local, at least it builds. Those willing to experiment can
easily pass USE_THREAD=1.
2020-07-03 18:56:33 +02:00
Willy Tarreau
739879a23b REORG: buffer: rename buffer.c to dynbuf.c
The include part was renamed by commit 2741c8c but I somehow missed
the renaming of the C file, whose name didn't match the H file anymore.
2020-06-29 09:26:59 +02:00
Willy Tarreau
38e8a1c7b8 MINOR: debug: add a new DEBUG_FD build option
When DEBUG_FD is set at build time, we'll keep a counter of per-FD events
in the fdtab. This counter is reported in "show fd" even for closed FDs if
not zero. The purpose is to help spot situations where an apparently closed
FD continues to be reported in loops, or where some events are dismissed.
2020-06-23 10:04:54 +02:00
Tim Duesterhus
588b3148d9 BUILD: Re-enable -Wimplicit-fallthrough
Getting rid of this warning is cleaner solved using a 'fall through' comment,
because it clarifies intent to a human reader.

This patch adjust a few places that cause -Wimplicit-fallthrough to trigger:

- Fix typos in the comment.
- Remove redundant 'no break' that trips up gcc from comment.
- Move the comment out of the block when the 'case' is completely surrounded
  by braces.
- Add comments where I could determine that the fall through was intentional.

Changes tested on

    gcc (Debian 9.3.0-13) 9.3.0
    Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

using

    make -j4 all TARGET=linux-glibc USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1
2020-06-11 16:49:37 +02:00
Tim Duesterhus
4989de2811 BUILD: Remove nowarn for warnings that do not trigger
Tested with

    make -j4 all TARGET=linux-glibc USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1

against

    gcc (Debian 9.3.0-13) 9.3.0
    Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
2020-06-11 15:08:27 +02:00
Willy Tarreau
7c18b54106 REORG: dgram: rename proto_udp to dgram
The set of files proto_udp.{c,h} were misleadingly named, as they do not
provide anything related to the UDP protocol but to datagram handling
instead, since currently all UDP processing is hard-coded where it's used
(dns, logs). They are to UDP what connection.{c,h} are to proto_tcp. This
was causing confusion about how to insert UDP socket management code,
so let's rename them right now to dgram.{c,h} which more accurately
matches what's inside since every function and type is already prefixed
with "dgram_".
2020-06-11 10:18:59 +02:00
Willy Tarreau
920214e8c4 BUILD: reorder objects in the Makefile for faster builds
Splitting large files and changing includes has changed the per-file
build time. After a careful reordering based on build time, we're now
down to 5.8s at -O0 on the PC at -j8 and 2.4-2.6s on the farm at -j120.
Some room for at least one file name was left on each line to ease
future additions.
2020-06-11 10:18:59 +02:00
Willy Tarreau
bcc6733fab REORG: check: extract the external checks from check.{c,h}
The health check code is ugly enough, let's take the external checks
out of it to simplify the code and shrink the file a little bit.
2020-06-11 10:18:58 +02:00
Willy Tarreau
51cd5956ee REORG: check: move tcpchecks away from check.c
Checks.c remains one of the largest file of the project and it contains
too many things. The tcpchecks code represents half of this file, and
both parts are relatively isolated, so let's move it away into its own
file. We now have tcpcheck.c, tcpcheck{,-t}.h.

Doing so required to export quite a number of functions because check.c
has almost everything made static, which really doesn't help to split!
2020-06-11 10:18:58 +02:00
Willy Tarreau
4aa573da6f REORG: include: move checks.h to haproxy/check{,-t}.h
All includes that were not absolutely necessary were removed because
checks.h happens to very often be part of dependency loops. A warning
was added about this in check-t.h. The fields, enums and structs were
a bit tidied because it's particularly tedious to find anything there.
It would make sense to split this in two or more files (at least
extract tcp-checks).

The file was renamed to the singular because it was one of the rare
exceptions to have an "s" appended to its name compared to the struct
name.
2020-06-11 10:18:58 +02:00
Willy Tarreau
48fbcae07c REORG: tools: split common/standard.h into haproxy/tools{,-t}.h
And also rename standard.c to tools.c. The original split between
tools.h and standard.h dates from version 1.3-dev and was mostly an
accident. This patch moves the files back to what they were expected
to be, and takes care of not changing anything else. However this
time tools.h was split between functions and types, because it contains
a small number of commonly used macros and structures (e.g. name_desc)
which in turn cause the massive list of includes of tools.h to conflict
with the callers.

They remain the ugliest files of the whole project and definitely need
to be cleaned and split apart. A few types are defined there only for
functions provided there, and some parts are even OS-specific and should
move somewhere else, such as the symbol resolution code.
2020-06-11 10:18:57 +02:00
Willy Tarreau
d0ef439699 REORG: include: move common/memory.h to haproxy/pool.h
Now the file is ready to be stored into its final destination. A few
minor reorderings were performed to keep the file properly organized,
making the various sections more visible (cache & lockless).

In addition and to stay consistent, memory.c was renamed to pool.c.
2020-06-11 10:18:57 +02:00
Willy Tarreau
3f567e4949 REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h
This splits the hathreads.h file into types+macros and functions. Given
that most users of this file used to include it only to get the definition
of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h
(i.e. types and macros).

All the thread management was left to haproxy/thread.h. It's worth noting
the drop of the trailing "s" in the name, to remove the permanent confusion
that arises between this one and the system implementation (no "s") and the
makefile's option (no "s").

For consistency, src/hathreads.c was also renamed thread.c.

A number of files were updated to only include thread-t which is the one
they really needed.

Some future improvements are possible like replacing empty inlined
functions with macros for the thread-less case, as building at -O0 disables
inlining and causes these ones to be emitted. But this really is cosmetic.
2020-06-11 10:18:56 +02:00
Willy Tarreau
fc80e30217 REORG: ebtree: clean up remains of the ebtree/ directory
The only leftovers were the unused compiler.h file and the LICENSE file
which is already mentioned in each and every ebtree file header.

A few build paths were updated in the contrib/ directory not to mention
this directory anymore, and all its occurrences were dropped from the
main makefile. From now on no other include path but include/ will be
needed anymore to build any file.
2020-06-11 09:31:11 +02:00
Willy Tarreau
ff0e8a44a4 REORG: ebtree: move the C files from ebtree/ to src/
As part of the include files cleanup, we're going to kill the ebtree
directory. For this we need to host its C files in a different location
and src/ is the right one.
2020-06-11 09:31:11 +02:00
William Lallemand
6a66a5ec9b REORG: ssl: move utility functions to src/ssl_utils.c
These functions are mainly used to extract information from
certificates.
2020-05-15 14:11:54 +02:00
William Lallemand
15e169447d REORG: ssl: move sample fetches to src/ssl_sample.c
Move all SSL sample fetches to src/ssl_sample.c.
2020-05-15 14:11:54 +02:00
William Lallemand
dad3105157 REORG: ssl: move ssl configuration to cfgparse-ssl.c
Move all the configuration parsing of the ssl keywords in cfgparse-ssl.c
2020-05-15 14:11:54 +02:00
William Lallemand
03c331c80a REORG: ssl: move the ckch_store related functions to src/ssl_ckch.c
Move the cert_key_and_chain functions:

int ssl_sock_load_files_into_ckch(const char *path, struct cert_key_and_chain *ckch, char **err);
int ssl_sock_load_pem_into_ckch(const char *path, char *buf, struct cert_key_and_chain *ckch , char **err);
void ssl_sock_free_cert_key_and_chain_contents(struct cert_key_and_chain *ckch);

int ssl_sock_load_key_into_ckch(const char *path, char *buf, struct cert_key_and_chain *ckch , char **err);
int ssl_sock_load_ocsp_response_from_file(const char *ocsp_path, char *buf, struct cert_key_and_chain *ckch, char **err);
int ssl_sock_load_sctl_from_file(const char *sctl_path, char *buf, struct cert_key_and_chain *ckch, char **err);
int ssl_sock_load_issuer_file_into_ckch(const char *path, char *buf, struct cert_key_and_chain *ckch, char **err);

And the utility ckch_store functions:

void ckch_store_free(struct ckch_store *store)
struct ckch_store *ckch_store_new(const char *filename, int nmemb)
struct ckch_store *ckchs_dup(const struct ckch_store *src)
ckch_store *ckchs_lookup(char *path)
ckch_store *ckchs_load_cert_file(char *path, int multi, char **err)
2020-05-15 14:11:54 +02:00
William Lallemand
6e9556b635 REORG: ssl: move crtlist functions to src/ssl_crtlist.c
Move the crtlist functions to src/ssl_crtlist.c and their definitions to
proto/ssl_crtlist.h.

The following functions were moved:

/* crt-list entry functions */
void ssl_sock_free_ssl_conf(struct ssl_bind_conf *conf);
char **crtlist_dup_filters(char **args, int fcount);
void crtlist_free_filters(char **args);
void crtlist_entry_free(struct crtlist_entry *entry);
struct crtlist_entry *crtlist_entry_new();

/* crt-list functions */
void crtlist_free(struct crtlist *crtlist);
struct crtlist *crtlist_new(const char *filename, int unique);

/* file loading */
int crtlist_parse_line(char *line, char **crt_path, struct crtlist_entry *entry, const char *file, int linenum, char **err);
int crtlist_parse_file(char *file, struct bind_conf *bind_conf, struct proxy *curproxy, struct crtlist **crtlist, char **err);
int crtlist_load_cert_dir(char *path, struct bind_conf *bind_conf, struct crtlist **crtlist, char **err);
2020-05-15 14:11:54 +02:00
Willy Tarreau
39b2fda915 BUILD: Makefile: add linux-musl to TARGET
Other users are using musl, namely on Docker. It builds fine with
linux-glibc-legacy but not linux-glibc, which needs to first disable
USE_BACKTRACE. Better add a valid entry for it instead of hacking
around another libc.
2020-04-16 15:17:13 +02:00
Willy Tarreau
48e8603a9c BUILD: makefile: fix expression again to detect ARM platform
I messed up the fix in 67b095e ("BUILD: makefile: fix regex syntax in
ARM platform detection"), I tried it by hand in the shell without "-v"
but left it in the expression. It works on ARM because it only finds
lines starting with '#' but on other platforms it insists for -latomic.
2020-03-18 08:21:57 +01:00
Willy Tarreau
67b095e797 BUILD: makefile: fix regex syntax in ARM platform detection
Commit d93e6ec ("BUILD: on ARM, must be linked to libatomic.") broke the
build due to a missing backslash in front of the '#':

  Makefile:331: *** invalid syntax in conditional.  Stop.

Let's address this and make sure we only pick relevant lines (and not
possibly empty lines).
2020-03-16 09:38:00 +01:00
David Carlier
d93e6ec4cc BUILD: on ARM, must be linked to libatomic.
For load/store operations, needs to be linked to.
tested on raspberry.
2020-03-16 08:33:17 +01:00
Willy Tarreau
4b61e694b4 BUILD: Makefile: the compiler-specific flags should all be in SPEC_CFLAGS
We used to have -Wall -Wextra -Werror in COPTS which are flags fed by
the various USE_* options, and all other warnings in SPEC_CFLAGS. This
makes it impossible to remove these -W* entries (typically -Wextra).

Let's move these 3 flags into SPEC_CFLAGS where they should have been.
Now it's possible to override SPEC_CFLAGS to match any compiler's
specificities, or to clear all warnings at once, or to replace them
all with "-w" to silence warnings.
2020-03-10 09:42:30 +01:00
Willy Tarreau
06c63aec95 CLEANUP: remove support for USE_MY_SPLICE
The splice() syscall has been supported in glibc since version 2.5 issued
in 2006 and is present on supported systems so there's no need for having
our own arch-specific syscall definitions anymore.
2020-03-10 07:23:41 +01:00
Willy Tarreau
3858b122a6 CLEANUP: remove support for USE_MY_EPOLL
This was made to support epoll on patched 2.4 kernels, and on early 2.6
using alternative libcs thanks to the arch-specific syscall definitions.
All the features we support have been around since 2.6.2 and present in
glibc since 2.3.2, neither of which are found in field anymore. Let's
simply drop this and use epoll normally.
2020-03-10 07:08:10 +01:00
Willy Tarreau
618ac6ea52 CLEANUP: drop support for USE_MY_ACCEPT4
The accept4() syscall has been present for a while now, there is no more
reason for maintaining our own arch-specific syscall implementation for
systems lacking it in libc but having it in the kernel.
2020-03-10 07:02:46 +01:00
Willy Tarreau
c3e926bf3b CLEANUP: remove support for Linux i686 vsyscalls
This was introduced 10 years ago to squeeze a few CPU cycles per syscall
on 32-bit x86 machines and was already quite old by then, requiring to
explicitly enable support for this in the kernel. We don't even know if
it still builds, let alone if it works at all on recent kernels! Let's
completely drop this now.
2020-03-10 06:55:52 +01:00
Ilya Shipitsin
2a950d02a9 DOC: assorted typo fixes in the documentation and Makefile
This is another round of cleanups in various docs and comments in the
Makefile.
2020-03-06 10:49:55 +01:00
Willy Tarreau
6f6d96de19 BUILD: makefile: do not modify the build options during make reg-tests
I'm quite fed up with having to rebuild everything from scratch after each
and every "make reg-tests", especially during bisects. The only reason for
this is that there are no build options passed to make for reg-tests, which
modifies the .build_opts file again, resulting in a change upon next build.
Let's just keep this file out of the dependency check for make reg-tests.
2020-03-04 19:29:12 +01:00
Willy Tarreau
c0bbdc196d BUILD: Makefile: include librt before libpthread
Statically building on for i386/x86_64 on linux+glibc 2.18 fails in rt with
undefined references to pthread_attr_init and a few others. Let's just swap
the two libs in order to fix this.
2020-03-04 12:02:27 +01:00
Willy Tarreau
f5b4e064dc MEDIUM: debug: add support for dumping backtraces of stuck threads
When a panic() occurs due to a stuck thread, we'll try to dump a
backtrace of this thread if the config directive USE_BACKTRACE is
set (which is the case on linux+glibc). For this we use the
backtrace() call provided by glibc and iterate the pointers through
resolve_sym_name(). In order to minimize the output (which is limited
to one buffer), we only do this for stuck threads, and we start the
dump above ha_panic()/ha_thread_dump_all_to_trash(), and stop when
meeting known points such as main/run_tasks_from_list/run_poll_loop.

If enabled without USE_DL, the dump will be complete with no details
except that pointers will all be given relative to main, which is
still better than nothing.

The new USE_BACKTRACE config option is enabled by default on glibc since
it has been present for ages. When it is set, the export-dynamic linker
option is enabled so that all non-static symbols are properly resolved.
2020-03-03 18:40:03 +01:00
Willy Tarreau
d1c847abbc BUILD: makefile: re-enable strict aliasing
For a very long time we've used to build without strict aliasing due to
very few places in the stick-tables code mostly, that initially we didn't
know how to deal with. The problem of doing this is that it encourages
to write possibly incorrect code such as the few SSL sample fetch functions
that were recently fixed.

All places causing aliasing errors on x86_64, i586, armv8, armv7 and
mips were fixed so it's about time to re-enable the warning hoping to
catch such errors early in the development cycle. As a bonus, this
removed about 5kB of code.
2020-02-25 10:24:51 +01:00
Willy Tarreau
03e7853581 BUILD: remove obsolete support for -mregparm / USE_REGPARM
This used to be a minor optimization on ix86 where registers are scarce
and the calling convention not very efficient, but this platform is not
relevant enough anymore to warrant all this dirt in the code for the sake
of saving 1 or 2% of performance. Modern platforms don't use this at all
since their calling convention already defaults to using several registers
so better get rid of this once for all.
2020-02-25 07:41:47 +01:00
Christian Lachner
c13223022c MINOR: build: add aix72-gcc build TARGET and power{8,9} CPUs
As haproxy wont build on AIX 7.2 using the old "aix52" TARGET a new
TARGET was introduced which adds two special CFLAGS to prevent the
loading of AIXs xmem.h and var.h. This is done by defining the
corresponding include-guards _H_XMEM and _H_VAR. Without excluding
those headers-files the build fails because of redefinition errors:

1)
  CC      src/mux_fcgi.o
In file included from /usr/include/sys/uio.h:90,
                 from /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/8.3.0/include-fixed/sys/socket.h:104,
                 from include/common/compat.h:32,
                 from include/common/cfgparse.h:25,
                 from src/mux_fcgi.c:13:
src/mux_fcgi.c:204:13: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token
  struct ist rem_addr;
             ^~~~~~~~

2)
  CC      src/cfgparse-listen.o
In file included from include/types/arg.h:31,
                 from include/types/acl.h:29,
                 from include/types/proxy.h:41,
                 from include/proto/log.h:34,
                 from include/common/cfgparse.h:30,
                 from src/mux_h2.c:13:
include/types/vars.h:30:8: error: redefinition of 'struct var'
 struct var {
        ^~~

Futhermore, to enable multithreading via USE_THREAD, the atomic
library was added to the LDFLAGS. Finally, two new CPUs were added
to simplify the usage of power8 and power9 optimizations.

This TARGET was only tested on GCC 8.3 and may or may not work on
IBM's native C-compiler (XLC).

Should be backported to 2.1.
2020-02-12 15:37:13 +01:00
Tim Duesterhus
541fe1ec52 MINOR: lua: Add HLUA_PREPEND_C?PATH build option
This complements the lua-prepend-path configuration option to allow
distro maintainers to add a default path for HAProxy specific Lua
libraries.
2020-01-24 09:22:03 +01:00
Willy Tarreau
ef915dc9ca BUILD: reorder the objects in the makefile
After a number of reorganization, addition of fcgi and the removal of
the legacy mode, some late files ended up being slow to build and were
slowing down the parallel build. Let's reorder them based on the build
time. Full build went down from 8.3-9.2s to 6.8s.
2019-11-25 19:47:23 +01:00
Willy Tarreau
c1b16734c0 BUILD: do not disable -Wformat-truncation anymore
There were very few entries to fix and this warning, while often
wrong, can actually spot future issues. If it can help developers
adjust their code in the future to make it more robust, it's not
necessarily that bad. Let's re-enable it and see how it goes.
2019-10-29 10:54:24 +01:00
David Carlier
5e4c8e2a67 BUILD/MEDIUM: threads: enable cpu_affinity on osx
Enable it but on a per thread basis only using Darwin native API.
2019-10-17 07:20:58 +02:00
David Carlier
7fa6fdf86a BUILD/SMALL: threads: enable threads on osx
Enable multi thread on Darwin platform too.
2019-10-17 07:18:00 +02:00
Willy Tarreau
c77c2d4f9c BUILD: makefile: work around yet another GCC fantasy (-Wstring-plus-int)
According to issue #294 some gcc versions suspect that developers are
having trouble dealing with string offsets and now emit another new
childish warning when mapping indexes to characters. Instead of annoying
developers each time it happens and ask them to modify their valid code,
let's just get rid of this absurd warning.
2019-09-24 09:43:34 +02:00
Christopher Faulet
99eff65f4f MEDIUM: mux-fcgi: Add the FCGI multiplexer
This multiplexer is only available on the backend side. It may handle
multiplexed connections if the FCGI application supports it. A FCGI application
must be configured on the backend to be used. If not redefined during the
request processing by the FCGI filter, this mux handles all mandatory
parameters.

There is a limitation on the way the requests are processed. The parameters must
be encoded into a uniq PARAMS record. It means, once encoded, all HTTP headers
and FCGI parameters must small enough to be store in a buffer. Otherwise, an
internal processing error is returned.
2019-09-17 10:18:54 +02:00
Christopher Faulet
78fbb9f991 MEDIUM: fcgi-app: Add FCGI application and filter
The FCGI application handles all the configuration parameters used to format
requests sent to an application. The configuration of an application is grouped
in a dedicated section (fcgi-app <name>) and referenced in a backend to be used
(use-fcgi-app <name>). To be valid, a FCGI application must at least define a
document root. But it is also possible to set the default index, a regex to
split the script name and the path-info from the request URI, parameters to set
or unset...  In addition, this patch also adds a FCGI filter, responsible for
all processing on a stream.
2019-09-17 10:18:54 +02:00
Christopher Faulet
63bbf284a1 MINOR: fcgi: Add code related to FCGI protocol
This code is independant and is only responsible to encode and decode part of
the FCGI protocol.
2019-09-17 10:18:54 +02:00
Christopher Faulet
4f0f88a9d0 MEDIUM: mux-h1/h1-htx: move HTX convertion of H1 messages in dedicated file
To avoid code duplication in the futur mux FCGI, functions parsing H1 messages
and converting them into HTX have been moved in the file h1_htx.c. Some
specific parts remain in the mux H1. But most of the parsing is now generic.
2019-09-17 10:18:54 +02:00
David Carlier
6c00eba63b BUILD/MINOR: auth: enabling for osx
macOS supports this but as part of libc.
Little typo fix while here.
2019-09-08 12:20:13 +02:00
Lukas Tribus
cc1eb1619f MINOR: build: add linux-glibc-legacy build TARGET
As discussed in issue #128, introduce a new build TARGET
linux-glibc-legacy to allow the build on old, legacy OS.

Should be backported to 2.0.
2019-09-01 17:28:10 +02:00
Willy Tarreau
172945fbad MINOR: ring: add a new mechanism for retrieving/storing ring data in buffers
Our circular buffers are well suited for being used as ring buffers for
not-so-structured data. The machanism here consists in making room in a
buffer before inserting a new record which is prefixed by its size, and
looking up next record based on the previous one's offset and size. We
can have up to 255 consumers watching for data (dump in progress, tail)
which guarantee that entrees are not recycled while they're being dumped.
The complete representation is described in the header file. For now only
ring_new(), ring_resize() and ring_free() are created.
2019-08-27 17:14:19 +02:00
Willy Tarreau
4151c753fc MINOR: trace: start to create a new trace subsystem
The principle of this subsystem will be to support taking live traces
at various places in the code with conditional triggers, filters, and
ability to lock on some elements. The traces will support typed events
and will be sent into sinks made of ring buffers, file descriptors or
remote servers.
2019-08-22 20:21:00 +02:00
Willy Tarreau
67b5a161b4 MINOR: sink: create definitions a minimal code for event sinks
The principle will be to be able to dispatch events to various destinations
called "sinks". This is already done in part in logs where log servers can
be either a UDP socket or a file descriptor. This will be needed with the
new trace subsystem where we may also want to add ring buffers. And it turns
out that all such destinations make sense at all places. Logs may need to be
sent to a TCP server via a ring buffer, or consulted from the CLI. Trace
events may need to be sent to stdout/stderr as well as to remote log servers.

This patch creates a new structure "sink" aiming at addressing these similar
needs. The goal is to merge together what is common to all of them, such as
the output format, the dropped events count, etc, and also keep separately
the target identification (network address, file descriptor). Provisions
were made to have a "waiter" on the sink. For a TCP log server it will be
the task to wake up after writing to the log buffer. For a ring buffer, it
could be the list of watchers on the CLI running a "tail" operation and
waiting for new events. A lock was also placed in the struct since many
operations will require some locking, including the FD ones. The output
formats covers those in use by logs and two extra ones prepending the ISO
time in front of the message (convenient for stdio/buffer).

For now only the generic infrastructure is present, no type-specific
output is implemented. There's the sink_write() function which prepares
and formats a message to be sent, trying hard to avoid copies and only
using pointer manipulation, where the type-specific code just has to be
added. Dropped messages are already counted (for now 100% drop). The
message is put into an iovec array as it will be trivial to use with
file descriptors and sockets.
2019-08-22 20:21:00 +02:00
Willy Tarreau
9eebd8a978 REORG: trace: rename trace.c to calltrace.c and mention it's not thread-safe
The function call tracing code is a quite old and was never ported to
support threads. It's not even sure whether it still works well, but
at least its presence creates confusion for future work so let's rename
it to calltrace.c and add a comment about its lack of thread-safety.
2019-08-22 20:21:00 +02:00
Christopher Faulet
fc9cfe4006 REORG: proto_htx: Move HTX analyzers & co to http_ana.{c,h} files
The old module proto_http does not exist anymore. All code dedicated to the HTTP
analysis is now grouped in the file proto_htx.c. So, to finish the polishing
after removing the legacy HTTP code, proto_htx.{c,h} files have been moved in
http_ana.{c,h} files.

In addition, all HTX analyzers and related functions prefixed with "htx_" have
been renamed to start with "http_" instead.
2019-07-19 09:24:12 +02:00
Christopher Faulet
711ed6ae4a MAJOR: http: Remove the HTTP legacy code
First of all, all legacy HTTP analyzers and all functions exclusively used by
them were removed. So the most of the functions in proto_http.{c,h} were
removed. Only functions to deal with the HTTP transaction have been kept. Then,
http_msg and hdr_idx modules were entirely removed. And finally the structure
http_msg was lightened of all its useless information about the legacy HTTP. The
structure hdr_ctx was also removed because unused now, just like unused states
in the enum h1_state. Note that the memory pool "hdr_idx" was removed and
"http_txn" is now smaller.
2019-07-19 09:24:12 +02:00
Willy Tarreau
3c55efb7dd BUILD: makefile: do not rely on shell substitutions to determine git version
Solaris's default shell doesn't support substitutions at the beginning or
end of variables, which are still used to determine the version based on
git. Since we added --abbrev=0 we don't need the last one. And using cut
it's trivial to replace the first one, actually simplifying the whole
expression.

This may be backported to all stable branches.
2019-06-22 08:28:32 +02:00
Willy Tarreau
e8d0719924 BUILD: makefile: adjust the sed expression of "make help" for solaris
Solaris's sed doesn't take the 'p' argument on the 's' command so
nothing is printed. Just passing ';p' fixes this without affecting
other implementations. Additionally, optional characters cannot be
matched using a question mark, which is always searched verbatim, so
the leading '#' wasn't stripped. Using \{0,1\} works fine everywhere
so let's use this instead.
2019-06-22 08:13:24 +02:00
Willy Tarreau
30a6f6402e BUILD: makefile: use :space: instead of digits to count commits
The 'tr' command on Solaris doesn't conform to POSIX and requires
brackets around ranges. So the sequence '0-9' is understood as the
3 characters '0', '-', and '9'. This causes tagged versions (those
with no commit after the last commit) to be numberred as an empty
string, resulting in an error being reported while computing the
version number.

All implementations support '[:space:]' to delete heading spaces,
so let's use this instead.

This may be backported to all stable versions.
2019-06-22 07:58:28 +02:00
Willy Tarreau
5fb76036a3 BUILD: makefile: do not use echo -n
On certain UNIXes (Solaris at least), echo -n displays "-n". Let's
simply re-arrange the "if" block in the help message not to use it.
2019-06-16 19:26:18 +02:00
Willy Tarreau
f8cc19a1fe BUILD: makefile: enable USE_RT on Solaris
It doesn't build on Ultra5 under Solaris 10 without this due to
clock_gettime() not being found.
2019-06-16 19:18:53 +02:00
Willy Tarreau
c31e5a5126 BUILD: makefile: make the obsolete target detection compatible with make-3.80
Older versions of GNU make do not support "else ifneq", let's split
this in two lines.
2019-06-16 17:53:39 +02:00
Willy Tarreau
364d6f529c BUILD: makefile: enable getaddrinfo on the linux-glibc target
getaddrinfo() has been available since glibc 2.3.3 or so and is generally
enabled by distro packagers. The main reason for not enabling it on Linux
in the past is that it was known broken on some libc alternatives. It's
the right moment to enable it by default with glibc.
2019-06-15 18:03:48 +02:00
Willy Tarreau
a26181e74b BUILD: makefile: enable TFO on linux platforms
TCP Fast Open is supported on all supported Linux kernels and on all
kernels shipped in supported distros, except the older 2.6.32 that
comes with RHEL6. However the option is harmless, will not prevent
from building and smoothly falls back even if forcefully enabled, so
it makes sense to enable it by default. It's still possible to pass
"USE_TFO=" to force it disabled if really desired.
2019-06-15 18:03:48 +02:00
Willy Tarreau
20e6cedc43 BUILD: makefile: enable linux namespaces by default on linux
Oldest kernel found on a supported Linux distro (2.6.32 + backports on
RHEL6) supports network namespaces, so we have no reason not to enable
them by default on the linux-glibc target.
2019-06-15 18:03:48 +02:00