6466 Commits

Author SHA1 Message Date
3c047d05af 1.7.1-alt1
- 1.7.1
2017-01-12 11:35:52 +03:00
5aad49c6b4 HAProxy 1.7.1
-----BEGIN PGP SIGNATURE-----
 
 iQIcBAABAgAGBQJYTzGyAAoJEE44bZycYXAvUw0P/1Avban7D7Ua5H48Xp9MeL3R
 YF1bjgC6wmMYzBB/xrLRnh8oLKyx02/b5yqepL1rakPrGVJ9dK7GGo9u00qnVJcL
 80G2iTa/f117dMm8UkSvJ/ab46cb/EUGSGZHU9EjiJJBKJcYxWo+tT4cgTIN5xHb
 lyUZyzFzFGPh1mPvNea50EIETNnijZPsoKioXfLceduWRFrsacNnJoPaGjP6dOBT
 SND8gUKvaZiqtJ/CdaxXeZ9fVx2ta700Dl5LWsNrt8qCoVZUHn159bND0QZSHurF
 hz7+TE/n3YVoD/uOdLasg8K0rvYatH5QdXjED0NVmvRV5egWOSz78pzhK4rkj0YY
 PY+2jdioJd2+ogPgbDaB9Pul1FHY9pyZId0dwPm3dCjyfw73ZKmZ03do2WYjf85Y
 pTLSEVVisc4BBAqqb27Xu44djs9f0KF0lT1XGMy6hOpNHELNWmdwHpK7L+ff5G0y
 E8FEmj1mLxMI4jPqLtcLUs1JrS35Vab4QtnL8eJnh1IV4ZRVx4FT1R3viqab1FgC
 vIWxYA5vVMDIcpnolZO3u9K2DCiZPDwcQY4IHucBBLWcUct0UV2CnF+6KFgwA8rO
 Z/3wiAtpLbIDUSKxffoEpIi9HlXE7REPR4YiC+vFdp9gnuINsdb8P3FUaaoA3x/7
 HwByWt9HBr6khWaH6VW2
 =+PoN
 -----END PGP SIGNATURE-----

Merge tag 'v1.7.1'

HAProxy 1.7.1

# gpg: Подпись сделана Вт 13 дек 2016 02:24:34 MSK
# gpg:                ключом RSA с ID 4E386D9C9C61702F
# gpg: запрашиваю ключ 4E386D9C9C61702F с сервера hkp wwwkeys.pgp.net
# gpg: ключ 4E386D9C9C61702F: импортирован открытый ключ "Willy Tarreau <w@1wt.eu>"
# gpg: требуется 3 с ограниченным доверием, 1 с полным, модель доверия classic
# gpg: глубина: 0  верных:   2  подписанных:   0  доверие: 0-, 0q, 0n, 0m, 0f, 2u
# gpg: срок следующей проверки таблицы доверия 2020-09-21
# gpg: Всего обработано: 1
# gpg:                 импортировано: 1  (RSA: 1)
# gpg: Действительная подпись от "Willy Tarreau <w@1wt.eu>"
# gpg: ВНИМАНИЕ: Данный ключ не заверен доверенной подписью!
# gpg:           Нет указаний на то, что подпись принадлежит владельцу.
# Отпечаток главного ключа: 0C95 68FA 5546 5655 1590  C5E4 4E38 6D9C 9C61 702F
2017-01-12 11:19:36 +03:00
Willy Tarreau
2c0b1b9774 [RELEASE] Released version 1.7.1
Released version 1.7.1 with the following main changes :
    - BUG/MEDIUM: proxy: return "none" and "unknown" for unknown LB algos
    - BUG/MINOR: stats: make field_str() return an empty string on NULL
    - DOC: Spelling fixes
    - BUG/MEDIUM: http: Fix tunnel mode when the CONNECT method is used
    - BUG/MINOR: http: Keep the same behavior between 1.6 and 1.7 for tunneled txn
    - BUG/MINOR: filters: Protect args in macros HAS_DATA_FILTERS and IS_DATA_FILTER
    - BUG/MINOR: filters: Invert evaluation order of HTTP_XFER_BODY and XFER_DATA analyzers
    - BUG/MINOR: http: Call XFER_DATA analyzer when HTTP txn is switched in tunnel mode
    - BUG/MAJOR: stream: fix session abort on resource shortage
    - BUG/MINOR: cli: allow the backslash to be escaped on the CLI
    - BUG/MEDIUM: cli: fix "show stat resolvers" and "show tls-keys"
    - DOC: Fix map table's format
    - DOC: Added 51Degrees conv and fetch functions to documentation.
    - BUG/MINOR: http: don't send an extra CRLF after a Set-Cookie in a redirect
    - DOC: mention that req_tot is for both frontends and backends
    - BUG/MEDIUM: variables: some variable name can hide another ones
    - BUG/MINOR: stats: fix be/sessions/max output in html stats
    - MINOR: proxy: Add fe_name/be_name fetchers next to existing fe_id/be_id
    - DOC: lua: Documentation about some entry missing
    - MINOR: Do not forward the header "Expect: 100-continue" when the option http-buffer-request is set
    - DOC: Add undocumented argument of the trace filter
    - DOC: Fix some typo in SPOE documentation
    - BUG/MINOR: cli: be sure to always warn the cli applet when input buffer is full
    - MINOR: applet: Count number of (active) applets
    - MINOR: task: Rename run_queue and run_queue_cur counters
    - BUG/MEDIUM: stream: Save unprocessed events for a stream
    - BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled
    - BUILD/MEDIUM: Fixing the build using LibreSSL
2016-12-13 00:24:34 +01:00
Luca Pizzamiglio
5d1d15e2f7 BUILD/MEDIUM: Fixing the build using LibreSSL
Fixing the build using LibreSSL as OpenSSL implementation.
Currently, LibreSSL 2.4.4 provides the same API of OpenSSL 1.0.1x,
but it redefine the OpenSSL version number as 2.0.x, breaking all
checks with OpenSSL 1.1.x.
The patch solves the issue checking the definition of the symbol
LIBRESSL_VERSION_NUMBER when Openssl 1.1.x features are requested.

(cherry picked from commit 578b169dcb653feb74d827238b879217b7220b89)
2016-12-13 00:21:29 +01:00
Christopher Faulet
68c8714e67 BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled
When an entity tries to get a buffer, if it cannot be allocted, for example
because the number of buffers which may be allocated per process is limited,
this entity is added in a list (called <buffer_wq>) and wait for an available
buffer.

Historically, the <buffer_wq> list was logically attached to streams because it
were the only entities likely to be added in it. Now, applets can also be
waiting for a free buffer. And with filters, we could imagine to have more other
entities waiting for a buffer. So it make sense to have a generic list.

Anyway, with the current design there is a bug. When an applet failed to get a
buffer, it will wait. But we add the stream attached to the applet in
<buffer_wq>, instead of the applet itself. So when a buffer is available, we
wake up the stream and not the waiting applet. So, it is possible to have
waiting applets and never awakened.

So, now, <buffer_wq> is independant from streams. And we really add the waiting
entity in <buffer_wq>. To be generic, the entity is responsible to define the
callback used to awaken it.

In addition, applets will still request an input buffer when they become
active. But they will not be sleeped anymore if no buffer are available. So this
is the responsibility to the applet I/O handler to check if this buffer is
allocated or not. This way, an applet can decide if this buffer is required or
not and can do additional processing if not.

[wt: backport to 1.7 and 1.6]
(cherry picked from commit a73e59b6901a164d19b1145e8511602d9814f28f)
2016-12-12 19:31:44 +01:00
Christopher Faulet
0e28d02d91 BUG/MEDIUM: stream: Save unprocessed events for a stream
A stream can be awakened for different reasons. During its processing, it can be
early stopped if no buffer is available. In this situation, the reason why the
stream was awakened is lost, because we rely on the task state, which is reset
after each processing loop.

In many cases, that's not a big deal. But it can be useful to accumulate the
task states if the stream processing is interrupted, especially if some filters
need to be called.

To be clearer, here is an simple example:

  1) A stream is awakened with the reason TASK_WOKEN_MSG.

  2) Because no buffer is available, the processing is interrupted, the stream
  is back to sleep. And the task state is reset.

  3) Some buffers become available, so the stream is awakened with the reason
  TASK_WOKEN_RES. At this step, the previous reason (TASK_WOKEN_MSG) is lost.

Now, the task states are saved for a stream and reset only when the stream
processing is not interrupted. The correspoing bitfield represents the pending
events for a stream. And we use this one instead of the task state during the
stream processing.

Note that TASK_WOKEN_TIMER and TASK_WOKEN_RES are always removed because these
events are always handled during the stream processing.

[wt: backport to 1.7 and 1.6]
(cherry picked from commit 9d810cae111991f1cd38d6ad37b4046c508884b8)
2016-12-12 19:31:44 +01:00
Christopher Faulet
0191d37548 MINOR: task: Rename run_queue and run_queue_cur counters
<run_queue> is used to track the number of task in the run queue and
<run_queue_cur> is a copy used for the reporting purpose. These counters has
been renamed, respectively, <tasks_run_queue> and <tasks_run_queue_cur>. So the
naming is consistent between tasks and applets.

[wt: needed for next fixes, backport to 1.7 and 1.6]
(cherry picked from commit 34c5cc98da7cad2c7e0733a7b86c8369d1dea5bd)
2016-12-12 19:31:44 +01:00
Christopher Faulet
0558590c72 MINOR: applet: Count number of (active) applets
As for tasks, 2 counters has been added to track :
  * the total number of applets : nb_applets
  * the number of active applets : applets_active_queue

[wt: needed for next fixes, to backport to 1.7 and 1.6]
(cherry picked from commit 1cbe74cd837b460d59b31a534d20f1ffd980b8f0)
2016-12-12 19:31:44 +01:00
Christopher Faulet
1233e1e201 BUG/MINOR: cli: be sure to always warn the cli applet when input buffer is full
[wt: may only strike if CLI commands are pipelined. Must be backported to 1.7
 and 1.6, where it's a bit different and in dumpstats.c]
(cherry picked from commit 90b5abe46ea507ae7831221555f5183d16864d4a)
2016-12-12 19:31:44 +01:00
Christopher Faulet
dda09d6825 DOC: Fix some typo in SPOE documentation
[wt: backport to 1.7]
(cherry picked from commit 1002aac161df5d418d18f8b16f38dfbcd2fa71a7)
2016-12-12 19:31:44 +01:00
Christopher Faulet
78cb5bff53 DOC: Add undocumented argument of the trace filter
[wt: to backport to 1.7]
(cherry picked from commit 31bfe1f262c218574686ffdabef19cb40ae65ac3)
2016-12-12 19:31:44 +01:00
Thierry FOURNIER / OZON.IO
ef1f7392b0 MINOR: Do not forward the header "Expect: 100-continue" when the option http-buffer-request is set
When the option "http-buffer-request" is set, HAProxy send itself the
"HTTP/1.1 100 Continue" response in order to retrieve the post content.
When HAProxy forward the request, it send the body directly after the
headers. The header "Expect: 100-continue" was sent with the headers.
This header is useless because the body will be sent in all cases, and
the server reponse is not removed by haproxy.

This patch removes the header "Expect: 100-continue" if HAProxy sent it
itself.
(cherry picked from commit 43ad11dc75d2f6513f1d884c031e789b419b3b72)
2016-12-12 19:31:43 +01:00
Thierry FOURNIER / OZON.IO
4597d88bc3 DOC: lua: Documentation about some entry missing
The parameter "value" of the function TXN.set_var() was not documented.

This is a regression from the commit 85d79c94a93211b156ec644e164b178f5d7101fa.
This patch must be backported in 1.7
(cherry picked from commit b210bcc559ffcbe691e9bd0eb9e0c91a94574045)
2016-12-12 19:31:43 +01:00
Marcin Deranek
2693f623c6 MINOR: proxy: Add fe_name/be_name fetchers next to existing fe_id/be_id
These 2 patches add ability to fetch frontend/backend name in your
logic, so they can be used later to make routing decisions (fe_name) or
taking some actions based on backend which responded to request (be_name).
In our case we needed a fetcher to be able to extract information we
needed from frontend name.
(cherry picked from commit d2471c2bdc98e2c014624ecff9747528397aee57)
2016-12-12 15:17:21 +01:00
Willy Tarreau
0f5e0dcb31 BUG/MINOR: stats: fix be/sessions/max output in html stats
"Tadas / XtGem" reported that the max value was wrong and would report
the current value instead. This needs to be backported to 1.7.
(cherry picked from commit 8e0f17543e4d6d5b1e8b87e05341d0d9fb53c203)
2016-12-12 15:17:21 +01:00
Thierry FOURNIER / OZON.IO
484855eef0 BUG/MEDIUM: variables: some variable name can hide another ones
The variable are compared only using text, the final '\0' (or the
string length) are not checked. So, the variable name "txn.internal"
matchs other one call "txn.int".

This patch fix this behavior

It must be backported ni 1.6 and 1.7
(cherry picked from commit d2f6f47597844e07c393bf9224c99bc31538512c)
2016-12-12 15:17:21 +01:00
Willy Tarreau
28c38fbf0a DOC: mention that req_tot is for both frontends and backends
Thierry reported that only frontends were mentionned while the stats
are for both. To be backported to 1.7.
(cherry picked from commit fb981bd7d1ca8c23097b0e70f3d4169820abd04e)
2016-12-12 15:16:11 +01:00
Matthieu Guegan
4971a0416c BUG/MINOR: http: don't send an extra CRLF after a Set-Cookie in a redirect
By investigating a keep-alive issue with CloudFlare, we[1] found that
when using the 'set-cookie' option in a redirect (302) HAproxy is adding
an extra `\r\n`.

Triggering rule :

`http-request redirect location / set-cookie Cookie=value if [...]`

Expected result :

```
HTTP/1.1 302 Found
Cache-Control: no-cache
Content-length: 0
Location: /
Set-Cookie: Cookie=value; path=/;
Connection: close
```

Actual result :

```
HTTP/1.1 302 Found
Cache-Control: no-cache
Content-length: 0
Location: /
Set-Cookie: Cookie=value; path=/;

Connection: close
```

This extra `\r\n` seems to be harmless with another HAproxy instance in
front of it (sanitizing) or when using a browser. But we confirm that
the CloudFlare NGINX implementation is not able to handle this. It
seems that both 'Content-length: 0' and extra carriage return broke RFC
(to be confirmed).

When looking into the code, this carriage-return was already present in
1.3.X versions but just before closing the connection which was ok I
think. Then, with 1.4.X the keep-alive feature was added and this piece
of code remains unchanged.

[1] all credit for the bug finding goes to CloudFlare Support Team

[wt: the bug was indeed present since the Set-Cookie was introduced
 in 1.3.16, by commit 0140f25 ("[MINOR] redirect: add support for
 "set-cookie" and "clear-cookie"") so backporting to all supported
 versions is desired]
(cherry picked from commit 35088f960de9e3331573d118adbbe956c7fbbf7d)
2016-12-08 14:56:42 +01:00
Ben Shillito
732912efdf DOC: Added 51Degrees conv and fetch functions to documentation.
Definitions and examples for 51d.single and 51d.all have been added to
configuration.txt so it now appears in online documentation in addition
to the README, The 51degrees-property-name-list entry has also been
updated to make it clear that multiple properties can be added.
(cherry picked from commit f25e8e56b4c591df1b2dfb2421fdcd94de31c41f)
2016-12-08 14:56:42 +01:00
Ruoshan Huang
0c0002b215 DOC: Fix map table's format
so the HTML doc parser can parse the table correctly
(cherry picked from commit 3c5e374142b173d7309f18220f126b11de9c2018)
2016-12-08 14:56:41 +01:00
Willy Tarreau
4764f040ca BUG/MEDIUM: cli: fix "show stat resolvers" and "show tls-keys"
The recent CLI reorganization managed to break these two commands
by having their parser return 1 (indicating an end of processing)
instead of 0 to indicate new calls to the io handler were needed.

Namely the faulty commits are :
  69e9644 ("REORG: cli: move show stat resolvers to dns.c")
  32af203 ("REORG: cli: move ssl CLI functions to ssl_sock.c")

The fix is trivial and there is no other loss of functionality. Thanks
to Dragan Dosen for reporting the issue and the faulty commits. The
backport is needed in 1.7.
(cherry picked from commit 3067bfa815afce9217c5f7cf0812e2575583a2fa)
2016-12-08 14:56:41 +01:00
Dragan Dosen
5dc0356aad BUG/MINOR: cli: allow the backslash to be escaped on the CLI
In 1.5-dev20, commit 48bcfda ("MEDIUM: dumpstat: make the CLI parser
understand the backslash as an escape char") introduced support for
backslash on the CLI, but it strips all backslashes in all arguments
instead of only unescaping them, making it impossible to pass a
backslash in an argument.

This will allow us to use a backslash in a command over the socket, eg.
"add acl #0 ABC\\XYZ".

[wt: this should be backported to 1.7 and 1.6]
(cherry picked from commit a1c35ab68d59f877b54dab7f296d3fdd6edff6e9)
2016-12-08 14:56:41 +01:00
Willy Tarreau
e7012e611f BUG/MAJOR: stream: fix session abort on resource shortage
In 1.6-dev2, commit 32990b5 ("MEDIUM: session: remove the task pointer
from the session") introduced a bug which can sometimes crash the process
on resource shortage. When stream_complete() returns -1, it has already
reattached the connection to the stream, then kill_mini_session() is
called and still expects to find the task in conn->owner. Note that
since this commit, the code has moved a bit and is now in stream_new()
but the problem remains the same.

Given that we already know the task around these places, let's simply
pass the task to kill_mini_session().

The conditions currently at risk are :
  - failure to initialize filters for the new stream (lack of memory or
    any filter returning < 0 on attach())
  - failure to attach filters (any filter returning < 0 on stream_start())
  - frontend's accept() returning < 0 (allocation failure)

This fix is needed in 1.7 and 1.6.
(cherry picked from commit 92b10c954defc82ceabdc9bba923320a157c664d)
2016-12-08 14:56:41 +01:00
Christopher Faulet
5961fb820a BUG/MINOR: http: Call XFER_DATA analyzer when HTTP txn is switched in tunnel mode
This allow a filter to start to analyze data in HTTP and to fallback in TCP when
data are tunneled.

[wt: backport desired in 1.7 - no impact right now but may impact the ability
 to backport future fixes]
(cherry picked from commit 6962f4e0d6dd1f1e976f815c1c8afaabfadc93fa)
2016-11-29 18:20:27 +01:00
Christopher Faulet
7ce81ff112 BUG/MINOR: filters: Invert evaluation order of HTTP_XFER_BODY and XFER_DATA analyzers
These 2 analyzers are responsible of the data forwarding in, respectively, HTTP
mode and TCP mode. Now, the analyzer responsible of the HTTP data forwarding is
called before the one responsible of the TCP data forwarding. This will allow
the filtering of tunneled data in HTTP.

[wt: backport desired in 1.7 - no impact right now but may impact the ability
 to backport future fixes]
(cherry picked from commit d47a1bd1d7311f70c368ccb016553a0d30fb123d)
2016-11-29 18:20:27 +01:00
Christopher Faulet
620469c129 BUG/MINOR: filters: Protect args in macros HAS_DATA_FILTERS and IS_DATA_FILTER
[wt: backport needed in 1.7]
(cherry picked from commit 4aad833924d9462ec8b00d26c9a831ebd4dbde14)
2016-11-29 18:20:27 +01:00
Christopher Faulet
79cbccf190 BUG/MINOR: http: Keep the same behavior between 1.6 and 1.7 for tunneled txn
In HAProxy 1.6, When "http-tunnel" option is enabled, HTTP transactions are
tunneled as soon as possible after the headers parsing/forwarding. When the
transfer length of the response can be determined, this happens when all data
are forwarded. But for responses with an undetermined transfer length this
happens when headers are forwarded. This behavior is questionable, but this is
not the purpose of this fix...

In HAProxy 1.7, the first use-case works like in 1.6. But the second one not
because of the data filtering. HAProxy was always trying to forward data until
the server closes the connection. So the transaction was never switched in
tunnel mode. This is the expected behavior when there is a data filter. But in
the default case (no data filter), it should work like in 1.6.

This patch fixes the bug. We analyze response data until the server closes the
connection only when there is a data filter.

[wt: backport needed in 1.7]
(cherry picked from commit 3235957685f2eb4797a302a773a0f8776f068463)
2016-11-29 18:20:27 +01:00
Christopher Faulet
178dcd86f0 BUG/MEDIUM: http: Fix tunnel mode when the CONNECT method is used
When a 2xx response to a CONNECT request is returned, the connection must be
switched in tunnel mode immediatly after the headers, and Transfer-Encoding and
Content-Length headers must be ignored. So from the HTTP parser point of view,
there is no body.

The bug comes from the fact the flag HTTP_MSGF_XFER_LEN was not set on the
response (This flag means that the body size can be determined. In our case, it
can, it is 0). So, during data forwarding, the connection was never switched in
tunnel mode and we were blocked in a state where we were waiting that the
server closes the connection to ends the response.

Setting the flag HTTP_MSGF_XFER_LEN on the response fixed the bug.

The code of http_wait_for_response has been slightly updated to be more
readable.

[wt: 1.7-only, this is not needed in 1.6]
(cherry picked from commit d1cd209b21865c69761e4d5005a9f36423a38479)
2016-11-29 18:20:27 +01:00
Tim Düsterhus
b52dd88e7f DOC: Spelling fixes
[wt: this contains spelling fixes for both doc and code comments,
 should be backported, ignoring the parts which don't apply]
(cherry picked from commit 4896c440b3e270a9acd41d2692ecc8bf5d7636fa)
2016-11-29 18:20:27 +01:00
Willy Tarreau
7de2aece6b BUG/MINOR: stats: make field_str() return an empty string on NULL
It already returns an empty string when the field is empty, but as a
preventive measure we should do the same when the string itself is a
NULL. While it is not supposed to happen, it will make the code more
resistant against failed allocations and unexpected results.

This fix should be backported to 1.7.

(cherry picked from commit ddc3e9e55de965432946c7d13c52756f71eb06b5)
2016-11-26 15:59:14 +01:00
Willy Tarreau
1b8659fa8e BUG/MEDIUM: proxy: return "none" and "unknown" for unknown LB algos
When a backend doesn't use any known LB algorithm, backend_lb_algo_str()
returns NULL. It used to cause "nil" to be printed in the stats dump
since version 1.4 but causes 1.7 to try to parse this NULL to encode
it as a CSV string, causing a crash on "show stat" in this case.

The only situation where this can happen is when "transparent" or
"dispatch" are used in a proxy, in which case the LB algorithm is
BE_LB_ALGO_NONE. Thus now we explicitly report "none" when this
situation is detected, and we preventively report "unknown" if any
unknown algorithm is detected, which may happen if such an algo is
added in the future and the function is not updated.

This fix must be backported to 1.7 and may be backported as far as
1.4, though it has less impact there.

(cherry picked from commit b3e111b4fd99137b6b8c4d687282a497ed082111)
2016-11-26 15:59:14 +01:00
Willy Tarreau
e59fcdd25e [RELEASE] Released version 1.7.0
Released version 1.7.0 with the following main changes :
    - SCRIPTS: make publish-release also copy the new SPOE doc
    - BUILD: http: include types/sample.h in proto_http.h
    - BUILD: debug/flags: remove test for SF_COMP_READY
    - CONTRIB: debug/flags: add check for SF_ERR_CHK_PORT
    - MINOR: lua: add function which return true if the channel is full.
    - MINOR: lua: add ip addresses and network manipulation function
    - CONTRIB: tcploop: scriptable TCP I/O for debugging purposes
    - CONTRIB: tcploop: implement fork()
    - CONTRIB: tcploop: implement logging when called with -v
    - CONTRIB: tcploop: update the usage output
    - CONTRIB: tcploop: support sending plain strings
    - CONTRIB: tcploop: don't report failed send() or recv()
    - CONTRIB: tcploop: add basic loops via a jump instruction
    - BUG/MEDIUM: channel: bad unlikely macro
    - CLEANUP: lua: move comment
    - CLEANUP: lua: control executed twice
    - BUG/MEDIUM: ssl: Store certificate filename in a variable
    - BUG/MINOR: ssl: Print correct filename when error occurs reading OCSP
    - CLEANUP: ssl: Remove goto after return dead code
    - CLEANUP: ssl: Fix bind keywords name in comments
    - DOC: ssl: Use correct wording for ca-sign-pass
    - CLEANUP: lua: avoid directly calling getsockname/getpeername()
    - BUG/MINOR: stick-table: handle out-of-memory condition gracefully
    - MINOR: cli: add private pointer and release function
    - MEDIUM: lua: Add cli handler for Lua
    - BUG/MEDIUM: connection: check the control layer before stopping polling
    - DEBUG: connection: mark the closed FDs with a value that is easier to detect
    - BUG/MEDIUM: stick-table: fix regression caused by recent fix for out-of-memory
    - BUG/MINOR: cli: properly decrement ref count on tables during failed dumps
    - BUG/MEDIUM: lua: In some case, the return of sample-fetche is ignored
    - MINOR: filters: Add check_timeouts callback to handle timers expiration on streams
    - MINOR: spoe: Add 'timeout processing' option to limit time to process an event
    - MINOR: spoe: Remove useless 'timeout ack' option
    - MINOR: spoe: Add 'option continue-on-error' statement in spoe-agent section
    - MINOR: spoe: Add "maxconnrate" and "maxerrrate" statements
    - MINOR: spoe: Add "option set-on-error" statement
    - MINOR: stats: correct documentation of process ID for typed output
    - BUILD: contrib: fix ip6range build on Centos 7
    - BUILD: fix build on Solaris 10/11
    - BUG/MINOR: cli: fix pointer size when reporting data/transport layer name
    - BUG/MINOR: cli: dequeue from the proxy when changing a maxconn
    - BUG/MINOR: cli: wake up the CLI's task after a timeout update
    - MINOR: connection: add a few functions to report the data and xprt layers' names
    - MINOR: connection: add names for transport and data layers
    - REORG: cli: split dumpstats.c in src/cli.c and src/stats.c
    - REORG: cli: split dumpstats.h in stats.h and cli.h
    - REORG: cli: move ssl CLI functions to ssl_sock.c
    - REORG: cli: move map and acl code to map.c
    - REORG: cli: move show stat resolvers to dns.c
    - MINOR: cli: create new function cli_has_level() to validate permissions
    - MINOR: server: create new function cli_find_server() to find a server
    - MINOR: proxy: create new function cli_find_frontend() to find a frontend
    - REORG: cli: move 'set server' to server.c
    - REORG: cli: move 'show pools' to memory.c
    - REORG: cli: move 'show servers' to proxy.c
    - REORG: cli: move 'show sess' to stream.c
    - REORG: cli: move 'show backend' to proxy.c
    - REORG: cli: move get/set weight to server.c
    - REORG: cli: move "show stat" to stats.c
    - REORG: cli: move "show info" to stats.c
    - REORG: cli: move dump_text(), dump_text_line(), and dump_binary() to standard.c
    - REORG: cli: move table dump/clear/set to stick_table.c
    - REORG: cli: move "show errors" out of cli.c
    - REORG: cli: make "show env" also use the generic keyword registration
    - REORG: cli: move "set timeout" to its own handler
    - REORG: cli: move "clear counters" to stats.c
    - REORG: cli: move "set maxconn global" to its own handler
    - REORG: cli: move "set maxconn server" to server.c
    - REORG: cli: move "set maxconn frontend" to proxy.c
    - REORG: cli: move "shutdown sessions server" to stream.c
    - REORG: cli: move "shutdown session" to stream.c
    - REORG: cli: move "shutdown frontend" to proxy.c
    - REORG: cli: move "{enable|disable} frontend" to proxy.c
    - REORG: cli: move "{enable|disable} server" to server.c
    - REORG: cli: move "{enable|disable} health" to server.c
    - REORG: cli: move "{enable|disable} agent" to server.c
    - REORG: cli: move the "set rate-limit" functions to their own parser
    - CLEANUP: cli: rename STAT_CLI_* to CLI_ST_*
    - CLEANUP: cli: simplify the request parser a little bit
    - CLEANUP: cli: remove assignments to st0 and st2 in keyword parsers
    - BUILD: server: remove a build warning introduced by latest series
    - BUG/MINOR: log-format: uncatched memory allocation functions
    - CLEANUP: log-format: useless file and line in json converter
    - CLEANUP/MINOR: log-format: unexport functions parse_logformat_var_args() and parse_logformat_var()
    - CLEANUP: log-format: fix return code of the function parse_logformat_var()
    - CLEANUP: log-format: fix return code of function parse_logformat_var_args()
    - CLEANUP: log-format: remove unused arguments
    - MEDIUM: log-format: strict parsing and enable fail
    - MEDIUM: log-format/conf: take into account the parse_logformat_string() return code
    - BUILD: ssl: make the SSL layer build again with openssl 0.9.8
    - BUILD: vars: remove a build warning on vars.c
    - MINOR: lua: add utility function for check boolean argument
    - MINOR: lua: Add tokenize function.
    - BUG/MINOR: conf: calloc untested
    - MINOR: http/conf: store the use_backend configuration file and line for logs
    - MEDIUM: log-format: Use standard HAProxy log system to report errors
    - CLEANUP: sample: report "converter" instead of "conv method" in error messages
    - BUG: spoe: Fix parsing of SPOE actions in ACK frames
    - MINOR: cli: make "show stat" support a proxy name
    - MINOR: cli: make "show errors" support a proxy name
    - MINOR: cli: make "show errors" capable of dumping only request or response
    - BUG/MINOR: freq-ctr: make swrate_add() support larger values
    - CLEANUP: counters: move from 3 types to 2 types
    - CLEANUP: cfgparse: cascade the warnif_misplaced_* rules
    - REORG: tcp-rules: move tcp rules processing to their own file
    - REORG: stkctr: move all the stick counters processing to stick-tables.c
    - DOC: update the roadmap file with the latest changes
2016-11-25 16:39:17 +01:00
Willy Tarreau
e2d4ff4579 DOC: update the roadmap file with the latest changes
A few points were already addressed. Mention that next changes are
for version 1.8 now.
2016-11-25 16:32:20 +01:00
Willy Tarreau
7d56221d57 REORG: stkctr: move all the stick counters processing to stick-tables.c
Historically we used to have the stick counters processing put into
session.c which became stream.c. But a big part of it is now in
stick-table.c (eg: converters) but despite this we still have all
the sample fetch functions in stream.c

These parts do not depend on the stream anymore, so let's move the
remaining chunks to stick-table.c and have cleaner files.

What remains in stream.c is everything needed to attach/detach
trackers to the stream and to update the counters while the stream
is being processed.
2016-11-25 16:10:05 +01:00
Willy Tarreau
397131093f REORG: tcp-rules: move tcp rules processing to their own file
There's no more reason to keep tcp rules processing inside proto_tcp.c
given that there is nothing in common there except these 3 letters : tcp.
The tcp rules are in fact connection, session and content processing rules.
Let's move them to "tcp-rules" and let them live their life there.
2016-11-25 15:57:38 +01:00
Willy Tarreau
d39ad449b9 CLEANUP: cfgparse: cascade the warnif_misplaced_* rules
There are 8 functions each repeating what another does and adding one
extra test. We used to have some copy-paste issues in the past due to
this. Instead we now make them simply rely on the previous one and add
the final test. It's much better and much safer. The functions could
be moved to inlines but they're used at a few other locations only,
it didn't make much sense in the end.
2016-11-25 15:16:12 +01:00
Willy Tarreau
ae9bea0591 CLEANUP: counters: move from 3 types to 2 types
We used to have 3 types of counters with a huge overlap :
  - listener counters : stats collected for each bind line
  - proxy counters : union of the frontend and backend counters
  - server counters : stats collected per server

It happens that quite a good part was common between listeners and
proxies due to the frontend counters being updated at the two locations,
and that similarly the server and proxy counters were overlapping and
being updated together.

This patch cleans this up to propose only two types of counters :
  - fe_counters: used by frontends and listeners, related to
    incoming connections activity
  - be_counters: used by backends and servers, related to outgoing
    connections activity

This allowed to remove some non-sensical counters from both parts. For
frontends, the following entries were removed :

  cum_lbconn, last_sess, nbpend_max, failed_conns, failed_resp,
  retries, redispatches, q_time, c_time, d_time, t_time

For backends, this ones was removed : intercepted_req.

While doing this it was discovered that we used to incorrectly report
intercepted_req for backends in the HTML stats, which was always zero
since it's never updated.

Also it revealed a few inconsistencies (which were not fixed as they
are harmless). For example, backends count connections (cum_conn)
instead of sessions while servers count sessions and not connections.

Over the long term, some extra cleanups may be performed by having
some counters update functions touching both the server and backend
at the same time, as well as both the frontend and listener, to
ensure that all sides have all their stats properly filled. The stats
dump will also be able to factor the dump functions by counter types.
2016-11-25 15:03:12 +01:00
Willy Tarreau
3758581e19 BUG/MINOR: freq-ctr: make swrate_add() support larger values
Reinhard Vicinus reported that the reported average response times cannot
be larger than 16s due to the double multiply being performed by
swrate_add() which causes an overflow very quickly. Indeed, with N=512,
the highest average value is 16448.

One solution proposed by Reinhard is to turn to long long, but this
involves 64x64 multiplies and 64->32 divides, which are extremely
expensive on 32-bit platforms.

There is in fact another way to avoid the overflow without using larger
integers, it consists in avoiding the multiply using the fact that
x*(n-1)/N = x-(x/N).

Now it becomes possible to store average values as large as 8.4 millions,
which is around 2h18mn.

Interestingly, this improvement also makes the code cheaper to execute
both on 32 and on 64 bit platforms :

Before :

00000000 <swrate_add>:
   0:   8b 54 24 04             mov    0x4(%esp),%edx
   4:   8b 0a                   mov    (%edx),%ecx
   6:   89 c8                   mov    %ecx,%eax
   8:   c1 e0 09                shl    $0x9,%eax
   b:   29 c8                   sub    %ecx,%eax
   d:   8b 4c 24 0c             mov    0xc(%esp),%ecx
  11:   c1 e8 09                shr    $0x9,%eax
  14:   01 c8                   add    %ecx,%eax
  16:   89 02                   mov    %eax,(%edx)

After :

00000020 <swrate_add>:
  20:   8b 4c 24 04             mov    0x4(%esp),%ecx
  24:   8b 44 24 0c             mov    0xc(%esp),%eax
  28:   8b 11                   mov    (%ecx),%edx
  2a:   01 d0                   add    %edx,%eax
  2c:   81 c2 ff 01 00 00       add    $0x1ff,%edx
  32:   c1 ea 09                shr    $0x9,%edx
  35:   29 d0                   sub    %edx,%eax
  37:   89 01                   mov    %eax,(%ecx)

This fix may be backported to 1.6.
2016-11-25 11:55:10 +01:00
Willy Tarreau
35069f84af MINOR: cli: make "show errors" capable of dumping only request or response
When dealing with many proxies, it's hard to spot response errors because
all internet-facing frontends constantly receive attacks. This patch now
makes it possible to demand that only request or response errors are dumped
by appending "request" or "reponse" to the show errors command.
2016-11-25 09:16:37 +01:00
Willy Tarreau
234ba2d8eb MINOR: cli: make "show errors" support a proxy name
Till now it was needed to know the proxy's ID while we do have the
ability to look up a proxy by its name now.
2016-11-25 08:56:55 +01:00
Willy Tarreau
a1b1ed53e7 MINOR: cli: make "show stat" support a proxy name
Till now it was needed to know the proxy's ID while we do have the
ability to look up a proxy by its name now.
2016-11-25 08:55:25 +01:00
Christopher Faulet
b5cff60ef5 BUG: spoe: Fix parsing of SPOE actions in ACK frames
For "SET-VAR" actions, data was not correctly parsed. 'idx' variable was not
correctly updated when the 3rd argument was parsed.
2016-11-25 08:09:10 +01:00
Willy Tarreau
97108e08ce CLEANUP: sample: report "converter" instead of "conv method" in error messages
This was inherited from the very early stick-tables code but it's about
time to produce understandable error messages :-)
2016-11-25 07:36:22 +01:00
Thierry FOURNIER / OZON.IO
8a4e4420fb MEDIUM: log-format: Use standard HAProxy log system to report errors
The function log format emit its own error message using Alert(). This
patch replaces this behavior and uses the standard HAProxy error system
(with memprintf).

The benefits are:
 - cleaning the log system

 - the logformat can ignore the caller (actually the caller must set
   a flag designing the caller function).

 - Make the usage of the logformat function easy for future components.
2016-11-25 07:32:58 +01:00
Thierry FOURNIER / OZON.IO
4ed1c9585d MINOR: http/conf: store the use_backend configuration file and line for logs
The error log of the directive use_backend doesn't provide the
file and line containing the declaration. This patch stores
theses informations.
2016-11-25 07:15:09 +01:00
Thierry FOURNIER / OZON.IO
5948b01149 BUG/MINOR: conf: calloc untested
A calloc is executed without check of its returns code.
2016-11-25 07:15:06 +01:00
Thierry FOURNIER / OZON.IO
8a1027aa45 MINOR: lua: Add tokenize function.
For tokenizing a string, standard Lua recommends to use regexes.
The followinf example splits words:

   for i in string.gmatch(example, "%S+") do
      print(i)
   end

This is a little bit overkill for simply split words. This patch
adds a tokenize function which quick and do not use regexes.
2016-11-24 21:35:34 +01:00
Thierry FOURNIER / OZON.IO
7f3aa8b62f MINOR: lua: add utility function for check boolean argument
Strangely, the Lua API doesn't provides a function like
luaL_checkboolean(). This little function add this one.
2016-11-24 21:35:10 +01:00
Willy Tarreau
e365815007 BUILD: vars: remove a build warning on vars.c
gcc 3.4.6 noticed a possibly unitialized variable in vars.c, and while it
cannot happen the way the function is used, it's surprizing that newer
versions did not report it.

This fix may be backported to 1.6.
2016-11-24 21:25:43 +01:00
Willy Tarreau
80ebacfc82 BUILD: ssl: make the SSL layer build again with openssl 0.9.8
Commit 1866d6d ("MEDIUM: ssl: Add support for OpenSSL 1.1.0")
introduced support for openssl 1.1.0 and temporarily broke 0.9.8.
In the end the port was not very hard given that the only cause of
build failures were functions supposedly absent from 0.9.8 that in
fact did exist.

Thus, adding a new #if to move these functions for versions older
than 0.9.8 was enough to fix the trouble. It received very light
testing, basically only an SSL bridge decrypting and re-encrypting
traffic, and checking that everything looks right. That said, the
functions specific to 0.9.8 here compared to 1.0.x are only
SSL_SESSION_set1_id_context(), EVP_PKEY_base_id(), and
X509_PUBKEY_get0_param().
2016-11-24 20:18:21 +01:00