sinjuginas/haproxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
haproxy/src/applet.c

117 lines
3.7 KiB
C
Raw Normal View History

MEDIUM: applet: add basic support for an applet run queue This will be needed so that we can schedule applets out of the streams. For now nothing calls the queue yet.
2015-04-13 17:11:11 +02:00
/*
* Functions managing applets
*
* Copyright 2000-2015 Willy Tarreau <w@1wt.eu>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*
*/
#include <stdio.h>
#include <stdlib.h>
REORG: include: update all files to use haproxy/api.h or api-t.h if needed All files that were including one of the following include files have been updated to only include haproxy/api.h or haproxy/api-t.h once instead: - common/config.h - common/compat.h - common/compiler.h - common/defaults.h - common/initcall.h - common/tools.h The choice is simple: if the file only requires type definitions, it includes api-t.h, otherwise it includes the full api.h. In addition, in these files, explicit includes for inttypes.h and limits.h were dropped since these are now covered by api.h and api-t.h. No other change was performed, given that this patch is large and affects 201 files. At least one (tools.h) was already freestanding and didn't get the new one added.
2020-05-27 12:58:42 +02:00
#include <haproxy/api.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 09:07:15 +02:00
#include <haproxy/applet.h>
REORG: include: move channel.h to haproxy/channel{,-t}.h The files were moved with no change. The callers were cleaned up a bit and a few of them had channel.h removed since not needed.
2020-06-04 21:07:02 +02:00
#include <haproxy/channel.h>
REORG: include: split mini-clist into haproxy/list and list-t.h Half of the users of this include only need the type definitions and not the manipulation macros nor the inline functions. Moves the various types into mini-clist-t.h makes the files cleaner. The other one had all its includes grouped at the top. A few files continued to reference it without using it and were cleaned. In addition it was about time that we'd rename that file, it's not "mini" anymore and contains a bit more than just circular lists.
2020-05-27 18:01:47 +02:00
#include <haproxy/list.h>
REORG: include: move stream.h to haproxy/stream{,-t}.h This one was not easy because it was embarking many includes with it, which other files would automatically find. At least global.h, arg.h and tools.h were identified. 93 total locations were identified, 8 additional includes had to be added. In the rare files where it was possible to finalize the sorting of includes by adjusting only one or two extra lines, it was done. But all files would need to be rechecked and cleaned up now. It was the last set of files in types/ and proto/ and these directories must not be reused anymore.
2020-06-04 23:46:14 +02:00
#include <haproxy/stream.h>
REORG: include: move stream_interface.h to haproxy/stream_interface{,-t}.h Almost no changes, removed stdlib and added buf-t and connection-t to the types to avoid a warning.
2020-06-04 20:45:39 +02:00
#include <haproxy/stream_interface.h>
REORG: include: move task.h to haproxy/task{,-t}.h The TASK_IS_TASKLET() macro was moved to the proto file instead of the type one. The proto part was a bit reordered to remove a number of ugly forward declaration of static inline functions. About a tens of C and H files had their dependency dropped since they were not using anything from task.h.
2020-06-04 17:25:40 +02:00
#include <haproxy/task.h>
MEDIUM: applet: add basic support for an applet run queue This will be needed so that we can schedule applets out of the streams. For now nothing calls the queue yet.
2015-04-13 17:11:11 +02:00
MINOR: applet: Count number of (active) applets As for tasks, 2 counters has been added to track : * the total number of applets : nb_applets * the number of active applets : applets_active_queue [wt: needed for next fixes, to backport to 1.7 and 1.6]
2016-12-06 09:13:22 +01:00
unsigned int nb_applets = 0;
MAJOR: threads/applet: Handle multithreading for applets A global lock has been added to protect accesses to the list of active applets. A process mask has also been added on each applet. Like for FDs and tasks, it is used to know which threads are allowed to process an applet. Because applets are, most of time, linked to a session, it should be sticky on the same thread. But in all cases, it is the responsibility of the applet handler to lock what have to be protected in the applet context.
2017-06-19 12:38:55 +02:00
MINOR: applet: make appctx use their own pool A long time ago, applets were seen as an alternative to connections, and since their respective sizes were roughly equal it appeared wise to share the same pool. Nowadays, connections got significantly larger but applets are not that often used, except for the cache. However applets are mostly complementary and not alternatives anymore, as it's very possible not to have a back connection or to share one with other streams. The connections will soon lose their addresses and their size will shrink so much that appctx won't fit anymore. Given that the old benefits of sharing these pools have long disappeared, let's stop doing this and have a dedicated pool for appctx.
2019-07-18 10:41:36 +02:00
DECLARE_POOL(pool_head_appctx, "appctx", sizeof(struct appctx));
MEDIUM: appctx: check for allocation attempts in buffer allocation callbacks The buffer allocation callback appctx_res_wakeup() used to rely on old tricks to detect if a buffer was already granted to an appctx, namely by checking the task's state. Not only this test is not valid anymore, but it's inaccurate. Let's solely on SI_FL_WAIT_ROOM that is now set on allocation failure by the functions trying to allocate a buffer. The buffer is now allocated on the fly and the flag removed so that the consistency between the two remains granted. The patch also fixes minor issues such as the function being improperly declared inline(!) and the fact that using appctx_wakeup() sets the wakeup reason to TASK_WOKEN_OTHER while we try to use TASK_WOKEN_RES when waking up consecutive to a ressource allocation such as a buffer.
2018-11-06 17:32:37 +01:00
/* Callback used to wake up an applet when a buffer is available. The applet
* <appctx> is woken up if an input buffer was requested for the associated
* stream interface. In this case the buffer is immediately allocated and the
* function returns 1. Otherwise it returns 0. Note that this automatically
* covers multiple wake-up attempts by ensuring that the same buffer will not
* be accounted for multiple times.
*/
int appctx_buf_available(void *arg)
{
struct appctx *appctx = arg;
MEDIUM: applet: Set the conn-stream as appctx owner instead of the stream-int Because appctx is now an endpoint of the conn-stream, there is no reason to still have the stream-interface as appctx owner. Thus, the conn-stream is now the appctx owner.
2021-12-20 17:09:39 +01:00
struct stream_interface *si = cs_si(appctx->owner);
MEDIUM: appctx: check for allocation attempts in buffer allocation callbacks The buffer allocation callback appctx_res_wakeup() used to rely on old tricks to detect if a buffer was already granted to an appctx, namely by checking the task's state. Not only this test is not valid anymore, but it's inaccurate. Let's solely on SI_FL_WAIT_ROOM that is now set on allocation failure by the functions trying to allocate a buffer. The buffer is now allocated on the fly and the flag removed so that the consistency between the two remains granted. The patch also fixes minor issues such as the function being improperly declared inline(!) and the fact that using appctx_wakeup() sets the wakeup reason to TASK_WOKEN_OTHER while we try to use TASK_WOKEN_RES when waking up consecutive to a ressource allocation such as a buffer.
2018-11-06 17:32:37 +01:00
/* allocation requested ? */
MEDIUM: stream-int: use si_rx_buff_{rdy,blk} to report buffer readiness The stream interface used to conflate a missing buffer and lack of buffer space into SI_FL_WAIT_ROOM but this causes difficulties as these cannot be checked at the same moment and are not resolved at the same moment either. Now we instead mark the buffer as presumably available using si_rx_buff_rdy() and mark it as unavailable+requested using si_rx_buff_blk(). The call to si_alloc_buf() was moved after si_stop_put(). This makes sure that the SI_FL_RX_WAIT_EP flag is cleared on allocation failure so that the function is called again if the callee fails to do its work.
2018-11-14 15:12:08 +01:00
if (!(si->flags & SI_FL_RXBLK_BUFF))
return 0;
si_rx_buff_rdy(si);
/* was already allocated another way ? if so, don't take this one */
if (c_size(si_ic(si)) || si_ic(si)->pipe)
MEDIUM: appctx: check for allocation attempts in buffer allocation callbacks The buffer allocation callback appctx_res_wakeup() used to rely on old tricks to detect if a buffer was already granted to an appctx, namely by checking the task's state. Not only this test is not valid anymore, but it's inaccurate. Let's solely on SI_FL_WAIT_ROOM that is now set on allocation failure by the functions trying to allocate a buffer. The buffer is now allocated on the fly and the flag removed so that the consistency between the two remains granted. The patch also fixes minor issues such as the function being improperly declared inline(!) and the fact that using appctx_wakeup() sets the wakeup reason to TASK_WOKEN_OTHER while we try to use TASK_WOKEN_RES when waking up consecutive to a ressource allocation such as a buffer.
2018-11-06 17:32:37 +01:00
return 0;
/* allocation possible now ? */
MEDIUM: dynbuf: remove last usages of b_alloc_margin() The function's purpose used to be to fail a buffer allocation if that allocation wouldn't result in leaving some buffers available. Thus, some allocations could succeed and others fail for the sole purpose of trying to provide 2 buffers at once to process_stream(). But things have changed a lot with 1.7 breaking the promise that process_stream() would always succeed with only two buffers, and later the thread-local pool caches that keep certain buffers available that are not accounted for in the global pool so that local allocators cannot guess anything from the number of currently available pools. Let's just replace all last uses of b_alloc_margin() with b_alloc() once for all.
2021-03-22 14:44:31 +01:00
if (!b_alloc(&si_ic(si)->buf)) {
MEDIUM: stream-int: use si_rx_buff_{rdy,blk} to report buffer readiness The stream interface used to conflate a missing buffer and lack of buffer space into SI_FL_WAIT_ROOM but this causes difficulties as these cannot be checked at the same moment and are not resolved at the same moment either. Now we instead mark the buffer as presumably available using si_rx_buff_rdy() and mark it as unavailable+requested using si_rx_buff_blk(). The call to si_alloc_buf() was moved after si_stop_put(). This makes sure that the SI_FL_RX_WAIT_EP flag is cleared on allocation failure so that the function is called again if the callee fails to do its work.
2018-11-14 15:12:08 +01:00
si_rx_buff_blk(si);
MEDIUM: appctx: check for allocation attempts in buffer allocation callbacks The buffer allocation callback appctx_res_wakeup() used to rely on old tricks to detect if a buffer was already granted to an appctx, namely by checking the task's state. Not only this test is not valid anymore, but it's inaccurate. Let's solely on SI_FL_WAIT_ROOM that is now set on allocation failure by the functions trying to allocate a buffer. The buffer is now allocated on the fly and the flag removed so that the consistency between the two remains granted. The patch also fixes minor issues such as the function being improperly declared inline(!) and the fact that using appctx_wakeup() sets the wakeup reason to TASK_WOKEN_OTHER while we try to use TASK_WOKEN_RES when waking up consecutive to a ressource allocation such as a buffer.
2018-11-06 17:32:37 +01:00
return 0;
MEDIUM: stream-int: use si_rx_buff_{rdy,blk} to report buffer readiness The stream interface used to conflate a missing buffer and lack of buffer space into SI_FL_WAIT_ROOM but this causes difficulties as these cannot be checked at the same moment and are not resolved at the same moment either. Now we instead mark the buffer as presumably available using si_rx_buff_rdy() and mark it as unavailable+requested using si_rx_buff_blk(). The call to si_alloc_buf() was moved after si_stop_put(). This makes sure that the SI_FL_RX_WAIT_EP flag is cleared on allocation failure so that the function is called again if the callee fails to do its work.
2018-11-14 15:12:08 +01:00
}
MEDIUM: appctx: check for allocation attempts in buffer allocation callbacks The buffer allocation callback appctx_res_wakeup() used to rely on old tricks to detect if a buffer was already granted to an appctx, namely by checking the task's state. Not only this test is not valid anymore, but it's inaccurate. Let's solely on SI_FL_WAIT_ROOM that is now set on allocation failure by the functions trying to allocate a buffer. The buffer is now allocated on the fly and the flag removed so that the consistency between the two remains granted. The patch also fixes minor issues such as the function being improperly declared inline(!) and the fact that using appctx_wakeup() sets the wakeup reason to TASK_WOKEN_OTHER while we try to use TASK_WOKEN_RES when waking up consecutive to a ressource allocation such as a buffer.
2018-11-06 17:32:37 +01:00
task_wakeup(appctx->t, TASK_WOKEN_RES);
return 1;
}
/* Default applet handler */
MEDIUM: task: extend the state field to 32 bits It's been too short for quite a while now and is now full. It's still time to extend it to 32-bits since we have room for this without wasting any space, so we now gained 16 new bits for future flags. The values were not reassigned just in case there would be a few hidden u16 or short somewhere in which these flags are placed (as it used to be the case with stream->pending_events). The patch is tagged MEDIUM because this required to update the task's process() prototype to use an int instead of a short, that's quite a bunch of places.
2021-03-02 16:09:26 +01:00
struct task *task_run_applet(struct task *t, void *context, unsigned int state)
MEDIUM: applet: implement a run queue for active appctx The new function is called for each round of polling in order to call any active appctx. For now we pick the stream interface from the appctx's owner. At the moment there's no appctx queued yet, but we have everything needed to queue them and remove them.
2015-04-19 09:59:31 +02:00
{
MAJOR: applets: Use tasks, instead of rolling our own scheduler. There's no real reason to have a specific scheduler for applets anymore, so nuke it and just use tasks. This comes with some benefits, the first one being that applets cannot induce high latencies anymore since they share nice values with other tasks. Later it will be possible to configure the applets' nice value. The second benefit is that the applet scheduler was not very thread-friendly, having a big lock around it in prevision of this change. Thus applet-intensive workloads should now scale much better with threads. Some more improvement is possible now : some applets also use a task to handle timers and timeouts. These ones could now be simplified to use only one task.
2018-05-25 16:58:52 +02:00
struct appctx *app = context;
MEDIUM: applet: Set the conn-stream as appctx owner instead of the stream-int Because appctx is now an endpoint of the conn-stream, there is no reason to still have the stream-interface as appctx owner. Thus, the conn-stream is now the appctx owner.
2021-12-20 17:09:39 +01:00
struct stream_interface *si;
MEDIUM: appctx/debug: force a crash if an appctx spins over itself forever If an appctx is caught spinning over itself at more than 100000 loops per second and for more than one second, the process will be aborted and the offender reported on the console and logs. Typical figures usually are just a few tens to hundreds per second over a very short time so there is a huge margin here. Using even higher values could also work but there is the risk of not being able to catch offenders if multiple ones start to bug at the same time and share the load. This code should ideally be disabled for stable releases, though in theory nothing should ever trigger it.
2019-04-25 19:12:26 +02:00
unsigned int rate;
BUG/MINOR: applet: Notify the other side if data were consumed by an applet If an applet consumed output data (the amount of output data has changed between before and after the call to the applet), the producer is notified. It means CF_WRITE_PARTIAL and CF_WROTE_DATA are set on the output channel and the opposite stream interface is notified some room was made in its input buffer. This way, it is no longer the applet responsibility to take care of it. However, it doesn't matter if the applet does the same. Said like that, it looks like an improvement not a bug. But it really fixes a bug in the lua, for HTTP applets. Indeed, applet:receive() and applet:getline() are buggy for HTTP applets. Data are consumed but the producer is not notified. It means if the payload is not fully received in one time, the applet may be blocked because the producer remains blocked (it is time dependent). This patch must be backported as far as 2.0 (only for the HTX part).
2021-04-27 17:08:10 +02:00
size_t count;
MEDIUM: applet: implement a run queue for active appctx The new function is called for each round of polling in order to call any active appctx. For now we pick the stream interface from the appctx's owner. At the moment there's no appctx queued yet, but we have everything needed to queue them and remove them.
2015-04-19 09:59:31 +02:00
MAJOR: applets: Use tasks, instead of rolling our own scheduler. There's no real reason to have a specific scheduler for applets anymore, so nuke it and just use tasks. This comes with some benefits, the first one being that applets cannot induce high latencies anymore since they share nice values with other tasks. Later it will be possible to configure the applets' nice value. The second benefit is that the applet scheduler was not very thread-friendly, having a big lock around it in prevision of this change. Thus applet-intensive workloads should now scale much better with threads. Some more improvement is possible now : some applets also use a task to handle timers and timeouts. These ones could now be simplified to use only one task.
2018-05-25 16:58:52 +02:00
if (app->state & APPLET_WANT_DIE) {
__appctx_free(app);
return NULL;
MAJOR: applet: applet scheduler rework. In order to authorize call of appctx_wakeup on running task: - from within the task handler itself. - in futur, from another thread. The appctx is considered paused as default after running the handler. The handler should explicitly call appctx_wakeup to be re-called. When the appctx_free is called on a running handler. The real free is postponed at the end of the handler process.
2017-06-26 16:36:53 +02:00
}
MAJOR: applets: Use tasks, instead of rolling our own scheduler. There's no real reason to have a specific scheduler for applets anymore, so nuke it and just use tasks. This comes with some benefits, the first one being that applets cannot induce high latencies anymore since they share nice values with other tasks. Later it will be possible to configure the applets' nice value. The second benefit is that the applet scheduler was not very thread-friendly, having a big lock around it in prevision of this change. Thus applet-intensive workloads should now scale much better with threads. Some more improvement is possible now : some applets also use a task to handle timers and timeouts. These ones could now be simplified to use only one task.
2018-05-25 16:58:52 +02:00
MEDIUM: applet: Set the conn-stream as appctx owner instead of the stream-int Because appctx is now an endpoint of the conn-stream, there is no reason to still have the stream-interface as appctx owner. Thus, the conn-stream is now the appctx owner.
2021-12-20 17:09:39 +01:00
si = cs_si(app->owner);
MAJOR: applets: Use tasks, instead of rolling our own scheduler. There's no real reason to have a specific scheduler for applets anymore, so nuke it and just use tasks. This comes with some benefits, the first one being that applets cannot induce high latencies anymore since they share nice values with other tasks. Later it will be possible to configure the applets' nice value. The second benefit is that the applet scheduler was not very thread-friendly, having a big lock around it in prevision of this change. Thus applet-intensive workloads should now scale much better with threads. Some more improvement is possible now : some applets also use a task to handle timers and timeouts. These ones could now be simplified to use only one task.
2018-05-25 16:58:52 +02:00
/* We always pretend the applet can't get and doesn't want to
* put, it's up to it to change this if needed. This ensures
* that one applet which ignores any event will not spin.
BUG/MAJOR: applet: use a separate run queue to maintain list integrity If an applet wakes up and causes the next one to sleep, the active list is corrupted and cannot be scanned anymore, as the process then loops over the next element. In order to avoid this problem, we move the active applet list to a run queue and reinit the active list. Only the first element of this queue is checked, and if the element is not removed, it is removed and requeued into the active list. Since we're using a distinct list, if an applet wants to requeue another applet into the active list, it properly gets added to the active list and not to the run queue. This stops the infinite loop issue that could be caused with Lua applets, and in any future configuration where two applets could be attached together.
2015-09-25 17:56:16 +02:00
*/
MINOR: stream-int: rename si_applet_{want|stop|cant}_{get|put} It doesn't make sense to limit this code to applets, as any stream interface can use it. Let's rename it by simply dropping the "applet_" part of the name. No other change was made except updating the comments.
2018-11-06 18:46:37 +01:00
si_cant_get(si);
MINOR: stream-int: replace si_{want,stop}_put() with si_rx_endp_{more,done}() Here it's only a 1-to-1 replacement.
2018-11-14 17:54:13 +01:00
si_rx_endp_done(si);
MEDIUM: applet: implement a run queue for active appctx The new function is called for each round of polling in order to call any active appctx. For now we pick the stream interface from the appctx's owner. At the moment there's no appctx queued yet, but we have everything needed to queue them and remove them.
2015-04-19 09:59:31 +02:00
MEDIUM: stream-int: use si_rx_buff_{rdy,blk} to report buffer readiness The stream interface used to conflate a missing buffer and lack of buffer space into SI_FL_WAIT_ROOM but this causes difficulties as these cannot be checked at the same moment and are not resolved at the same moment either. Now we instead mark the buffer as presumably available using si_rx_buff_rdy() and mark it as unavailable+requested using si_rx_buff_blk(). The call to si_alloc_buf() was moved after si_stop_put(). This makes sure that the SI_FL_RX_WAIT_EP flag is cleared on allocation failure so that the function is called again if the callee fails to do its work.
2018-11-14 15:12:08 +01:00
/* Now we'll try to allocate the input buffer. We wake up the applet in
* all cases. So this is the applet's responsibility to check if this
* buffer was allocated or not. This leaves a chance for applets to do
* some other processing if needed. The applet doesn't have anything to
* do if it needs the buffer, it will be called again upon readiness.
*/
if (!si_alloc_ibuf(si, &app->buffer_wait))
MINOR: stream-int: replace si_{want,stop}_put() with si_rx_endp_{more,done}() Here it's only a 1-to-1 replacement.
2018-11-14 17:54:13 +01:00
si_rx_endp_more(si);
MEDIUM: stream-int: use si_rx_buff_{rdy,blk} to report buffer readiness The stream interface used to conflate a missing buffer and lack of buffer space into SI_FL_WAIT_ROOM but this causes difficulties as these cannot be checked at the same moment and are not resolved at the same moment either. Now we instead mark the buffer as presumably available using si_rx_buff_rdy() and mark it as unavailable+requested using si_rx_buff_blk(). The call to si_alloc_buf() was moved after si_stop_put(). This makes sure that the SI_FL_RX_WAIT_EP flag is cleared on allocation failure so that the function is called again if the callee fails to do its work.
2018-11-14 15:12:08 +01:00
BUG/MINOR: applet: Notify the other side if data were consumed by an applet If an applet consumed output data (the amount of output data has changed between before and after the call to the applet), the producer is notified. It means CF_WRITE_PARTIAL and CF_WROTE_DATA are set on the output channel and the opposite stream interface is notified some room was made in its input buffer. This way, it is no longer the applet responsibility to take care of it. However, it doesn't matter if the applet does the same. Said like that, it looks like an improvement not a bug. But it really fixes a bug in the lua, for HTTP applets. Indeed, applet:receive() and applet:getline() are buggy for HTTP applets. Data are consumed but the producer is not notified. It means if the payload is not fully received in one time, the applet may be blocked because the producer remains blocked (it is time dependent). This patch must be backported as far as 2.0 (only for the HTX part).
2021-04-27 17:08:10 +02:00
count = co_data(si_oc(si));
MAJOR: applets: Use tasks, instead of rolling our own scheduler. There's no real reason to have a specific scheduler for applets anymore, so nuke it and just use tasks. This comes with some benefits, the first one being that applets cannot induce high latencies anymore since they share nice values with other tasks. Later it will be possible to configure the applets' nice value. The second benefit is that the applet scheduler was not very thread-friendly, having a big lock around it in prevision of this change. Thus applet-intensive workloads should now scale much better with threads. Some more improvement is possible now : some applets also use a task to handle timers and timeouts. These ones could now be simplified to use only one task.
2018-05-25 16:58:52 +02:00
app->applet->fct(app);
BUG/MEDIUM: applet: always check a fast running applet's activity before killing In issue #277 is reported a strange problem related to a fast-spinning applet which seems to show valid progress being made. It's uncertain how this can happen, maybe some very specific timing patterns manage to place just a few bytes in each buffer and result in the peers applet being called a lot. But it appears possible to artificially cross the spinning threshold by asking for monster stats page (500 MB) and limiting the send() size to 1 MSS (1460 bytes), causing the stats page to be called for very small blocks which most often do not leave enough room to place a new chunk. The idea developed in this patch consists in not crashing for an applet which reaches a very high call rate if it shows some indication of progress. Detecting progress on applets is not trivial but in our case we know that they must at least not claim to wait for a buffer allocation if this buffer is present, wait for room if the buffer is empty, ask for more data without polling if such data are still present, nor leave with an empty input buffer without having written anything nor read anything from the other side while a shutw is pending. Doing so doesn't affect normal behaviors nor abuses of our existing applets and does at least protect against an applet performing an early return without processing events, or one causing an endless loop by asking for impossible conditions. This must be backported to 2.0.
2019-10-11 14:15:47 +02:00
BUG/MINOR: applet: Notify the other side if data were consumed by an applet If an applet consumed output data (the amount of output data has changed between before and after the call to the applet), the producer is notified. It means CF_WRITE_PARTIAL and CF_WROTE_DATA are set on the output channel and the opposite stream interface is notified some room was made in its input buffer. This way, it is no longer the applet responsibility to take care of it. However, it doesn't matter if the applet does the same. Said like that, it looks like an improvement not a bug. But it really fixes a bug in the lua, for HTTP applets. Indeed, applet:receive() and applet:getline() are buggy for HTTP applets. Data are consumed but the producer is not notified. It means if the payload is not fully received in one time, the applet may be blocked because the producer remains blocked (it is time dependent). This patch must be backported as far as 2.0 (only for the HTX part).
2021-04-27 17:08:10 +02:00
/* now check if the applet has released some room and forgot to
* notify the other side about it.
*/
if (count != co_data(si_oc(si))) {
si_oc(si)->flags |= CF_WRITE_PARTIAL | CF_WROTE_DATA;
si_rx_room_rdy(si_opposite(si));
}
BUG/MEDIUM: applet: always check a fast running applet's activity before killing In issue #277 is reported a strange problem related to a fast-spinning applet which seems to show valid progress being made. It's uncertain how this can happen, maybe some very specific timing patterns manage to place just a few bytes in each buffer and result in the peers applet being called a lot. But it appears possible to artificially cross the spinning threshold by asking for monster stats page (500 MB) and limiting the send() size to 1 MSS (1460 bytes), causing the stats page to be called for very small blocks which most often do not leave enough room to place a new chunk. The idea developed in this patch consists in not crashing for an applet which reaches a very high call rate if it shows some indication of progress. Detecting progress on applets is not trivial but in our case we know that they must at least not claim to wait for a buffer allocation if this buffer is present, wait for room if the buffer is empty, ask for more data without polling if such data are still present, nor leave with an empty input buffer without having written anything nor read anything from the other side while a shutw is pending. Doing so doesn't affect normal behaviors nor abuses of our existing applets and does at least protect against an applet performing an early return without processing events, or one causing an endless loop by asking for impossible conditions. This must be backported to 2.0.
2019-10-11 14:15:47 +02:00
/* measure the call rate and check for anomalies when too high */
rate = update_freq_ctr(&app->call_rate, 1);
if (rate >= 100000 && app->call_rate.prev_ctr && // looped more than 100k times over last second
((b_size(si_ib(si)) && si->flags & SI_FL_RXBLK_BUFF) || // asks for a buffer which is present
(b_size(si_ib(si)) && !b_data(si_ib(si)) && si->flags & SI_FL_RXBLK_ROOM) || // asks for room in an empty buffer
(b_data(si_ob(si)) && si_tx_endp_ready(si) && !si_tx_blocked(si)) || // asks for data already present
(!b_data(si_ib(si)) && b_data(si_ob(si)) && // didn't return anything ...
(si_oc(si)->flags & (CF_WRITE_PARTIAL|CF_SHUTW_NOW)) == CF_SHUTW_NOW))) { // ... and left data pending after a shut
stream_dump_and_crash(&app->obj_type, read_freq_ctr(&app->call_rate));
}
MAJOR: applets: Use tasks, instead of rolling our own scheduler. There's no real reason to have a specific scheduler for applets anymore, so nuke it and just use tasks. This comes with some benefits, the first one being that applets cannot induce high latencies anymore since they share nice values with other tasks. Later it will be possible to configure the applets' nice value. The second benefit is that the applet scheduler was not very thread-friendly, having a big lock around it in prevision of this change. Thus applet-intensive workloads should now scale much better with threads. Some more improvement is possible now : some applets also use a task to handle timers and timeouts. These ones could now be simplified to use only one task.
2018-05-25 16:58:52 +02:00
si_applet_wake_cb(si);
channel_release_buffer(si_ic(si), &app->buffer_wait);
return t;
MEDIUM: applet: implement a run queue for active appctx The new function is called for each round of polling in order to call any active appctx. For now we pick the stream interface from the appctx's owner. At the moment there's no appctx queued yet, but we have everything needed to queue them and remove them.
2015-04-19 09:59:31 +02:00
}
Reference in New Issue Copy Permalink