68d02e5fa9
An interesting issue was met when testing the mux-to-mux forwarding code. In order to preserve fairness, in h2_snd_buf() if other streams are waiting in send_list or fctl_list, the stream that is attempting to send also goes to its list, and will be woken up by h2_process_mux() or h2_send() when some space is released. But on rare occasions, there are only a few (or even a single) streams waiting in this list, and these streams are just quickly removed because of a timeout or a quick h2_detach() that calls h2s_destroy(). In this case there's no even to wake up the other waiting stream in its list, and this will possibly resume processing after some client WINDOW_UPDATE frames or even new streams, so usually it doesn't last too long and it not much noticeable, reason why it was left that long. In addition, measures have shown that in heavy network-bound benchmark, this exact situation happens on less than 1% of the streams (reached 4% with mux-mux). The fix here consists in replacing these LIST_DEL_INIT() calls on h2s->list with a function call that checks if other streams were queued to the send_list recently, and if so, which also tries to resume them by calling h2_resume_each_sending_h2s(). The detection of late additions is made via a new flag on the connection, H2_CF_WAIT_INLIST, which is set when a stream is queued due to other streams being present, and which is cleared when this is function is called. It is particularly difficult to reproduce this case which is particularly timing-dependent, but in a constrained environment, a test involving 32 conns of 20 streams each, all downloading a 10 MB object previously showed a limitation of 17 Gbps with lots of idle CPU time, and now filled the cable at 25 Gbps. This should be backported to all versions where it applies.