BUG/MEDIUM: mux-h2: only restart sending when mux buffer is decongested

During performance tests, Emeric faced a case where the wakeups of
sc_conn_io_cb() caused by h2_resume_each_sending_h2s() was multiplied
by 5-50 and a lot of CPU was being spent doing this for apparently no
reason.

The culprit is h2_send() not behaving well with congested buffers and
small SSL records. What happens when the output is congested is that
all buffers are full, and data are emitted in 2kB chunks, which are
sufficient to wake all streams up again to ask them to send data again,
something that will obviously only work for one of them at best, and
waste a lot of CPU in wakeups and memcpy() due to the small buffers.
When this happens, the performance can be divided by 2-2.5 on large
objects.

Here the chosen solution against this is to keep in mind that as long
as there are still at least two buffers in the ring after calling
xprt->snd_buf(), it means that the output is congested and there's
no point trying again, because these data will just be placed into
such buffers and will wait there. Instead we only mark the buffer
decongested once we're back to a single allocated buffer in the ring.

By doing so we preserve the ability to deal with large concurrent
bursts while not causing a thundering herd by waking all streams for
almost nothing.

This needs to be backported to 2.7 and 2.6. Other versions could
benefit from it as well but it's not strictly necessary, and we can
reconsider this option if some excess calls to sc_conn_io_cb() are
faced.

Note that this fix depends on this recent commit:
    MINOR: buffer: add br_single() to check if a buffer ring has more than one buf

src/mux_h2.c

@@ -3835,8 +3835,17 @@ static int h2_send(struct h2c *h2c)
 		if (released)
 			offer_buffers(NULL, released);
 
-		/* wrote at least one byte, the buffer is not full anymore */
-		if (sent)
+		/* Normally if wrote at least one byte, the buffer is not full
+		 * anymore. However, if it was marked full because all of its
+		 * buffers were used, we don't want to instantly wake up many
+		 * streams because we'd create a thundering herd effect, notably
+		 * when data are flushed in small chunks. Instead we wait for
+		 * the buffer to be decongested again before allowing to send
+		 * again. It also has the added benefit of not pumping more
+		 * data from the other side when it's known that this one is
+		 * still congested.
+		 */
+		if (sent && br_single(h2c->mbuf))
 			h2c->flags &= ~(H2_CF_MUX_MFULL | H2_CF_DEM_MROOM);
 	}