iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

net: Add part of TCP counts explanations in snmp_counters.rst

Browse Source 
Add explanations of some generic TCP counters, fast open
related counters and TCP abort related counters and several
examples.

Signed-off-by: yupeng <yupeng0921@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
This commit is contained in:
yupeng 2018-11-16 11:17:40 -08:00 committed by David S. Miller
parent 17bf1693a6
commit 80cc49507b
1 changed files with 524 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

525
Documentation/networking/snmp_counter.rst
View File

@ -40,7 +40,7 @@ multicast packets, and would always be updated together with
IpExtOutOctets.
* IpExtInOctets and IpExtOutOctets
They are linux kernel extensions, no RFC definitions. Please note,
They are Linux kernel extensions, no RFC definitions. Please note,
RFC1213 indeed defines ifInOctets and ifOutOctets, but they
are different things. The ifInOctets and ifOutOctets include the MAC
layer header size but IpExtInOctets and IpExtOutOctets don't, they
@ -174,6 +174,163 @@ IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
IcmpInErrors.
General TCP counters
==================
* TcpInSegs
Defined in `RFC1213 tcpInSegs`_
.. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
The number of packets received by the TCP layer. As mentioned in
RFC1213, it includes the packets received in error, such as checksum
error, invalid TCP header and so on. Only one error won't be included:
if the layer 2 destination address is not the NIC's layer 2
address. It might happen if the packet is a multicast or broadcast
packet, or the NIC is in promiscuous mode. In these situations, the
packets would be delivered to the TCP layer, but the TCP layer will discard
these packets before increasing TcpInSegs. The TcpInSegs counter
isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs
counter would only increase 1.
* TcpOutSegs
Defined in `RFC1213 tcpOutSegs`_
.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
The number of packets sent by the TCP layer. As mentioned in RFC1213,
it excludes the retransmitted packets. But it includes the SYN, ACK
and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of
GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will
increase 2.
* TcpActiveOpens
Defined in `RFC1213 tcpActiveOpens`_
.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
It means the TCP layer sends a SYN, and come into the SYN-SENT
state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
increase 1.
* TcpPassiveOpens
Defined in `RFC1213 tcpPassiveOpens`_
.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
It means the TCP layer receives a SYN, replies a SYN+ACK, come into
the SYN-RCVD state.
TCP Fast Open
============
When kernel receives a TCP packet, it has two paths to handler the
packet, one is fast path, another is slow path. The comment in kernel
code provides a good explanation of them, I pasted them below::
It is split into a fast path and a slow path. The fast path is
disabled when:
- A zero window was announced from us
- zero window probing
is only handled properly on the slow path.
- Out of order segments arrived.
- Urgent data is expected.
- There is no buffer space left
- Unexpected TCP flags/window values/header lengths are received
(detected by checking the TCP header against pred_flags)
- Data is sent in both directions. The fast path only supports pure senders
or pure receivers (this means either the sequence number or the ack
value must stay constant)
- Unexpected TCP option.
Kernel will try to use fast path unless any of the above conditions
are satisfied. If the packets are out of order, kernel will handle
them in slow path, which means the performance might be not very
good. Kernel would also come into slow path if the "Delayed ack" is
used, because when using "Delayed ack", the data is sent in both
directions. When the TCP window scale option is not used, kernel will
try to enable fast path immediately when the connection comes into the
established state, but if the TCP window scale option is used, kernel
will disable the fast path at first, and try to enable it after kernel
receives packets.
* TcpExtTCPPureAcks and TcpExtTCPHPAcks
If a packet set ACK flag and has no data, it is a pure ACK packet, if
kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,
if kernel handles it in the slow path, TcpExtTCPPureAcks will
increase 1.
* TcpExtTCPHPHits
If a TCP packet has data (which means it is not a pure ACK packet),
and this packet is handled in the fast path, TcpExtTCPHPHits will
increase 1.
TCP abort
========
* TcpExtTCPAbortOnData
It means TCP layer has data in flight, but need to close the
connection. So TCP layer sends a RST to the other side, indicate the
connection is not closed very graceful. An easy way to increase this
counter is using the SO_LINGER option. Please refer to the SO_LINGER
section of the `socket man page`_:
.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
By default, when an application closes a connection, the close function
will return immediately and kernel will try to send the in-flight data
async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
to a positive number, the close function won't return immediately, but
wait for the in-flight data are acked by the other side, the max wait
time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
when the application closes a connection, kernel will send a RST
immediately and increase the TcpExtTCPAbortOnData counter.
* TcpExtTCPAbortOnClose
This counter means the application has unread data in the TCP layer when
the application wants to close the TCP connection. In such a situation,
kernel will send a RST to the other side of the TCP connection.
* TcpExtTCPAbortOnMemory
When an application closes a TCP connection, kernel still need to track
the connection, let it complete the TCP disconnect process. E.g. an
app calls the close method of a socket, kernel sends fin to the other
side of the connection, then the app has no relationship with the
socket any more, but kernel need to keep the socket, this socket
becomes an orphan socket, kernel waits for the reply of the other side,
and would come to the TIME_WAIT state finally. When kernel has no
enough memory to keep the orphan socket, kernel would send an RST to
the other side, and delete the socket, in such situation, kernel will
increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
TcpExtTCPAbortOnMemory:
1. the memory used by the TCP protocol is higher than the third value of
the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:
.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
2. the orphan socket count is higher than net.ipv4.tcp_max_orphans
* TcpExtTCPAbortOnTimeout
This counter will increase when any of the TCP timers expire. In such
situation, kernel won't send RST, just give up the connection.
* TcpExtTCPAbortOnLinger
When a TCP connection comes into FIN_WAIT_2 state, instead of waiting
for the fin packet from the other side, kernel could send a RST and
delete the socket immediately. This is not the default behavior of
Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,
you could let kernel follow this behavior.
* TcpExtTCPAbortFailed
The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is
satisfied. If an internal error occurs during this process,
TcpExtTCPAbortFailed will be increased.
.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
examples
=======
@ -220,3 +377,369 @@ and its corresponding Echo Reply packet are constructed by:
* 48 bytes data (default value of the ping command)
So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
tcp 3-way handshake
------------------
On server side, we run::
nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
Listening on [0.0.0.0] (family 0, port 9000)
On client side, we run::
nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
The server listened on tcp 9000 port, the client connected to it, they
completed the 3-way handshake.
On server side, we can find below nstat output::
nstatuser@nstat-b:~$ nstat | grep -i tcp
TcpPassiveOpens 1 0.0
TcpInSegs 2 0.0
TcpOutSegs 1 0.0
TcpExtTCPPureAcks 1 0.0
On client side, we can find below nstat output::
nstatuser@nstat-a:~$ nstat | grep -i tcp
TcpActiveOpens 1 0.0
TcpInSegs 1 0.0
TcpOutSegs 2 0.0
When the server received the first SYN, it replied a SYN+ACK, and came into
SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
of the 3-way handshake is a pure ACK without data, so
TcpExtTCPPureAcks increased 1.
When the client sent SYN, the client came into the SYN-SENT state, so
TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
1, TcpOutSegs increased 2.
TCP normal traffic
-----------------
Run nc on server::
nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
Listening on [0.0.0.0] (family 0, port 9000)
Run nc on client::
nstatuser@nstat-a:~$ nc -v nstat-b 9000
Connection to nstat-b 9000 port [tcp/*] succeeded!
Input a string in the nc client ('hello' in our example)::
nstatuser@nstat-a:~$ nc -v nstat-b 9000
Connection to nstat-b 9000 port [tcp/*] succeeded!
hello
The client side nstat output::
nstatuser@nstat-a:~$ nstat
#kernel
IpInReceives 1 0.0
IpInDelivers 1 0.0
IpOutRequests 1 0.0
TcpInSegs 1 0.0
TcpOutSegs 1 0.0
TcpExtTCPPureAcks 1 0.0
TcpExtTCPOrigDataSent 1 0.0
IpExtInOctets 52 0.0
IpExtOutOctets 58 0.0
IpExtInNoECTPkts 1 0.0
The server side nstat output::
nstatuser@nstat-b:~$ nstat
#kernel
IpInReceives 1 0.0
IpInDelivers 1 0.0
IpOutRequests 1 0.0
TcpInSegs 1 0.0
TcpOutSegs 1 0.0
IpExtInOctets 58 0.0
IpExtOutOctets 52 0.0
IpExtInNoECTPkts 1 0.0
Input a string in nc client side again ('world' in our exmaple)::
nstatuser@nstat-a:~$ nc -v nstat-b 9000
Connection to nstat-b 9000 port [tcp/*] succeeded!
hello
world
Client side nstat output::
nstatuser@nstat-a:~$ nstat
#kernel
IpInReceives 1 0.0
IpInDelivers 1 0.0
IpOutRequests 1 0.0
TcpInSegs 1 0.0
TcpOutSegs 1 0.0
TcpExtTCPHPAcks 1 0.0
TcpExtTCPOrigDataSent 1 0.0
IpExtInOctets 52 0.0
IpExtOutOctets 58 0.0
IpExtInNoECTPkts 1 0.0
Server side nstat output::
nstatuser@nstat-b:~$ nstat
#kernel
IpInReceives 1 0.0
IpInDelivers 1 0.0
IpOutRequests 1 0.0
TcpInSegs 1 0.0
TcpOutSegs 1 0.0
TcpExtTCPHPHits 1 0.0
IpExtInOctets 58 0.0
IpExtOutOctets 52 0.0
IpExtInNoECTPkts 1 0.0
Compare the first client-side nstat and the second client-side nstat,
we could find one difference: the first one had a 'TcpExtTCPPureAcks',
but the second one had a 'TcpExtTCPHPAcks'. The first server-side
nstat and the second server-side nstat had a difference too: the
second server-side nstat had a TcpExtTCPHPHits, but the first
server-side nstat didn't have it. The network traffic patterns were
exactly the same: the client sent a packet to the server, the server
replied an ACK. But kernel handled them in different ways. When the
TCP window scale option is not used, kernel will try to enable fast
path immediately when the connection comes into the established state,
but if the TCP window scale option is used, kernel will disable the
fast path at first, and try to enable it after kerenl receives
packets. We could use the 'ss' command to verify whether the window
scale option is used. e.g. run below command on either server or
client::
nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
Netid Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp 0 0 192.168.122.250:40654 192.168.122.251:9000
ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98
The 'wscale:7,7' means both server and client set the window scale
option to 7. Now we could explain the nstat output in our test:
In the first nstat output of client side, the client sent a packet, server
reply an ACK, when kernel handled this ACK, the fast path was not
enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.
In the second nstat output of client side, the client sent a packet again,
and received another ACK from the server, in this time, the fast path is
enabled, and the ACK was qualified for fast path, so it was handled by
the fast path, so this ACK was counted into TcpExtTCPHPAcks.
In the first nstat output of server side, fast path was not enabled,
so there was no 'TcpExtTCPHPHits'.
In the second nstat output of server side, the fast path was enabled,
and the packet received from client qualified for fast path, so it
was counted into 'TcpExtTCPHPHits'.
TcpExtTCPAbortOnClose
--------------------
On the server side, we run below python script::
import socket
import time
port = 9000
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('0.0.0.0', port))
s.listen(1)
sock, addr = s.accept()
while True:
time.sleep(9999999)
This python script listen on 9000 port, but doesn't read anything from
the connection.
On the client side, we send the string "hello" by nc::
nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
Then, we come back to the server side, the server has received the "hello"
packet, and the TCP layer has acked this packet, but the application didn't
read it yet. We type Ctrl-C to terminate the server script. Then we
could find TcpExtTCPAbortOnClose increased 1 on the server side::
nstatuser@nstat-b:~$ nstat | grep -i abort
TcpExtTCPAbortOnClose 1 0.0
If we run tcpdump on the server side, we could find the server sent a
RST after we type Ctrl-C.
TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout
-----------------------------------------------
Below is an example which let the orphan socket count be higher than
net.ipv4.tcp_max_orphans.
Change tcp_max_orphans to a smaller value on client::
sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
Client code (create 64 connection to server)::
nstatuser@nstat-a:~$ cat client_orphan.py
import socket
import time
server = 'nstat-b' # server address
port = 9000
count = 64
connection_list = []
for i in range(64):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((server, port))
connection_list.append(s)
print("connection_count: %d" % len(connection_list))
while True:
time.sleep(99999)
Server code (accept 64 connection from client)::
nstatuser@nstat-b:~$ cat server_orphan.py
import socket
import time
port = 9000
count = 64
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('0.0.0.0', port))
s.listen(count)
connection_list = []
while True:
sock, addr = s.accept()
connection_list.append((sock, addr))
print("connection_count: %d" % len(connection_list))
Run the python scripts on server and client.
On server::
python3 server_orphan.py
On client::
python3 client_orphan.py
Run iptables on server::
sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
Type Ctrl-C on client, stop client_orphan.py.
Check TcpExtTCPAbortOnMemory on client::
nstatuser@nstat-a:~$ nstat | grep -i abort
TcpExtTCPAbortOnMemory 54 0.0
Check orphane socket count on client::
nstatuser@nstat-a:~$ ss -s
Total: 131 (kernel 0)
TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0
Transport Total IP IPv6
* 0 - -
RAW 1 0 1
UDP 1 1 0
TCP 14 13 1
INET 16 14 2
FRAG 0 0 0
The explanation of the test: after run server_orphan.py and
client_orphan.py, we set up 64 connections between server and
client. Run the iptables command, the server will drop all packets from
the client, type Ctrl-C on client_orphan.py, the system of the client
would try to close these connections, and before they are closed
gracefully, these connections became orphan sockets. As the iptables
of the server blocked packets from the client, the server won't receive fin
from the client, so all connection on clients would be stuck on FIN_WAIT_1
stage, so they will keep as orphan sockets until timeout. We have echo
10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
only keep 10 orphan sockets, for all other orphan sockets, the client
system sent RST for them and delete them. We have 64 connections, so
the 'ss -s' command shows the system has 10 orphan sockets, and the
value of TcpExtTCPAbortOnMemory was 54.
An additional explanation about orphan socket count: You could find the
exactly orphan socket count by the 'ss -s' command, but when kernel
decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel
doesn't always check the exactly orphan socket count. For increasing
performance, kernel checks an approximate count firstly, if the
approximate count is more than tcp_max_orphans, kernel checks the
exact count again. So if the approximate count is less than
tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
would find TcpExtTCPAbortOnMemory is not increased at all. If
tcp_max_orphans is large enough, it won't occur, but if you decrease
tcp_max_orphans to a small value like our test, you might find this
issue. So in our test, the client set up 64 connections although the
tcp_max_orphans is 10. If the client only set up 11 connections, we
can't find the change of TcpExtTCPAbortOnMemory.
Continue the previous test, we wait for several minutes. Because of the
iptables on the server blocked the traffic, the server wouldn't receive
fin, and all the client's orphan sockets would timeout on the
FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
10 timeout on the client::
nstatuser@nstat-a:~$ nstat | grep -i abort
TcpExtTCPAbortOnTimeout 10 0.0
TcpExtTCPAbortOnLinger
---------------------
The server side code::
nstatuser@nstat-b:~$ cat server_linger.py
import socket
import time
port = 9000
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('0.0.0.0', port))
s.listen(1)
sock, addr = s.accept()
while True:
time.sleep(9999999)
The client side code::
nstatuser@nstat-a:~$ cat client_linger.py
import socket
import struct
server = 'nstat-b' # server address
port = 9000
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
s.connect((server, port))
s.close()
Run server_linger.py on server::
nstatuser@nstat-b:~$ python3 server_linger.py
Run client_linger.py on client::
nstatuser@nstat-a:~$ python3 client_linger.py
After run client_linger.py, check the output of nstat::
nstatuser@nstat-a:~$ nstat | grep -i abort
TcpExtTCPAbortOnLinger 1 0.0