网卡丢包排查二

2023-08-15 04:06| 来源: 网络整理| 查看: 265

上次没查到根本原因，这次继续排查，发现和我们发的包有很大的关系，根据上次rx_drop的含义，丢的数据可能是以下几种原因：

The softnet backlog full Bad VLAN tags Packets received with unknown or unregistered protocols IPv6 frames when the server is configured only for ipv4 一 dropwatch使用 1.1 使用

丢包排查的神器，上篇已经说过，这次在仔细用以下，dropwatch目前要求首先内核必须大于等于2.6.30。

yum install dropwatch

采用交互方式运行：

[root@localhost ~]# dropwatch -l kas >start

启动，按ctrl+C停止。

dropwatch> start Enabling monitoring... Kernel monitoring activated. Issue Ctrl-C to stop monitoring 15 drops at ip6_mc_input+1ad (0xffffffff8164414d) 117 drops at ip_rcv+cf (0xffffffff815ca47f) 2 drops at __udp4_lib_mcast_deliver+2ad (0xffffffff815fbe3d) 6 drops at ip_rcv_finish+1b4 (0xffffffff815c9e04) 2 drops at __netif_receive_skb_core+3d2 (0xffffffff81586d82) 2 drops at ip_forward+1d0 (0xffffffff815cbee0) 1 drops at skb_queue_purge+18 (0xffffffff81572ec8) 13 drops at ip_rcv_finish+1b4 (0xffffffff815c9e04) 7 drops at tcp_v4_rcv+87 (0xffffffff815f0197) 25 drops at ip6_mc_input+1ad (0xffffffff8164414d) 15 drops at ip_rcv_finish+1b4 (0xffffffff815c9e04) 118 drops at ip_rcv+cf (0xffffffff815ca47f) 3 drops at ip_forward+1d0 (0xffffffff815cbee0) 1 drops at __brk_limit+3e1e36a8 (0xffffffffc05f46a8) 1 drops at __netif_receive_skb_core+3d2 (0xffffffff81586d82) 4 drops at ip_rcv_finish+1b4 (0xffffffff815c9e04) 7 drops at tcp_v4_rcv+87 (0xffffffff815f0197) 17 drops at ip6_mc_input+1ad (0xffffffff8164414d) 112 drops at ip_rcv+cf (0xffffffff815ca47f) 6 drops at ip_forward+1d0 (0xffffffff815cbee0) 5 drops at ip_rcv_finish+1b4 (0xffffffff815c9e04)

丢包主要几种在：118 drops at ip_rcv+cf (0xffffffff815ca47f)这个函数，查看drop位置：

#grep -w "ip_rcv" /boot/System.map-$(uname -r) -A10 ffffffff815ca3b0 T ip_rcv ffffffff815ca7c0 t ip4_frag_match ffffffff815ca810 t ipqhashfn ffffffff815ca8c0 t ip4_hashfn ffffffff815ca8f0 t ip4_frag_free ffffffff815ca910 t ip4_frag_init ffffffff815ca9f0 t ipv4_frags_exit_net ffffffff815caa30 t ipv4_frags_init_net ffffffff815cab30 t ip_expire ffffffff815cac50 T ip_defrag ffffffff815cba30 T ip_check_defrag

我们注意到0xffffffff815ca47f在ffffffff815ca3b0和ffffffff815ca7c0之间。如果dropwatch显示的地址，则可以通过以下命令进行定位：

cat /proc/kallsyms|grep xxx(地址前面xx位)

找下内核源码去查看：

找到相应版本后，搜索ip_rcv [https://elixir.bootlin.com/linux/v3.10.69/source/net/ipv4/ip_input.c#L375](https://elixir.bootlin.com/linux/v3.10.69/source/net/ipv4/ip_input.c#L375) 内核ip_rcv代码

大概是pkt_type类型不对,丢弃掉不是发往本机的报文，skb->pkt_type在网卡接收报文处理以太网头时会根据dst mac设置，协议栈的书会讲不是发往本机的广播报文会在二层被丢弃，但是这个和我们的不符合，我们是采用混杂模式，所以不存在mac不匹配的问题。

二其他丢包查询办法 2.1 ip查看丢包 $ip -s -s link ls eno2 24: eno2: mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 78:ac:44:0f:16:00 brd ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped overrun mcast 16179332325329 79927306089 0 123106137 0 315991391 RX errors: length crc frame fifo missed 0 0 0 0 0 TX: bytes packets errors dropped carrier collsns 0 0 0 0 0 0 TX errors: aborted fifo window heartbeat transns 0 0 0 0 8 2.2 netstat 查看丢包情况 $ netstat -i Kernel Interface table Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eno1 1500 132108 0 24242 0 7034 0 0 0 BMRU eno2 1500 80057630017 0 123636508 0 0 0 0 0 BMPRU eno3 1500 173704853 0 527651 0 52369033 0 0 0 BMPRU eno4 1500 8497 0 0 0 0 0 0 0 BMU enp59s0f0 1500 0 0 0 0 0 0 0 0 BMU enp59s0f1 1500 0 0 0 0 0 0 0 0 BMU lo 65536 581220807 0 0 0 581220807 0 0 0 LRU vboxnet0 1500 0 0 0 0 20880 0 0 0 BMRU virbr0 1500 0 0 0 0 0 0 0 0 BMU 2.3 查看具体协议的丢包

比如udp丢包

netstat -su IcmpMsg: InType0: 112 InType3: 165692 InType8: 7 InType11: 7434 OutType0: 7 OutType3: 169248 OutType5: 7 OutType8: 153 Udp: 126528161 packets received 77827 packets to unknown port received. 20408348 packet receive errors 120504641 packets sent 20408348 receive buffer errors 0 send buffer errors UdpLite: IpExt: InNoRoutes: 4 InMcastPkts: 26550045 OutMcastPkts: 2576 InBcastPkts: 456933 OutBcastPkts: 36 InOctets: 1176854170075 OutOctets: 1113352074110 InMcastOctets: 7163508127 OutMcastOctets: 165775 InBcastOctets: 138811485 OutBcastOctets: 2808 InCsumErrors: 21934887 InNoECTPkts: 1058031660 InECT0Pkts: 2361 2.4查看各个协议的收发情况 #cat /proc/net/snmp Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates Ip: 1 64 1054834458 21934887 0 5631 159 0 652874483 636977013 0 52 256 3409 889 256 7277 0 23887 Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps Icmp: 173245 34807 0 165692 7434 0 0 0 7 112 0 0 0 0 169415 0 169248 0 0 0 7 153 7 0 0 0 0 IcmpMsg: InType0 InType3 InType8 InType11 OutType0 OutType3 OutType5 OutType8 IcmpMsg: 112 165692 7 7434 7 169248 7 153 Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts InCsumErrors Tcp: 1 200 120000 -1 547226 349549 365407 43767 184 505212431 516379275 219665 0 489834 0 Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors Udp: 126561398 77827 20408348 120536958 20408348 0 0 UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors UdpLite: 0 0 0 0 0 0 0 三参考

如何分析linux tcp/ip 丢包问题 systemtap能做什么？第一篇

【本文地址】

网卡丢包排查二

网卡丢包排查二

今日新闻

推荐新闻