Huge Packet Drops (Tx drops) Observed on NetScaler

Posted 2021-12-13 lsgxeva

Huge Packet Drops (Tx drops) Observed on NetScaler

来源  https://support.citrix.com/article/CTX215843

Shell Command : nsconmsg101 -K newnslog -g nic_err -s disptime=1 -d current

Shell Command : nsconmsg101 -K newnslog -g nic_err -s disptime=1 -d statswt0

 

Article | NetworkingPerformance | Created: 15 Dec 2014 | Modified: 11 Aug 2016

 

Applicable Products

• NetScaler

Symptoms or Error

• Huge packet drops was observed on NetScaler. The drops are TX drops (software drops).
• Customer attempted to ping NSIP and could see that most of the packet gets dropped. All configured services goes down and comes up frequently. Users could see that application gets disconnected very frequently (Applications have been launched through NetScaler Gateway).
• We can see from newnslog that "nic_err_congestionlimit_pkts_dropped" and "nic_err_tx_dropped" counters" are incrementing:

  /var/nslog]$ nsconmsg101 -K newnslog -g nic_err -s disptime=1 -d current | grep congest |more
     6195       0        6159352      48887     6944 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:17:38 2014
     6201       0        6208229      48877     6962 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:17:45 2014
     6402       0        6257108      48879     6943 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:25:32 2014
     6462       0        6305991      48883     6953 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:27:45 2014
     6468       0        6354868      48877     6932 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:27:52 2014
     6805       0        6403744      48876     6942 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:40:55 2014
  /var/nslog]$ nsconmsg101 -K newnslog -g nic_err_tx_dropped -s time=04Dec2014:17:00 -s disptime=1 -d current | more
   
  reltime:mili second between two records Thu Dec  4 17:06:08 2014
    Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
        0 2697427        8897070      48877     6942 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:08 2014
        1    7040        8945947      48877     6942 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:15 2014
        2    7170        8994878      48931     6824 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:22 2014
        3   78151        9092633      97755    13885 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:07:40 2014 

---

Solution

Ensure that you are not using Management port (interface 0/1) for production traffic.

The best practice is to use management card only for Management traffic and the Network card for the Production traffic.

Management interfaces uses software RSS instead of hardware RSS, so this can impact performance.

Enabled the interface 1/1 and then disabled the interface 0/1.

---

Problem Cause

Management port (interface 0/1) used for production/data traffic.

---

 

 

 

NetScaler VPX Loses Network Connectivity on VMware ESXi 5.1.0 2191751, VMware ESXi 5.5 2143827 and also on VMware ESXi 6.0

来源 https://support.citrix.com/article/CTX200278

 

Article | ConnectivityNetworkingThird PartyFixed Issues | 163 found this helpful | Created: 05 Nov 2014 | Modified: 29 Sep 2016

 

Applicable Products

• NetScaler

Symptoms or Error

The following are the issues encountered when NetScaler VPX is run on VMware ESXi 5.1.0 2191751, VMware ESXi 5.5 2143827 and also on VMware ESXi 6.0 

• Network disconnectivity.

• NetScaler GUI access is affected.

• When running show interfaces command from NetScaler CLI you might notice that one of the interfaces has stalls and errors.

---

Solution

To fix this issue upgrade to NetScaler 10.5 build 55.8. This firmware can be download from Citrix Downloads.

For more information, see Fixed Issues section from NetScaler 10.5 Build 55.8 Release Notes.

The following are the workarounds for this issue:

Workaround 1

Install NetScaler VPX appliance on a previously known good VMware version and build.

Workaround 2

1. SSH and log on to Citrix NetScaler VPX appliance as nsroot.

2. Type shell.

3. Change directory (cd) to /flash/boot.

4. Create file /flash/boot/loader.conf.local (if not present) with same permissions as/flash/boot/loader.conf. Add the following line and reboot:
   hw.em.txd=512
   Note: To create the file, use command touch loader.conf.local.

vi Commands

The following are the vi commands to edit the document:

1. From NetScaler shell type:
   vi <filename>

2. Move the cursor to the last character of text in the file, type "a" and click Enter.

3. Type the line:
   hw.em.txd=512

4. Press the ESC key and then ":" key. The cursor will move to the bottom of the page, then type wq!.

---

Problem Cause

NetScaler VPX network connectivity issue on VMware ESXi 5.1.0 build 2191751 and VMware ESXi 5.5 build 2143827 is caused by "tx_ring_length" mismatch, which causes TX stalls.

---

Additional Resources

For more information refer to VMware Knowledge Base - 2092809.

---

 

 

Netscaler NIC Counters

来源 https://support.citrix.com/article/CTX125102

 

Article | Other | Created: 14 Jun 2017 | Modified: 14 Jun 2017

 

Applicable Products

• NetScaler

Information

This article provides a correlation of the newnslog NIC counters, their SNMP counterpart, and a brief description.

Newnslog NIC Counter

The table below lists the 59 different newnslog NIC counters, a simple description of the counter, and the matching SNMP object name.

Newnslog Counter Name	SNMP Name	Description
nic_tot_rx_bytes	TotRxBytes	Number of bytes received by this interface
nic_rx_avg_bandwidth	RxAvgBandwidthUsage	The average rx bandwidth on interface in bits per second
nic_tot_rx_packets	TotRxPkts	Number of packets received by this interface
nic_rx_avg_packet_rate	RxAvgPacketRate	The average rate of incoming packets on the interface since the system start
nic_tot_rx_mbits	TotRxMbits	Number of megabits received by this interface
nic_tot_tx_bytes	TotTxBytes	Number of bytes transmitted by this interface
nic_tx_avg_bandwidth	TxAvgBandwidthUsage	The average tx bandwidth on interface in bits per second
nic_tot_tx_packets	TotTxPkts	Number of packets transmitted by this interface
nic_tx_avg_packet_rate	TxAvgPacketRate	The average rate of outgoing packets on the interface since the system start
nic_tot_tx_mbits	TotTxMbits	Number of megabits transmitted by this interface
nic_tot_multicast_pkts	nicTotMulticastPkts	Number of multicast packets received by this interface
nic_tot_netscaler_pkts	TotNetScalerPkts	Number of NetScaler packets received by this interface
nic_tot_rx_lacpdus	RxLacpdu	Number of LACPDUs received by this interface
nic_tot_tx_lacpdus	TxLacpdu	Number of LACPDUs transmitted by this interface
nic_err_rx	ErrPktRx	Number of erroneous packets received by this interface
nic_err_tx	ErrPktTx	Number of errors encountered during transmitting by this interface
nic_cur_link_state	CurLinkState	Current link state
nic_cur_link_uptime	CurLinkUptime	Current link uptime
nic_cur_link_downtime	CurLinkDowntime	Current link downtime
nic_err_dropped_pkts	ErrDroppedRxPkts	Number of received packets dropped by this interface
nic_err_tx_dropped	ErrDroppedTxPkts	Number of packets dropped in transmission by this interface
nic_tot_bdg_mac_moved	TotMacMoved	Number of times MAC moved between ports
nic_err_bdg_muted	ErrNicMuted	Number of times NIC become bridging-muted because of loop detection
nic_cur_txqlen	CurTxQLen	Number of packets queued for transmission
nic_err_link_hangs	ErrLinkHangs	Number of NIC hangs detected
nic_err_duplex_mismatch	ErrDuplexMismatch	Number of duplex mismatches detected
nic_err_rx_nonsb	ErrTxNoNSB	Number of times there was not enough memory for processing of the received data
nic_err_rx_crc	RxCRCErrors	Number of received packets with CRC errors (alignment or FCS)
nic_err_rx_long_frame	RxFrameErrors	Number of too long packets received on the interface
nic_err_rx_alignment	RxAlignmentErrors	The number of alignment errors received on the interface
nic_err_ifInDiscards	ErrIfInDiscards	Number of inbound error-free packets discarded internally by NIC
nic_err_ifOutDiscards	nicErrIfOutDiscards	Number of outbound error-free packets discarded
nic_err_link_reinits	LinkReinits	Number of link re-initializations (attempts to establish the link)
nic_err_link_sts_stalls	nicStsStalls	Number of Status stalls detected
nic_err_link_tx_stalls	nicTxStalls	Number of Transmit stalls detected
nic_err_link_rx_stalls	nicRxStalls	Number of Receive stalls detected
nic_err_link_errdisables	nicErrDisables	Number of times link has been error-disabled
nic_cur_Ptxqlen	HpTxQLen	Number of high-priority packets queued for transmission
nic_cur_LPtxqlen	LpTxQLen	Number of low-priority packets queued for transmit
nic_err_tx_nonucpkt	ErrTxNonUcPkt	Number of output non-unicast packets
nic_err_tx_collisions	TxCollisions	Number of collisions in transmission (half-duplex only)
nic_err_tx_excess_collisions	TxExcessCollisions	Number of excess collisions in transmission (half-duplex only)
nic_err_tx_late_collisions	TxLateCollisions	Number of late collisions in transmission (half-duplex only)
nic_err_tx_multi_collision_frame	TxMultiCollisionErrors	Number of multi collision during transmission (half-duplex only)
nic_err_tx_carrier	TxCarrierError	Number of carrier errors during transmission
nic_err_rx_nobufs	ErrRxNoBuffs	Number of times there was no memory for Rx replacement buffers
nic_err_tx_overflow	ErrTxOverflow	Number of times there was an overflow of the Tx ring (Informational)
nic_err_congested_pkts_dropped	ErrCongestedPktsDrops	Number of output packets dropped because of congestion
nic_err_congestionlimit_pkts_dropped	ErrCongestionLimitPktDrops	Number of output packets dropped during the congestion time
nic_err_rx_FCS	ErrRxFCS	Number of packets received with Frame Check Sequence (FCS) errors
nic_err_rx_fifo	ErrRxFIFO	NIC receive FIFO memory overruns
nic_tot_outXoffSent	TotXoffSent	Number of XOFF pause frames sent
nic_tot_outXonSent	TotXonSent	Number of XON pause frames sent
nic_tot_xoffPauseFramesReceived	TotRxXoffPause	Number of XOFF pause frames received
nic_tot_xoffStateEntered	TotXoffStateEntered	Number of times the XOFF state was entered
nic_tot_xonPauseFramesReceived	TotRxXonPause	Number of XON pause frames received
nic_err_tx_deferred	ErrTxDeferred	Number of times a packet transmission was deferred (Informational - half-duplex only)
nic_err_tx_fifo	ErrTxFIFO	NIC transmission FIFO under-runs
nic_err_tx_heartbeat	ErrTxHeartBeat	No 10Mb link heartbeats (Informational - 10Mb half-duplex only)

 

====================== End