The Tech Center, vSphere

Monitoring vmxnet3 Ring Buffer full condition

Posted by

What is Ring buffer?

Ring buffer is a data structure that uses a single, fixed-size buffer as if it were connected end-to-end. This structure lends itself easily to buffering data streams.

The useful property of a circular buffer is that it does not need to have its elements shuffled around when one is consumed. (If a non-circular buffer were used then it would be necessary to shift all elements when one is consumed.) In other words, the circular buffer is well-suited as a FIFO buffer while a standard, non-circular buffer is well suited as a LIFO buffer. Circular buffering makes a good implementation strategy for a queue that has fixed maximum size.

Source: https://en.wikipedia.org/wiki/Circular_buffer

How many Ring buffers does vmxnet3 use?

vmxnet3 uses 4 Ring buffers.

We can use ethtool -g to check the maximum and current settings for Linux. The command needs to be executed with in the Guest OS

root@vc [ ~ ]# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       4096
TX:             4096
Current hardware settings:
RX:             256
RX Mini:        0
RX Jumbo:       128
TX:             512

For windows we can use Device manger

Please Note: The current settings of the device are Guest OS specific.

Monitoring  if a VM is hitting buffer full condition

There is no log on VMware side that keeps a track of Ring buffer status. There are logs within Guest OS that do this.  However, it is still possible to monitor this  from VMware side using vsish commands.

All problems with a VM ruining on an ESXi host are assigned to vSphere admin’s here

From the house of sarcastic vSphere admin

The code below monitors the VMs on a host for buffer states and creates a log at user defined location. 

Code(buf_mon.sh)

#!/bin/sh
bufmon(){
sleeptime=$1
filepath=$2
for i in $(net-stats -l | grep -ivE "vmnic|vmk|PortNum" | awk '{print $6"," $1 ","$4}');
do
vmname=$(echo $i| awk -F"," '{print $1}');
portID=$(echo $i| awk -F"," '{print $2}');
portset=$(echo $i| awk -F"," '{print $3}');
outofbuf=$(vsish -e cat /net/portsets/$portset/ports/$portID/vmxnet3/rxSummary | grep "running out of buffers:");
ring1=$(vsish -e cat /net/portsets/$portset/ports/$portID/vmxnet3/rxSummary | grep "1st ring is full:");
ring2=$(vsish -e cat /net/portsets/$portset/ports/$portID/vmxnet3/rxSummary | grep "2nd ring is full:");
day_now=$(date '+%Y-%m-%d');
time_now=$(date '+%H:%M:%S');
echo $day_now,$time_now,$vmname,$outofbuf,$ring1,$ring2|sed 's/ running out of buffers://'|sed 's/ # of times the 1st ring is full://' | sed 's/# of times the 2nd ring is full://' >>$filepath;
echo -ne #
sleep $sleeptime;
done
}
filename=buf$(date '+%d%m%H%M%S').csv
vmfsuuid=$1
sleeptime=$2
nuofruns=$3
mkdir /vmfs/volumes/$vmfsuuid/buf_log
echo Date,Time,VM name,Running out of buffers,#of times the 1st ring is full,# of times the 2nd ring is full >/vmfs/volumes/$vmfsuuid/buf_log/$filename
while [ $nuofruns -gt 0 ]
do
bufmon $sleeptime /vmfs/volumes/$vmfsuuid/buf_log/$filename;
nuofruns=$(($nuofruns - 1));
done

Executing the script

  • Save the above code in a file lest say ‘buf_mon.sh’ using your preferred text editor (I like vi)
  • Change the permissions on buf_mon.sh to make is executable
  • Execute it 
[root@vmzoneblog:/tmp] Chmod +x ./buf_mon.sh
[root@vmzoneblog:/tmp] ./buf_mon.sh 5bdc2212-e8b913c4-1540-e0db550bb0d6 10 3

Notice the line 2 above. There are 3 parameters that I have passed to the script

  1. Parameter 1: 5bdc2212-e8b913c4-1540-e0db550bb0d6 is the uuid of the datastore where the script will store the log
  2. Parameter 2: Sleep time between the collections
  3. Parameter 3: Number of collections to perform

Please Note:

  • The script stores the data under buf_log directory on the specified datastore
  • All parameters are mandatory
  • Parameter are positional in nature you cannot change their placement
  • Total run time of the script will be approximately (data collection time + Parameter 2)*Parameter 3