Cluster Interconnect Network Benchmarking

Netperf is a great tool for benchmarking and stress testing your cluster interconnect. Oracle RAC demands a high bandwidth, low latency interconnect in order to effectively utilize Cache Fusion. Cache Fusion, in a nutshell, means allowing each server in the cluster to access blocks of data in each other’s memory, rather than going to disk. In this particular cluster, with 128GB of memory in each server, and 4 servers, we have an effective 512GB cache. Therefore, the cluster interconnect becomes critical because it is essentially, another (slightly slower) memory bus. Oracle uses UDP to communicate between servers in the cluster, not only for heartbeat, but also for these fast server to server memory copy operations.

Here’s how to benchmark your cluster interconnect network:

1. Download the source tarball from ftp://ftp.netperf.org/netperf/netperf-2.4.5.tar.gz
2. Extract to a directory, go to that directory, “./configure --enable-demo=yes;make;make install”
3. Add the following line to your /etc/services file: “netperf 12865/tcp”
4. Add /etc/xinetd.d/netperf with the following contents:

# default: on
# description: Netperf service
service netperf
{
               disable = no
               socket_type = stream
               protocol = tcp
               wait = no
               user = root
               server = /usr/local/bin/netserver
} 

5. Open port 12865 TCP and UDP on your iptables configuration.
6. Test your installation by typing “netperf”, which will initiate a test to the loopback adapter for 10 seconds.
7. Repeat the above steps for all servers in the cluster.

Now, to test your cluster interconnect, here are some tests you can run:

1. TCP Stream Test (10 seconds): “root@server1# netperf –H server2 –t TCP_STREAM”
2. UDP Stream Test (10 seconds): “root@server1# netperf –H server2 –t UDP_STREAM”
3. UDP Request/Response (10 seconds): “root@server1# netperf –H server2 –t UDP_RR”

And, the test that I believe best represents continuous Oracle Interconnect (Cache Fusion) traffic:

1 hour long UDP stream test:

[root@server1 ~]# netperf -t UDP_STREAM -D 10 -l 3600 -H server2
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to server2 (x.x.x.x) port 0 AF_INET : demo
Interim result: 9920.38 10^6bits/s over 10.01 seconds
Interim result: 9925.23 10^6bits/s over 10.00 seconds
Interim result: 9922.43 10^6bits/s over 10.00 seconds

After an hour you should get a result like this:

Socket  Message  Elapsed      Messages               
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

1048576   65507   300.05    5682971      0    9925.78
1048576           300.05    3647379           6370.45 

It looks like we are getting 9925 megabits from our 10 gigabit ethernet connections. We are using jumbo frames. My experience is that you can’t get more than 5-6mb without jumbo frames (9000 MTU). You also want to verify the errors should be 0, otherwise you might have some packet loss or physical network issues on the cluster interconnect.

I recommend running these tests and benchmarks on both the public and private network connections, between every server.

Happy benchmarking!