Target host OS and database configuration

Set your configuration options to maximize the performance of a target host in a Delphix engine deployment. These network-tuning changes should improve performance for any data source

Linux/Redhat/CentOs

Tune the Kernel NFS client

In Linux, the number of simultaneous NFS requests is limited by the Remote Procedure Call (RPC) subsystem. The maximum number of simultaneous requests defaults to 16. Maximize the number of simultaneous requests by changing the kernel tunable sunrpc.tcp_slot_table_entries value to 128.

To ensure that the interface does not drop packets because the driver is configured with one receive queue, use the following commands to view the adaptor policy/increase Rx queue length:

<LinuxHost> $ ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:22:BB:CC:DD:22
          inet addr:www.xxx.yyy.zzz  Bcast:www.xxx.yyy.zzz  Mask:255.255.255.0
          inet6 addr: feee::222:bbff:fffc:ddd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:760729910 errors:0 dropped:700 overruns:0 frame:0
          TX packets:309094054 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1023150307866 (952.8 GiB)  TX bytes:190673864056 (177.5 GiB)

RHEL4 through RHEL5.6

1. As a superuser, run the following command to change the instantaneous value of simultaneous RPC commands:

   Copy

   # sysctl -w sunrpc.tcp_slot_table_entries=128

2. Edit the file /etc/modprobe.d/modprobe.conf.dist and change the line:

   Copy

   install sunrpc /sbin/modprobe --first-time --ignore-install sunrpc && { /bin/mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs > /dev/null2>&1 || :;

   to

   Copy

   install sunrpc /sbin/modprobe --first-time --ignore-install sunrpc && { /bin/mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs > /dev/null2>&1 ; /sbin/sysctl -w sunrpc.tcp_slot_table_entries=128; }

   Improper changes to the modprobe.conf.dist file may disrupt the use of NFS on the system. Check with your system administrator or operating system vendor for assistance. Save a copy of the modprobe.conf.dist in a directory other than /etc/modprobe.d before starting.

RHEL 5.7 through RHEL 6.2

1. As a superuser, run the following command to change the instantaneous value of simultaneous RPC commands:

   Copy

   # sysctl -w sunrpc.tcp_slot_table_entries=128

2. If it doesn't already exist, create the file /etc/modprobe.d/rpcinfowith the following contents:

   Copy

   options sunrpc tcp_slot_table_entries=128

RHEL 6.3 onwards

Beginning with RHEL 6.3, the number of RPC slots is dynamically managed by the system and does not need to be tuned. Although the sunrpc.tcp_slot_table_entries tuneable still exists, it has a default value of 2, instead of 16 as in prior releases. The maximum number of simultaneous requests is determined by the new tuneable, sunrpc.tcp_max_slot_table_entries which has a default value of 65535.

Tune TCP optimizations for high throughput

For high-throughput, high-latency operations such as NFS, the Linux TCP stack may require additional tuning to ensure optimal performance. The following kernel parameters improve socket buffer usage, reduce retransmission overhead, and allow the TCP window to scale effectively. These recommendations apply across major Linux distributions.

System administrators may apply these settings using either /etc/sysctl.conf or a distribution-specific fileunder /etc/sysctl.d/. Delphix does not require a specific implementation method.

TCP Buffers

Increase the read/write buffer sizes to support high-throughput IO operations.

net.ipv4.tcp_rmem = 4096 16777216 16777216
net.ipv4.tcp_wmem = 4096 4194304 16777216

TCP Selective Acknowledgments (SACK)

Enable selective acknowledgment to reduce retransmission overhead and improve efficiency.

net.ipv4.tcp_sack = 1

Automatic TCP Window Scaling

Allow dynamic window scaling to adapt to variable network performance conditions.

net.ipv4.tcp_window_scaling = 1

TCP Timestamps

Enable timestamps to aid round-trip time measurement and retransmission control.

net.ipv4.tcp_timestamps = 1

Applying the Settings

After updating your sysctl configuration file, apply the new values:

sysctl -p

NFSv4 only - enable recover lost locks  

RHEL 6.6 onwards

By default, the Redhat NFSv4 client does not attempt to reclaim locks that were lost due to a lease expiration event. This can cause an application to encounter unexpected EIO errors on system calls such as write. The Delphix use case requires the NFSv4 client to attempt to reclaim lost locks that were due to lease expiration. An NFS client module parameter, 'recover_lost_locks', is used to change the default behavior. Use the following command to check if the "recover_lost_locks" option is set to 1:

grep recover_lost_locks /etc/modprobe.d/*.conf

If the option is currently set to 0, change it to 1. If it is missing, as a superuser, run the following two commands to enable the NFS client to recover the lost locks feature:

# cat > /etc/modprobe.d/nfs4-locks.conf <<EOF
options nfs recover_lost_locks=1
EOF
 
# [ -d "/sys/module/nfs" ] && echo Y > /sys/module/nfs/parameters/recover_lost_locks

IBM AIX^® 

AIX NFSv4 configuration requirements (7.2 and 7.3)

1. An NFS Domain must be configured.

2. The nfsrgyd service must be running.

3. The NFS server IP address from the Delphix Engine must be mappable to an FQDN.

4. Configure the nfsv4 domain on the AIX target host ^{bash-3.2# chnfsdom test.com}.

5. Start the nfsrgyd service and confirm it is active:

   Copy

   bash-3.2# startsrc -s nfsrgyd
   
   bash-3.2# lssrc -s nfsrgyd
   Subsystem         Group            PID          Status
    nfsrgyd          nfs              7536760      active

6. Confirm the IP address can be resolved:

   Copy

   bash-3.2$ host 172.16.105.81
   81.105.16.172.in-addr.arpa is  dcol1.delphix.com

   Reference:    IBM AIX: HOW TO SETUP NFSV4 MOUNT IN CLIENT AND SERVER.

Tune the Kernel NFS Client

On AIX, by default the maximum I/O size used for NFS read or write requests is 64K. When Oracle does I/O larger than 64K, the I/O is broken down into smaller requests that are serialized. This may result in poor I/O performance. IBM can provide an Authorized Program Analysis Report (APAR) that allows the I/O size to be configured to a larger value.

1. Determine the appropriate APAR for the version of AIX you are using:

   AIX Version	APAR Name
   6.1	IV24594
   7.1	IV24688

2. Check if the required APAR is already installed by running the following command. If the APAR is installed, you will see a message similar to this:

   Copy

   # /usr/sbin/instfix -ik IV24594

3. If the APAR is not yet installed, you will see a message similar to this:

   Copy

   All filesets for IV24594 were found.
   There was no data for IV24594 in the fix database.

4. Download and install the APAR as necessary. To find the APARs, perform a search at http://www.ibm.com/us/en/. Specify the name of the APAR you are looking for from step 1. A system reboot is necessary after installing the APAR.

5. Configure the maximum read and write sizes using the commands below:

   Copy

   # nfso -p -o nfs_max_read_size=524288
   # nfso -p -o nfs_max_write_size=524288

6. Confirm the correct settings using the command.

   You should see an output similar to this:

   Copy

   # nfso -L nfs_max_read_size -L nfs_max_write_size  
   
   NAME                CUR   DEF   BOOT  MIN   MAX   UNIT   TYPE  DEPENDENCIES  
   --------------------------------------------------------------------------------  
   nfs_max_read_size   512K  64K   512K  512   512K  Bytes  D  
   --------------------------------------------------------------------------------  
   nfs_max_write_size  512K  64K   512K  512   512K  Bytes  D  
   --------------------------------------------------------------------------------  

OS-specific tuning recommendations for Windows

Receive side scaling

Follow these instructions here to enable RSS: Enable Receive Side Scaling (RSS) on Staging/Target Network Interfaces hosted on the target, such as Oracle, SQL Server, Sybase, or vFiles. They should be applied to all Delphix targets.