Performance Tuning - Overview
Brett Lee
=================================


Principles:
----------------
1.  Go for the low hanging fruit first / get the biggest bang for the buck
2.  Do things in big chunks / avoid small, repetitive tasks
3.  Look for potential problems in the major areas
4.  Measure performance, make incremental change, reassess performance


Major Areas:
----------------
1.  CPU
2.  Memory
3.  Disk / Array
4.  Filesystems
5.  Networking
6.  NFS
7.  OS Specific


CPU
----------------
1.  Disable all unnecessary processes
    - they won't be competing for time on the CPU
2.  Optimize cache hits
    - build systems with sufficient L1/L2 cache
    - utilize CPU affinity
    - when L2 cache is shared, create processor groups using shared L2 CPUs
3.  Increase time on CPU of critical processes via classes, priorities, time
    quanta and designated IRQ handling


Memory (RAM)
----------------
1.  Sufficient RAM to avoid paging / swapping
2.  Use multi-channel / high-speed RAM


Disk / Array
----------------
1.  Build arrays with data striped across a large number of disks
2.  Increase the array stripe size to be as large as possible (without wasting disk)
    - 8k is common as it is the default block size for UFS / VxFS
3.  Maximize the array cache
4.  Opt for RAID 3 or 5 with "hot-standby" disks over RAID with mirroring
5.  Opt for disks with fast access and seek times
6.  Tune SATA/IDE disk(s) using `hdparm`
    - enable DMA
      - use DMA channels and less CPU
    - enable auto-readahead with sufficient readahead size
      + beneficial for sequential reads of large files
    - enable unmasking of interrupts
7.  Tune SCSI disk(s) using `sdparm`
8.  Enable write-back cache (use with caution)
9.  Disable power management during periods of heavy use


Filesystem
----------------
1.  Disable atime
2.  Tune the block size to reflect the data to be stored on the particular filesystem
3.  If using an array, opt to have the array stripe size be equal to a multiple of the block size
4.  Use ramdisks (with caution)
5.  On large filesystems, decrease the amount of reserved space
6.  On newer filesystems, enable write-back cache (use with caution) and configure fewer writes
7.  Use an appropriate I/O scheduler and set job priorities if supported


Networking
----------------
1.  Tune the NIC using ndd, ethtool, mii-tool, etc.
    - top speed, full duplex, packet size, etc.
2.  Tune TCP / UDP settings and buffers
    - some OSs auto-tune
    - TCP tuning for high-bandwidth / high latency links
      + send buffer size = bandwidth * round-trip time
    - UDP, increase receive buffer size
3.  Increase the MTU


NFS (if used)
----------------
1.  Increase rsize and wsize
2.  Use async (respond before commit)
3.  On stable networks with heavy congestion, use TCP instead of UDP
    - smaller retrans when packets dropped
    - better support for different network speeds
4.  Increase the number of threads for the NFS server
5.  See: http://nfs.sourceforge.net/nfs-howto/ar01s05.html


OS Specific (kinda-sorta, as source is typically available) & Common Tools
----------------
1.  Solaris
    - strace - print STREAMS trace messages
    - truss - trace system calls and signals
    - dtrace - DTrace dynamic tracing compiler and tracing utility
      - http://docs.sun.com/app/docs/doc/817-6223

2.  Linux
    - strace - trace system calls and signals
    - Oprofile - for CPU bound processes
      - http://oprofile.sourceforge.net/news/
    - SystemTap (STAP) - dTrace knockoff
      - http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/SystemTap_Beginners_Guide

3.  Common
    - sar, iostat, vmstat, mpstat, netstat, prstat, top, atop, iotop ...