Everything Penguin

Focusing on Linux-based Operating Systems
htDig Search:

Operating Systems
  • /pub/OS/Linux

  • Storage
  • File Systems
  • HPC
  • /pub/Storage

  • Networking
  • /pub/Networking

  • Network Services
  • /pub/NetworkServices

  • Security
  • /pub/Security
  • Keytool/OpenSSL

  • Clustering
  • HA
  • DRM

  • Development
  • Design
  • C/C++
  • Java
  • Perl
  • Python
  • Shell
  • Web / J2EE

  • Not Linux ?
  • BSD
  • HP-UX
  • Mac
  • Solaris
  • VM
  • Windows
  • /pub/OS

  • Other
  • /pub
  • /pub/3rdParty
  •  Parent Directory

    Linux Application Profiling - GNU
    Brett Lee
    ======================================
    
    GNU is great, but it has only gone so far.
    Think Intel or Portland Group compilers.
    
    Topics
    ---------------------------------
    
    
    1.  Binutils
    
    2.  Debugging - GNU
        - GNU gdb
        - Stuff U can use
    
    3.  Profiling - GNU
        - GNU gprof
        - GNU gcov
    
    4.  Profiling - Linux
        - OProfile
        - PAPI
        - Perfmon, PerfCTR, SystemTap, etc.
        - Valgrind
        - ftrace
        - strace
    
    5.  Profiling - Other OS's and Applications
        - Other Applications
    
    
    Binutils:
    ===============================
    
    A good place to start is reading the man page for each of these:
    
       ~> rpm -ql binutils  | grep -v share
       /usr/bin/addr2line
       /usr/bin/ar
       /usr/bin/as
       /usr/bin/c++filt
       /usr/bin/gprof
       /usr/bin/ld
       /usr/bin/nm
       /usr/bin/objcopy
       /usr/bin/objdump
       /usr/bin/ranlib
       /usr/bin/readelf
       /usr/bin/size
       /usr/bin/strings
       /usr/bin/strip
    
       See:
        binutils - http://sourceware.org/binutils/docs/
    
    
    Background (Debugging)
    ===============================
    
    
    Preprocess, Compile, Assemble, Link
    -------------------------------------
    
    From: 'gcc --help':
    
      -E                       Preprocess only; do not compile, assemble or link
      -S                       Compile only; do not assemble or link
      -c                       Compile and assemble, but do not link
      -o <file>                Place the output into <file>
    
    
    
    gdb - GNU debugging:
    -------------------------------------
    
    Start here:
    
       http://sourceware.org/gdb/current/onlinedocs/gdb/
    
    
    
    * Compile without gdb:
    *************************************************
    
    ~>g++ hellogdb.C -o hellogdb
    
    
    * Look at the headers:
    ~>objdump -x hellogdb
    
    hellogdb:     file format elf64-x86-64
    hellogdb
    architecture: i386:x86-64, flags 0x00000112:
    EXEC_P, HAS_SYMS, D_PAGED
    start address 0x0000000000400750
    
    Program Header:
    ...
    ...
    Sections:
    Idx Name          Size      VMA               LMA               File off  Algn
      0 .interp       0000001c  0000000000400200  0000000000400200  00000200  2**0
                      CONTENTS, ALLOC, LOAD, READONLY, DATA
      1 .note.ABI-tag 00000020  000000000040021c  000000000040021c  0000021c  2**2
                      CONTENTS, ALLOC, LOAD, READONLY, DATA
      2 .gnu.hash     00000030  0000000000400240  0000000000400240  00000240  2**3
                      CONTENTS, ALLOC, LOAD, READONLY, DATA
      3 .dynsym       00000138  0000000000400270  0000000000400270  00000270  2**3
    ...
    <cut for brevity>
    ...
     22 .data         00000004  0000000000600e60  0000000000600e60  00000e60  2**2
                      CONTENTS, ALLOC, LOAD, DATA
     23 .bss          00000120  0000000000600e70  0000000000600e70  00000e64  2**4
                      ALLOC
     24 .comment      00000114  0000000000000000  0000000000000000  00000e64  2**0
                      CONTENTS, READONLY
    SYMBOL TABLE:
    0000000000400200 l    d  .interp        0000000000000000              .interp
    ...
    ...
    
    
    
    * Now compile with gdb and check the headers
    *************************************************
    
    ~>g++ hellogdb.C -o hellogdb -ggdb
    ~>objdump -x hellogdb
    ...
    <cut for brevity>
    ...
     22 .data         00000004  0000000000600e60  0000000000600e60  00000e60  2**2
                      CONTENTS, ALLOC, LOAD, DATA
     23 .bss          00000120  0000000000600e70  0000000000600e70  00000e64  2**4
                      ALLOC
     24 .comment      00000114  0000000000000000  0000000000000000  00000e64  2**0
                      CONTENTS, READONLY
     25 .debug_aranges 00000030  0000000000000000  0000000000000000  00000f78 2**0
                      CONTENTS, READONLY, DEBUGGING
     26 .debug_pubnames 0000001b  0000000000000000  0000000000000000  00000fa8 2**0
                      CONTENTS, READONLY, DEBUGGING
     27 .debug_info   00006807  0000000000000000  0000000000000000  00000fc3  2**0
                      CONTENTS, READONLY, DEBUGGING
     28 .debug_abbrev 0000072a  0000000000000000  0000000000000000  000077ca  2**0
                      CONTENTS, READONLY, DEBUGGING
     29 .debug_line   000003e9  0000000000000000  0000000000000000  00007ef4  2**0
                      CONTENTS, READONLY, DEBUGGING
     30 .debug_frame  000000a0  0000000000000000  0000000000000000  000082e0  2**3
                      CONTENTS, READONLY, DEBUGGING
     31 .debug_str    0000073a  0000000000000000  0000000000000000  00008380  2**0
                      CONTENTS, READONLY, DEBUGGING
     32 .debug_loc    00000130  0000000000000000  0000000000000000  00008aba  2**0
                      CONTENTS, READONLY, DEBUGGING
    SYMBOL TABLE:
    0000000000400200 l    d  .interp        0000000000000000              .interp
    
    
    *** See the additional .debug_* sections for gdb to work with ??? ***
    
    
    
    * Could it get any worse?  Yep, assembly:
    *********************************************
    
    ~>objdump -d hellogdb 2>&1 | head -15
    
    hellogdb:     file format elf64-x86-64
    
    Disassembly of section .init:
    
    0000000000400690 <_init>:
      400690:       48 83 ec 08             sub    $0x8,%rsp
      400694:       e8 e3 00 00 00          callq  40077c <call_gmon_start>
      400699:       e8 62 01 00 00          callq  400800 <frame_dummy>
      40069e:       e8 9d 03 00 00          callq  400a40 <__do_global_ctors_aux>
      4006a3:       48 83 c4 08             add    $0x8,%rsp
      4006a7:       c3                      retq
    Disassembly of section .plt:
    
    
    
    
    
    
    Stuff U can use:
    -------------------------------------
    
    The key to the above cryptic stuff is that there is sense to the binary.
    Learn more about the format of ELF and this begins to make sense.
    
    For example:
      * Compiling turns the source code into machine code.
        - After compiling, it is not ready to be run.  It still needs to be linked.
        - But if it is machine code, how does the linker know what to "link"?
        - And what about clashes with memory locations when combining multiple
          object files that all use the same address?
        - Or any of the other issues, like the names of functions?
      * Recall the seemingly cryptic output from 'objdump'.  There was:
          - Program Header
          - Sections
          - Symbol Table
        - These things listed above provide that sort of information.
        - For example, the Symbols are maps between names and memory locations.
          - Thus without the symbols, its pretty hard to do much of anything.
          - Note that the symbols can be in the binary or in a seperate file.
            - The Linux kernel is typically delivered with the symbols in:
              /boot/System.map
    
    ** For a practical demonstration of using symbols, take a look at what happens
       when the sysbols are there, and when they are not.
    
       - This example attempts to set a breakpoint in "main":
    
         # With symbols:
         # -------------------
         ~>ls
         hellogdb  hellogdb.C
         ~>gdb hellogdb
         (gdb) b main
         Breakpoint 1 at 0x4008a4: file hellogdb.C, line 11.
         (gdb) q
    
    
         # Strip symbols:
         # -------------------
         ~>strip hellogdb
         ~>objdump -d hellogdb
    
         ~>objdump -t hellogdb
         hellogdb:     file format elf64-x86-64
    
         SYMBOL TABLE:
         no symbols
    
    
         # Without symbols:
         # -------------------
         ~>gdb hellogdb
         Reading symbols from .../hellogdb...(no debugging symbols found)...done.
         (gdb) b main
         Function "main" not defined.
         Make breakpoint pending on future shared library load? (y or [n])
    
         *** Without the symbols, the debugger doesn't know where "main" is !!!
    
    
       - Provide symbols via a file:
         ------------------------------
    
         GNU gdb *does* provide support for debugging stripped files.
         The symbol file must be loaded seperatly.  This means that the symbol
         file must have been created prior to stripping the file, and that you must
         have the file.  This can be done one of two ways:
    
         1.  have the symbol file in the same dir with the extension ".syms"
         2.  load using -s argument
    
         # Load symbol file:
         # -------------------
         ~> g++ hellogdb.C -o hellogdb -ggdb
         ~> nm -Ca hellogdb | c++filt > hellogdb.syms
         ~> strip hellogdb
         ~> gdb hellogdb
    
         Or if another name:
         ~> gdb -s symbolfile -readnow hellogdb
    
    
      See:     
        http://gcc.gnu.org/onlinedocs/gcc/
        http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html
        http://www.gnu.org/software/gdb/documentation/
        http://www.network-theory.co.uk/docs/gccintro/
        http://www.cs.umd.edu/class/spring2001/cmsc420/gdb.html
    
    
    
    ####################################################
    
    
    2. Profiling 
    ====================
    
    
    gprof - GNU profiling
    -----------------------
    
      GNU gprof works at the application level.
    
      - This means that gprof traces the execution only as far as the
        border of the application.
      - Does it look at libraries?  No.
      - Does it look at system calls?  No.
      - Does it look inside the kernel?  No.
      - Does it look at functions in the application?  Yes.
    
    
      HOWTO:
    
      1.  Compile the program with the "-pg" option.
      2.  Execute the program (this creates a 'gmon.out' file).
      3.  Use gprof to interpolate between the output and the binary.
           ~> gprof <binary> gmon.out
           or:
           ~> gprof <binary> gmon.out > profileData.out'
    
    
      Sample:
    
      Results indicate the amount of time and # calls for each function.
      Like:
    
        Each sample counts as 0.01 seconds.
          %   cumulative   self              self     total
         time   seconds   seconds    calls  ms/call  ms/call  name
         80.14      0.04     0.04        1    40.07    45.08  do_amort
         10.02      0.05     0.01   360001     0.00     0.00  round
         10.02      0.05     0.01                             main
    
    
      See:
          http://sourceware.org/binutils/docs/gprof/
          http://www.ibm.com/developerworks/library/l-gnuprof.html
    
    
    
    
    
    gcov - GNU coverage
    ----------------------------
    
       gcov builds upon gprof
         - gprof provides per-function statistics
         - gcov provides per-line statistics
         - gprof used to do this in old versions of gcc
    
       Useful when:
         - Seeking to optimize the application for performance
           - Determine which lines are executed the most
         - Testing an application
            - determine if, and how many times the tests ran that line
            - finding code that is not ever executed 
    
       Limitations include:
         - Works only with GCC
         - Works best with specific programming styles
    
       gcov-kernel:
         - As you might imagine, gcov sounds ideal for the Linux kernel
         - Well, enter gcov-kernel:
           - http://ltp.sourceforge.net/coverage/gcov.php
    
    
       See:
         http://gcc.gnu.org/onlinedocs/gcc/Gcov.html
         http://ltp.sourceforge.net/coverage/gcov.php
         http://lwn.net/2002/0207/a/gcov-kernel.php3
         http://www.mjmwired.net/kernel/Documentation/gcov.txt
         http://sourceware.org/binutils/docs-2.20/gprof/Line_002dby_002dline.html#Line_002dby_002dline
    
    
    
    OProfile (for Linux)
    ----------------------------
    
      - OProfile is a system wide profiler.
        - It characterizes both the system and the application.
      - It does not require that the application be compiled with profiling.
      - It works via a kernel module.
    
    
      ~> rpm -qa | grep -i oprof
         oprofile-0.9.4-15.el5
      ~> rpm -ql `rpm -qa | grep -i oprof` | grep -v share
         /usr/bin/opannotate
         /usr/bin/oparchive
         /usr/bin/opcontrol
         /usr/bin/opgprof
         /usr/bin/ophelp
         /usr/bin/opimport
         /usr/bin/opjitconv
         /usr/bin/opreport
         /usr/bin/oprofiled
      ~> grep oprofile /lib/modules/2.6.18-8.el5/modules.dep
         /lib/modules/2.6.18-8.el5/kernel/arch/x86_64/oprofile/oprofile.ko:
      ~> rpm -q --whatprovides /lib/modules/2.6.18-8.el5/kernel/arch/x86_64/oprofile/oprofile.ko
         kernel-2.6.18-8.el5
      ~# insmod /lib/modules/2.6.18-8.el5/kernel/arch/x86_64/oprofile/oprofile.ko
      ~# lsmod | grep oprofile
         oprofile              139953  0
      ~# rmmod oprofile
    
    
      Usage is provided in the documentation.
    
    
      See:
        Usage:   http://oprofile.sourceforge.net/doc/index.html
        Example: http://www.novell.com/coolsolutions/feature/11789.html
    
    
    PAPI !
    ----------------------------
    http://icl.cs.utk.edu/papi/faq/
    
    
    Perfmon (for Linux)
    ----------------------------
      - Perfmon - http://perfmon2.sourceforge.net
      - Used to access the hardware performance counters
    
    
    PerfCTR (for Linux)
    ----------------------------
    
    
    PerfSuite
    ----------------------------
    
    
    SystemTap 
    ----------------------------
      See:
        http://sourceware.org/systemtap/wiki
    
    
    
    
    ftrace
    ----------------------------
    
    
      See:
        http://sourceware.org/systemtap/wiki
    
    
    
    
    Other OS's and Applications
    ----------------------------
    
    OS's:
    
        Solaris, BSD and HP-UX* use ELF binaries
        This simplifies things - especially for GNU binaries
        Ok, not all HP-UX...
    
    
        Solaris
          - Use 'DTrace'
    
        HP-UX:
          - Use 'Glance'
    
        BSD's:
          - er, GCC  :)
    
    
    Applications:
    
        Java Performance
    
          err, how about OProfile... :)
    
          See:
            http://java.sun.com/docs/books/performance/
            http://java.sun.com/docs/books/performance/1st_edition/html/JPTOC.fm.html
    
    
    Linux Kernel:
    
        Use kdump:
    
        http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-kdump.html
        http://debuginfo.centos.org/5/i386/
    
    
    
    
    

    Other Sites

    RFC's
  • FAQ's
  • IETF
  • RFC Sourcebook

  • Linux
  • Linux - Intro
  • Linux Kernel
  • Linux Kernel (LKML)
  • Bash - Intro
  • Bash - Advanced
  • Command Line
  • System Administration
  • Network Administration
  • Man Pages (& more)
  • More Guides
  • Red Hat Manuals
  • HOWTO's

  • Reference/Tutorials
  • C++ @ cppreference
  • C++ @ cplusplus
  • CSS @ echoecho
  • DNS @ Zytrax
  • HTML @ W3 Schools
  • Java @ Sun
  • LDAP @ Zytrax
  • Linux @ YoLinux
  • MySQL
  • NetFilter
  • Network Protocols
  • OpenLDAP
  • Quagga
  • Samba
  • Unix Programming



  • This site contains many of my notes from research into different aspects of the Linux kernel as well as some of the software provided by GNU and others. Thouugh these notes are not fully comprehensive or even completetly accurate, they are part of my on-going attempt to better understand this complex field. And, they are your to use.

    Should you wish to report any errors or suggestions, please let me know.

    Should you wish to make a donation for anything you may have learned here, please direct that donation to the ASPCA, with my sincere thanks.

    Brett Lee
    Everything Penguin

    The code for this site, which is just a few CGI scripts, may be found on GitHub (https://github.com/userbrett/cgindex).

    For both data encryption and password protection, try Personal Data Security (https://www.trustpds.com).


    "We left all that stuff out. If there's an error, we have this routine called 'panic', and when its called, the machine crashes, and you holler down the hall, 'Hey, reboot it.'"

        - Dennis Ritchie on Unix (vs Multics)


    Google
    [ Powered by Red Hat Linux ] [ Powered by Apache Server] [ Powered by MySQL ]

    [ Statistics by AWStats ]