ARP cache - Why don't my entries expire?
Brett Lee
==============================================================================


You should know that ARP is the communication protocol used to obtain a 
MAC address.  MAC addresses are used in addressing an Ethernet
frame so that it will go from host to host.  MAC addresses are used on
a single subnet and are not routable.

You should also know that ARP entries are maintained in a cache so that
they do not have to be obtained every time a transmission occurs.  The
entries do not stay there forever, but instead they expire and are removed
from the cache.

But some seem to expire faster than others.  In this brief we'll look at
some of the reasons why this might occur.


This is my ARP cache:

[root@linux test]# arp -vni eth1
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.0.111            ether   00:1B:24:0C:C4:F6   C                     eth1
192.168.0.1              ether   00:10:4F:07:6A:9B   C                     eth1
192.168.10.11            ether   00:E0:29:84:5C:AB   C                     eth1
192.168.10.111           ether   00:1B:24:0C:C4:F3   C                     eth1
192.168.0.121            ether   00:1B:24:5B:DA:27   C                     eth1
192.168.10.121           ether   00:1B:24:5B:DA:24   C                     eth1
192.168.100.131          ether   00:1B:24:0C:C6:BA   C                     eth1
192.168.10.131           ether   00:1B:24:0C:C6:B7   C                     eth1
Entries: 9      Skipped: 1      Found: 8


The eight eth1 entries are displayed.  The lone eth0 entry is not.
The ARP entries will expire and be removed when "gc_stale_time" is reached.

Kinda.

That is, the "timeout counter" begins for each entry in the ARP cache after
it is last used.  This makes sense as you don't want to have to keep ARPing
for a MAC address of a host that you are in constant communication.  Its like
talking to your friend and saying, BTW, who are you again?

So lets say you don't talk to your "friend" for longer than "gc_stale_time"
(typically 60 seconds).  The ARP entry does not just disappear, but instead
a garbage collection process must reap the entry from the cache.  This reaper
also has a timer (gc_interval) associated with it, so the first run of
the reaper after gc_stale_time is reached would remove the entry.

Kinda.

What am I talking about?  Well, not all of the "expired" ARP cache entries will
get removed by the reaper.  One reason might be that your system uses a route
cache.  If the expired entry in the ARP cache is also an entry in the route cache,
it will not be removed from the ARP cache.  It first must expire from the
routing cache, be reaped from it, and then they can be reaped from the ARP
cache.

So how do we look at the route cache.  A couple ways:

[root@linux test]# ip -s route show cache
192.168.100.131 from 192.168.10.121 dev eth1  src 192.168.10.254
    cache <src-direct,redirect>  users 1 used 2 age 80sec mtu 1500 advmss 1460 iif eth1
...

or

[root@linux tmp]# route -Cn
Kernel IP routing cache
Source          Destination     Gateway         Flags Metric Ref    Use Iface
172.16.177.10   192.168.10.121  192.168.10.121        0      0        0 eth1
192.168.10.121  192.168.10.254  192.168.10.254  il    0      0       98 lo
172.16.3.44     192.168.10.11   192.168.10.11         0      0       86 eth1
192.168.10.131  192.168.0.111   192.168.0.111   ri    0      0        2 eth1
192.168.10.10   192.168.10.255  192.168.10.255  ibl   0      0        1 lo
192.168.10.10   192.168.10.254  192.168.10.254  il    0      0       33 lo
...

Yeah, so if the address is a gateway and it is expired from the routing cache,
is past the gc_stale_time and the reapers' gc_interval has been reached, the
entry is gonna be deleted for sure, right?

Kinda.

One more gotch is that the entry MIGHT just be a permanent entry in the ARP
cache.  Yep, you can add a fixed, unremovable entry using the "ip" command:

[root@linux tmp]# ip neigh add 10.0.0.3 lladdr 0:0:0:0:0:1 dev eth0 nud perm
[root@linux tmp]# ip neigh chg 10.0.0.3 dev eth0 reachable


OK, So finally we're ready to delete the ARP cache, right?

Here we go:

[root@linux tmp]# arp -d *
arpcache: Unknown host


Arrgh, it doesn't work.  Lets try this:

[root@linux tmp]# arp -d 192.168.0.111; arp -vn
Address            HWtype  HWaddress        Flags Mask      Iface
192.168.0.111              (incomplete)                     eth1
192.168.0.1        ether   00:10:4F:07:6A:9B   C            eth1
192.168.10.11      ether   00:E0:29:84:5C:AB   C            eth1
192.168.10.111     ether   00:1B:24:0C:C4:F3   C            eth1
192.168.0.121      ether   00:1B:24:5B:DA:27   C            eth1
172.16.1.254       ether   00:07:B4:02:7F:02   C            eth0
192.168.10.121     ether   00:1B:24:5B:DA:24   C            eth1
192.168.100.131    ether   00:1B:24:0C:C6:BA   C            eth1
192.168.10.131     ether   00:1B:24:0C:C6:B7   C            eth1
Entries: 9      Skipped: 0      Found: 9


Arrgh2, I don't want to do that for each one!

Lets try try this:

[root@linux tmp]# ip neigh flush all; arp -vn; cat /proc/net/arp
Address            HWtype  HWaddress        Flags Mask      Iface
192.168.0.111              (incomplete)                     eth1
192.168.0.1                (incomplete)                     eth1
192.168.10.11              (incomplete)                     eth1
192.168.10.111             (incomplete)                     eth1
192.168.0.121              (incomplete)                     eth1
172.16.1.254               (incomplete)                     eth0
192.168.10.121             (incomplete)                     eth1
192.168.100.131            (incomplete)                     eth1
192.168.10.131             (incomplete)                     eth1
IP address       HW type     Flags       HW address            Mask     Device
192.168.0.111    0x1         0x0         00:1B:24:0C:C4:F6     *        eth1
192.168.0.1      0x1         0x0         00:10:4F:07:6A:9B     *        eth1
192.168.10.11    0x1         0x2         00:E0:29:84:5C:AB     *        eth1
192.168.10.111   0x1         0x0         00:1B:24:0C:C4:F3     *        eth1
192.168.0.121    0x1         0x0         00:1B:24:5B:DA:27     *        eth1
172.16.1.254     0x1         0x0         00:07:B4:02:7F:01     *        eth0
192.168.10.121   0x1         0x0         00:1B:24:5B:DA:24     *        eth1
192.168.100.131  0x1         0x0         00:1B:24:0C:C6:BA     *        eth1
192.168.10.131   0x1         0x0         00:1B:24:0C:C6:B7     *        eth1


Well, thats a little bit better, but we see two things:

1) The cache didn't actually flush (according to "arp -vn").  Instead,
all we saw was that the MAC addresses were deleted.

2) The ARP entries in the /proc filesystem didn't change.


Deleting these ARP entries is pretty tough row to hoe. :)

Good luck!
-Brett

Resource:
IP Command Reference - Alexey N. Kuznetsov
  /usr/share/doc/iproute-2.4.7/ip-cref.ps