Time Management for System Administrators

It's no replacement for a good malware or virus scanner. However, I picked this example because it is a good demonstration of some rudimentary shell-programming techniques, and you'll learn something about networking, too. When we're done, you'll have a simple tool you can use on your own network to detect this particular problem. I've used this tool to convince management to purchase a real virus scanner.

What's one sign that a machine is infected with some kind of worm? How about a quick test to see which machines are ARPing the most?

Spyware/worms/virii often try to connect to randomly selected machines on your network. When a machine tries to talk to a local IP address for the first time, it sends an ARP packet to find out its Ethernet (MAC) address. On the other hand, normal (uninfected) machines generally talk to a few machines only: the servers they use and their local router. Detecting a machine that is sending considerably more ARP packets than other machines on the network is often a sign that the machine is infected.

Let's build a simple shell pipeline to collect the next 100 ARP packets seen on your network and determine which hosts generated more ARP packets than their peers. It's sort of a 'most likely to ARP' award. The last time I did this on a 50-host network, I found 2 machines infested with worms.

These commands should work on any Unix/Linux or Unix-like system. You will need the tcpdump command and root access. The command which tcpdump tells you if you have tcpdump installed. Sniffing packets from your network has privacy concerns. Only do this if you have permission.

Here's the final command that I came up with (sorry to spoil the surprise): $ sudo tcpdump -l -n arp | grep 'arp who-has' | head -100 | awk '{ print $NF }' |sort | uniq -c | sort -n

The command is too long to fit on one line of this book, so I put a backslash at the end of the first part to continue it across two lines. You don't have to type the backlash, and you shouldn't press Enter in its place.

The output looks like this: tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en0, link-type EN10MB (Ethernet), capture size 96 bytes 1 192.168.1.104 2 192.168.1.231 5 192.168.1.251 7 192.168.1.11 7 192.168.1.148 7 192.168.1.230 8 192.168.1.254 11 192.168.1.56 21 192.168.1.91 30 192.168.1.111 101 packets captured 3079 packets received by filter 0 packets dropped by kernel

Ignore the headers. The middle lines show a count followed by an IP address. During my experiment, host 192.168.1.111 sent 30 ARP packets, while 192.168.104 only sent 1. Most machines rarely ARPed in that time period, but two hosts had four to six times as many ARPs as some of the other machines! Those were my two problem children. A quick scan with some anti-virus software and they were as good as new.

Here's how I built this command line. I started with this command: $ sudo tcpdump -l -n arp

sudo means to run the next command as root. It will most likely ask for a password. If you don't use sudo in your environment, you might use something like it, or you can run this entire sequence as root. Just be careful. To err is human; to really screw up, be careless with root.

tcpdump listens to the local Ethernet. The -l flag is required if we're going to pipe the output to another program because, unlike other programs, tcpdump does something special with output buffering so that it runs faster. However, when piping the output, we need it to act more normal. The -n means don't do DNS lookups for each IP address we see. The arp means that we only want tcpdump to display ARP packets.

(If you are concerned about privacy of your network, I'd like to point out some good news. There isn't much private data available to your eyes if, at the sniffing end, you filter out everything besides ARP packets.)

Run the command yourself. In fact, you will learn more if you try each command as you read this. Nothing here deletes any data. Of course, it may be illegal to snoop packets on your network, so be warned. Only do this on a network where you have permission to snoop packets.