Russ Garrett

A quick note on the black art of Cisco configuration. Conveniently the Catalyst 6500 series (and likely higher models which use dCEF) has a different method of configuring NetFlow from lower-end switches. The Cisco docs don’t really touch on why this is. (This guide is based on IOS 12.2(33)SXH on the Sup720. Your mileage most likely will vary.)

So, firstly enable NetFlow like you would on any other IOS switch. It’s worth noting that at some point during the configuration you’ll likely get one of those trademark heart-stopping console freezes for up to 20 seconds. It’s not clear if this actually interrupts switching.

switch(config)#interface Te2/2
switch(config-if)#ip flow ingress
switch(config-if)#ip flow egress

I understand that this command used to be called ip route-cache flow, just to add to the confusion.

Now enable NDE to export your data to something like flow-tools:

switch(config)#ip flow-export source Vlan1
switch(config)#ip flow-export version 5 origin-as # This is where it hangs a while...
switch(config)#ip flow-export destination x.x.x.x yyyy

At this point you can run sh ip flow export to see your many flows being exported. Well, except you can’t, because on the 6500, the ip flow class of commands only deal with NetFlow for packets which hit the supervisor module, i.e. forwarding cache misses. (Older cat6500 hardware would merit a discussion of MSFCs and PFCs here, but my hardware isn’t old, so we don’t need that complication.)

So, to enable NetFlow and NDE for dCEF switched packets throughout the switch, the appropriate incantations are done using the mls series of commands:

switch(config)#mls netflow
switch(config)#mls nde sender version 5
switch(config)#mls flow ip interface-full

Confusingly, although it uses the NDE collector you configured earlier, you must view the MLS NDE stats differenlty, by using sh mls nde.

More detail can be found in the Configuring Netflow section of the Catalyst 6500 config guide.

Tags: catalyst, cisco, ios, Networking.

We recently had an issue where one of two fibre pairs between one of our core switches and a new high-capacity edge rack got nudged a little too violently, then started throwing errors in one direction. Annoyingly, this didn’t get detected, and since it was a load-balanced link connectivity seemed fine.

Until we moved one of our user charts machines, which turns out to be quite sensitive to packet loss, into the rack. When we hit peak traffic (just about time to go to the pub), it started to time out, and we started serving and rendering the wrong charts in the wrong places. People hate it when that happens.

The tool for consistently reproducing these problems is already built into every modern Linux distribution. The traffic-shaping system includes the netem module which provides a huge array of network emulation possibilities. In our case, it was as simple as:

# tc qdisc change dev eth0 root netem loss 5%

We re-ran our test and it failed first time.

For a detailed reference on Linux network emulation see this documentation.

Tags: linux, netem, Networking, testing.