Buggy

A while back I posted about Undocumented Features.  Well Cisco recommended we upgrade the IOS on our core router to see if that would alleviate the bug. Our old IOS was version 12.2(33)SXH5.  The new one was 12.2(33)SXH7.

We did the upgrade last Thursday but it seemed to not fix anything. I still saw the”static NATs” building in the translation table.

Oh well. I figured at least we’re at a new rev. We’ll just have to troubleshoot some more to find the true problem.

Cue the next day. At lunch we were working on configuring our new wireless controller. The last bit of configuration we needed to do would bounce the port-channel interface between the controller and the core router.

We had permission to do this during lunch because it was Friday and there are always fewer people online. Besides, we were only affecting the wireless network in one building. What could go wrong?

Well the high resurfaced in a big way. When the port-channel interface bounced, it rebooted the entire core router!

Twice…

Holy Crap! I just bright down the entire network. Remote sites. Internet. Our website. Everything.

Granted it was only down for about 10 minutes total, but you’d be surprised how many people actually work thru their lunch based on the number of calls I got.

To make matters a little worse, the CIO and Assistant CIO were on the road. They had decided to check in on a nearby remote site right at the time the reboot happened.

So I gather all the dump files from the router and send them off to Cisco. They come back and tell me the problem IS a bug and it deals with NAT, just like the old IOS. Looks like the upgrade actually made things worse.

So Cisco gave us several options. Apply the workarounds, which essentially meant re-engineer our entire NAT infrastructure. Out of the question. Or wait for the eventual release of a new IOS with the fix in it, which is vapor-ware since they could only estimate a release date. Or fall-back to the old release which won’t solve the underlying NAT issue. Or request a special engineering build of the IOS that would take a few days to make but would address the underlying bug until the public release was finished.

The last option is what we chose. Now it’s the wait game to see when this special build will actually be complete.

For the moment we’re just not going to do anything that would cause any interfaces to bounce.

I’ll let you know how things turn out.