IOS Upgrade

According to Guy Almes, “There are three kinds of death in this world.  There’s heart death, there’s brain death, and there’s being off the network.”

Over the past couple of weeks I’ve been forced to upgrade and downgrade the IOS for my core switch several times because of a bug in the code.

I’ve been on pins and needles the entire time since at any time a simple interface reset would cause the switch to reboot, bringing my entire network to a screetching halt.  Obviously that is not a good thing, but all I could do was wait on Cisco to come through with a new IOS that should fix the bug.

Buggy

A while back I posted about Undocumented Features.  Well Cisco recommended we upgrade the IOS on our core router to see if that would alleviate the bug. Our old IOS was version 12.2(33)SXH5.  The new one was 12.2(33)SXH7.

We did the upgrade last Thursday but it seemed to not fix anything. I still saw the”static NATs” building in the translation table.

Oh well. I figured at least we’re at a new rev. We’ll just have to troubleshoot some more to find the true problem.

Cue the next day. At lunch we were working on configuring our new wireless controller. The last bit of configuration we needed to do would bounce the port-channel interface between the controller and the core router.

We had permission to do this during lunch because it was Friday and there are always fewer people online. Besides, we were only affecting the wireless network in one building. What could go wrong?

Well the high resurfaced in a big way. When the port-channel interface bounced, it rebooted the entire core router!

Twice…

Undocumented Features

Upon returning to work following my “vacation,” I dove right back into business as usual.  I needed to add a static NAT to my core router to support access to an external web resource.  No big deal.

• Choose and reserve an appropriate IP in our database – Check
• Add the new command to our configuration documentation – Check
• Enter the new command in the core router – Uh, Houston we have a problem!

I got a %Memory Not Available error.  What?!  It made no sense, so I started troubleshooting.

The Worst Day Since Yesterday

Thursday was a very long day at work.  It seemed that everything wanted to happen all at once.  You know how those days go, I’m sure.  So, to make me feel better, I’m naming this blog after one of my favorite songs that I first heard on one of my favorite shows.

The day started with a plan, actually.  I’d noticed that the NAT rules on our core router were not quite functioning properly several weeks ago.  I’ve got multiple pools configured for different groups of systems (mainly servers, virtual servers, and clients, but also for each of our remote sites).  Apparently when I put this all together I fudged the ACLs that assign the IPs to those pools somehow, because I’ve noticed that some clients will translate properly, but others will translate into the server pools instead of the client ones.  Not a huge deal as far as access goes, since a public IP is a public IP and the client will be able to get to the Internet.  But the server pool has more privileges on our DMZ than the clients, so it could be a small security risk, if an employee wanted to do something they shouldn’t.

When the issue arose, I recreated the config in our test lab, recreated and confirmed the issue, and fixed the config to prevent the issue.  Problem solved, right?