Happy Monday!

Monday’s are wonderful, aren’t they?  I mean, it’s awesome to walk in first thing in the morning and find things are already broken.  The gremlins were working overtime this weekend, I see.

I get in and I’ve not even logged into my PC yet and one of the desktop techs approaches me about a possible user port being down.  So he waits patiently at my door, chatting with other folks outside my office as I finish logging in and trying to get into the switch remotely to see what’s going on.  Unfortunately, I cannot get into the switch.  I can ping it, but telnet and web access weren’t working. 

Ugh.

Redundancy, Redundancy, Redundancy…I’ll say it again Redundancy.

The rep from my ISP just called and asked me how my Internet and WAN connections were doing.  He seemed taken aback by my answer of just fine.  Apparently, someone cut some fiber somewhere and took out our entire county.

Except us.

I had a hunch this call would be coming.  My CIO had been waiting at my office when I walked in the door this morning (a little late, too!  **gulp**).  She asked a very similar question, but I informed her that from what I could tell everything was working just fine.  I’d not received any automated outage notifications either from our ISP or from any of our internal network monitoring devices.  She’d heard from some collegues at the local community college that they were down and they wanted to know our status.  I told her I’d look at our systems and let her know, but at the moment all is working and as far as I could tell we never went down, even for a blip.

The Worst Day Since Yesterday

Thursday was a very long day at work.  It seemed that everything wanted to happen all at once.  You know how those days go, I’m sure.  So, to make me feel better, I’m naming this blog after one of my favorite songs that I first heard on one of my favorite shows.

The day started with a plan, actually.  I’d noticed that the NAT rules on our core router were not quite functioning properly several weeks ago.  I’ve got multiple pools configured for different groups of systems (mainly servers, virtual servers, and clients, but also for each of our remote sites).  Apparently when I put this all together I fudged the ACLs that assign the IPs to those pools somehow, because I’ve noticed that some clients will translate properly, but others will translate into the server pools instead of the client ones.  Not a huge deal as far as access goes, since a public IP is a public IP and the client will be able to get to the Internet.  But the server pool has more privileges on our DMZ than the clients, so it could be a small security risk, if an employee wanted to do something they shouldn’t.

When the issue arose, I recreated the config in our test lab, recreated and confirmed the issue, and fixed the config to prevent the issue.  Problem solved, right?