Back in May I wrote a post, What’s In a Checkbox.
Just what’s in a checkbox? That is the question I was asking myself after 6+ hours and 5 engineers finally figured out 1 little checkbox needed to be unchecked for an entire phone system to start working.
Well, this time I must have won some sick and twisted lottery of suffering!
As many of my friends and colleagues know, my company was recently acquired. As part of the arrangement, much of our existing IT infrastructure is be decommissioned over the next 2 years. While this fact is bittersweet, the task of actually accomplishing this is nothing if not bitter.
By this I simply mean the laborious tasks of undoing and redoing everything that has been built over the past 5 years, while continuing to take care of day-to-day operations, is a tall order.
The first move was our corporate web site, and this went very smoothly thanks to professionals on 3 different teams working together.
Next on the agenda was to transfer all e-mail traffic to corporate starting after week 2 of the acquisition. It was decided we would use Microsoft Outlook’s RPC over HTTPS option to connect via the Internet to the corporate Exchange server. This would minimize the impact to end-users as we were already using the same configuration for the past 2 years in order to create a convenient and secure tunnel for e-mail traffic without need of slow VPN connections.
Well, things were moving fast. We had migrated 1/3 of our internal customers within a few days – and then HTTPS traffic ground to a halt on our firewall.
What does this mean?
Not only did e-mail quit working – all traffic, going to secure web sites, was out of service.
We engaged resources from all sides:
-
Myself, a Branch Network Administrator and Software Engineer
-
Another Branch’s Network Administrator and Technology Manager
-
A corporate Exchange Administrator, WAN Engineer, and Project Manager
-
2 WatchGuard Engineers
-
1 ISP WAN Engineer
-
and finally 1 Cisco Engineer
We ran studies, and tried this and that, and spent days and days trying to figure out just what was happening. Countless hours on phone conversations yielded little to no results. Finally, we opted to go with another plan: connecting a VPN tunnel to corporate, a recommendation made by a sister branch.
Security assessments had to be completed to ensure that attaching our foreign network to the corporate WAN wouldn’t disrupt network performance or wouldn’t create any unnecessary security risks.
Our Cisco Engineer was called in to talk on the painful process of reclassifying our entire network, or if simpler steps could be taken. In the process of talking through this, I asked if he was familiar with our firewall. Luckily, he had worked with them before, and had a friend that really knew his way around.
He attached his notebook computer to our network, simply to ensure he was seeing the same thing we did, and sure enough he did. He then proceeded to walk through our configuration with his friend and landed on 1 checkbox that might just fix the problem.
He unchecked the box (1:1 NAT for the HTTPS service, if anyone is interested), and traffic instantly began flowing – like a river had been turned loose from its dam.
One checkbox made the difference, and my question was that after 3 weeks and a slew of experts looking at the problem, why did 1 checkbox save the day? Why didn’t my firewall vendor even recommend this after talking with 2 of their engineers? Were they not listening to the problems I was describing? Could they not see the configuration and traffic from themselves (via their remote monitoring tools)? Did I simply not have them looking at the right symptoms?
Why did it take 1 man less than 2 hours to figure it out when so many could not figure it out? Why did our sister division (with the same network configuration) not even see it 8 months prior?
It just goes to show you, who you know is much more important than what you know!
Ken Stewart’s blog, ChangeForge.com, focuses on the collision between the constantly changing worlds of business and technology. To learn more about Ken, visit his about page. You may also find Ken on FriendFeed, Twitter, and LinkedIn.
