ISA 2004 Lockdown Mode Issue (Cont’d)

ryanwetmore — Mon, 22 Jun 2009 19:51:37 +0000

I finally got a callback from Microsoft today for our ISA 2004 Lockdown issue. The support engineer told me that the logs that he collected pointed to the Web Proxy component of ISA on one of our firewalls was causing it not to start the firewall services. We rummaged around the ISA settings looking for the “HTTP Filter” in the Enterprise Add-Ins section of the ISA console. We noticed that this was disabled so we enabled it. Went to the Sevices MMC and tried to restart the firewall services. No go…. Same error message.

Next we looked through the NTFS permissions in the “Microsoft ISA Server” directory on the C: drive. All permissions seemed to be correct and not modified.

I downloaded the latest ISA 2004 patch from KB article 954264 (http://support.microsoft.com/kb/954264) as per the MS engineer and applied it to the box and rebooted. Checked the firewall services and they did not start.

The support engineer suggested doing a repair of ISA 2004 using the original installation media. Needless to say I wasn’t to keen on doing this given the awful track record of doing a repair on Windows installation from the installation media. But, I reluctantly went along for the ride. I downloaded the ISA installation files from one of our file shares to the local drive of the firewall. Then we proceeded to run the installer and selected the “Repair” option. After about 10 minutes of holding my breath, the repair process completed and we rebooted the ISA server.

To my total astonishment, the firewall services restarted as if nothing ever happened. Now I can breath again. I really wasn’t looking forward to a ISA server re-build. Needless to say, I will monitor the server for the next couple days for any issues.

ISA 2004 Lockdown Mode Issue

ryanwetmore — Sun, 21 Jun 2009 00:35:16 +0000

Came in to work Thursday morning to notice that one of our three ISA 2004 nodes in a NLB cluster had stopped passing traffic. The reason I noticed was because I couldn’t connect to the HP Management Log console to check the logs on that particular ISA server. Pretty weird considering I could RDP to the server to check the event logs. Here is what I found in the event logs:

Event Type: Information
Event Source: Microsoft ISA Server Control
Event Category: None
Event ID: 21263
Date:  6/18/2009
Time:  3:26:37 AM
User:  N/A
Computer: *********
Description:
ISA Server switched back to the primary Configuration Storage server *********** after using the alternate Configuration Storage server.

Event Type: Error
Event Source: Microsoft Firewall
Event Category: None
Event ID: 14019
Date:  6/18/2009
Time:  3:26:59 AM
User:  N/A
Computer: **********
Description:
ISA Server failed to load the firewall policy configuration. The failure occurred while loading the policy rule “Corporate FTP Server”.

Event Type: Error
Event Source: Microsoft ISA Server Web Proxy
Event Category: None
Event ID: 21177
Date:  6/18/2009
Time:  3:27:00 AM
User:  N/A
Computer: ***********
Description:
The Web filter [OWA Forms-Based Authentication Filter] failed to reload the configuration. If you recently applied changes to the configuration, verify that these changes are configured properly.

Event Type: Error
Event Source: Microsoft ISA Server Web Proxy
Event Category: None
Event ID: 21177
Date:  6/18/2009
Time:  3:27:00 AM
User:  N/A
Computer: ************
Description:
The Web filter [Link Translation Filter] failed to reload the configuration. If you recently applied changes to the configuration, verify that these changes are configured properly.

Event Type: Error
Event Source: Microsoft ISA Server Web Proxy
Event Category: None
Event ID: 21177
Date:  6/18/2009
Time:  3:27:00 AM
User:  N/A
Computer: ************
Description:
The Web filter [HTTP Filter] failed to reload the configuration. If you recently applied changes to the configuration, verify that these changes are configured properly.

Event Type: Error
Event Source: Microsoft ISA Server Control
Event Category: None
Event ID: 21209
Date:  6/18/2009
Time:  3:27:00 AM
User:  N/A
Computer:************
Description:
The ISA Server configuration agent was unable to upload the configuration to the ISA Server services. This could be due to a corrupt configuration. The ISA Server configuration agent is reverting the configuration back to the last known configuration. The service that failed to load the configuration is: fwsrv.

Event Type: Error
Event Source: Microsoft ISA Server Control
Event Category: None
Event ID: 21210
Date:  6/18/2009
Time:  3:27:06 AM
User:  N/A
Computer: ************

Description:
The new configuration cannot be set, and configuration settings cannot be reverted to last known good values. As a result ISA Server is now in lockdown mode. For more information, see the topic Lockdown Mode in ISA Server online help. The error description is: Some configuration changes were not applied. See the Windows event viewer for more details.

____________________________________________________________________

OH, CRAP!!!!! That was a more polite way of expressing my feelings at the time.

Needless to say my first stop for troubleshooting this issue was Google. Didn’t find a whole heck of a lot about this issue, but did find out that this “lockdown” mode was a way for ISA server to help protect the internal network if something went wrong with ISA server or if it was being DDOS’d, etc, etc.

Being that we host some critical applications for our customers, I decided that I should call Microsoft Support. After a callback, I was on the phone for a couple of hours w/ an MS ISA support engineer. Looking through the ISA console, we found that two firewall rules that used the same HTTP listener would throw an error when we right-clicked and selected “Properties”. We deleted and recreated the HTTP listener (after we backed them up ) for these rules and tried to re-start the firewall services. No go…

Then we decided to recreate the two firewall rules (after backing them up ). Tried to restart the firewall services again. Another no go…

This time the MS ISA engineer decided it was time to escalate this issue to his colleagues. He initiated an ISA log collection with something called the “ISA Data Packager”. See more about this tool from another blog here: http://tinyurl.com/cjd4uh .

Needless to say the issue has not been resolved yet. I’m beginning to think that the ISA configuration file (mostly XML) might be corrupted.

Stay Tuned……

Ryan Wetmore's Adventures in Life and IT

ISA 2004 Lockdown Mode Issue (Cont’d)

ISA 2004 Lockdown Mode Issue