ISA 2004 Lockdown Mode Issue

Came in to work Thursday morning to notice that one of our three ISA 2004 nodes in a NLB cluster had stopped passing traffic.  The reason I noticed was because I couldn’t connect to the HP Management Log console to check the logs on that particular ISA server. Pretty weird considering I could RDP to the server to check the event logs.    Here is what I found in the event logs:

Event Type: Information
Event Source: Microsoft ISA Server Control
Event Category: None
Event ID: 21263
Date:  6/18/2009
Time:  3:26:37 AM
User:  N/A
Computer: *********
Description:
ISA Server switched back to the primary Configuration Storage server *********** after using the alternate Configuration Storage server.

Event Type: Error
Event Source: Microsoft Firewall
Event Category: None
Event ID: 14019
Date:  6/18/2009
Time:  3:26:59 AM
User:  N/A
Computer: **********
Description:
ISA Server failed to load the firewall policy configuration. The failure occurred while loading the policy rule “Corporate FTP Server”.

Event Type: Error
Event Source: Microsoft ISA Server Web Proxy
Event Category: None
Event ID: 21177
Date:  6/18/2009
Time:  3:27:00 AM
User:  N/A
Computer: ***********
Description:
The Web filter [OWA Forms-Based Authentication Filter] failed to reload the configuration. If you recently applied changes to the configuration, verify that these changes are configured properly.

Event Type: Error
Event Source: Microsoft ISA Server Web Proxy
Event Category: None
Event ID: 21177
Date:  6/18/2009
Time:  3:27:00 AM
User:  N/A
Computer: ************
Description:
The Web filter [Link Translation Filter] failed to reload the configuration. If you recently applied changes to the configuration, verify that these changes are configured properly.

Event Type: Error
Event Source: Microsoft ISA Server Web Proxy
Event Category: None
Event ID: 21177
Date:  6/18/2009
Time:  3:27:00 AM
User:  N/A
Computer: ************
Description:
The Web filter [HTTP Filter] failed to reload the configuration. If you recently applied changes to the configuration, verify that these changes are configured properly.

Event Type: Error
Event Source: Microsoft ISA Server Control
Event Category: None
Event ID: 21209
Date:  6/18/2009
Time:  3:27:00 AM
User:  N/A
Computer:************
Description:
The ISA Server configuration agent was unable to upload the configuration to the ISA Server services. This could be due to a corrupt configuration. The ISA Server configuration agent is reverting the configuration back to the last known configuration. The service that failed to load the configuration is: fwsrv.

Event Type: Error
Event Source: Microsoft ISA Server Control
Event Category: None
Event ID: 21210
Date:  6/18/2009
Time:  3:27:06 AM
User:  N/A
Computer: ************

Description:
The new configuration cannot be set, and configuration settings cannot be reverted to last known good values. As a result ISA Server is now in lockdown mode. For more information, see the topic Lockdown Mode in ISA Server online help. The error description is: Some configuration changes were not applied. See the Windows event viewer for more details.

____________________________________________________________________

OH, CRAP!!!!!  That was a more polite way of expressing my feelings at the time.

Needless to say my first stop for troubleshooting this issue was Google.  Didn’t find a whole heck of a lot about this issue, but did find out that this “lockdown” mode was a way for ISA server to help protect the internal network if something went wrong with ISA server or if it was being DDOS’d, etc, etc.

Being that we host some critical applications for our customers, I decided that I should call Microsoft Support.  After a callback, I was on the phone for a couple of hours w/ an MS ISA support engineer. Looking through the ISA console, we found that two firewall rules that used the same HTTP listener would throw an error when we right-clicked and selected “Properties”.  We deleted and recreated the HTTP listener (after we backed them up ) for these rules and tried to re-start the firewall services.  No go…

Then we decided to recreate the two firewall rules (after backing them up ).  Tried to restart the firewall services again.  Another no go…

This time the MS ISA engineer decided it was time to escalate this issue to his colleagues.  He initiated an ISA log collection with something called the “ISA Data Packager”.  See more about this tool from another blog here:  http://tinyurl.com/cjd4uh . 

Needless to say the issue has not been resolved yet.  I’m beginning to think that the ISA configuration file (mostly XML) might be corrupted.

Stay Tuned……