Friday, May 29, 2009

WebLogic's SerializedSystemIni.dat

In one of my earlier jobs, I once encountered a peculiar problem with WebLogic's SerializedSystemIni.dat file. Upon restarting a WebLogic admin or managed server, we would encounter an exception like:

<1/06/2009 02:23:35 PM EST> <Warning> <Security> <BEA-090066> <Problem handling boot identity. The following exception was generated: weblogic.security.internal.SerializedSystemIniException: [Security:090208]Corrupt SerializedSystemIni.dat>
<1/06/2009 02:23:35 PM EST> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:


We quickly noticed that the SerializedSystemIni.dat file was 0 bytes in size. As any good admin would do, we blamed the developers for corrupting this file. We then restored the file from backup, and everything went smoothly... for a while.

Unfortunately for us, the problem occurred again sometime later in a restricted test environment. This time we knew it was a WebLogic quirk of some sort.

SerializedSystemIni.dat Explained

If you've ever looked through config.xml, you may have noticed password values that look like "{3DES}bOH6MEd8S82sxg2Nk=". This is how WebLogic stores passwords encrypted with the "Triple DES" algorithm (see here for a description of 3DES).

When WebLogic stores or retrieves one of these passwords, it uses a hash key which is stored in the SerializedSystemIni.dat file.

That's right, the key to all your encrypted admin and DB passwords is right there on your file system. Read access is all that is needed to compromise a WebLogic domain. Perhaps worse, you could be making DB passwords available.

Take a look at your WebLogic servers... who has read access to your WebLogic files?

If you're wondering where this file is located, its in the domain directory for WebLogic 8 (or earlier) and in the 'security' directory for WebLogic 9 or later.

Solving the 0-Byte SerializedSystemIni.dat Problem

It wasn't until I understood the purpose of this file that I was able to solve the 0-byte problem.

Some time after we noticed SerializedSystemIni.dat was being corrupted, we realized the problem was occurring after the server had run out of disk space (sometimes several weeks later and the disk was no longer full).

After a lot of googling and well-intended but limited help from BEA, we discovered that SerializedSystemIni.dat is periodically re-written by the WebLogic Admin server. If the server is unable to write the contents of the file (disk is full, etc), WebLogic will lose the hash key and will no longer be able to decrypt any passwords. I.e. if you're the admin, you'd better have backups or personal indemnity insurance.

So if you are wondering why your WebLogic servers won't start, and SerializedSystemIni.dat is 0 bytes in size, you now know why.

The solution? Don't let your server's disk fill up.

Directory Listing - A Simple WebLogic Application

In this article I will show how to create a simple WebLogic application that lets you navigate directories and files on a server. This can be one of the handiest utilities. I use it all the time to give developers access to logs in test and pre-production environments.


But first, a warning... Be careful what files you make available to others. Some files, especially within a WebLogic domain directory, contain information that MUST be kept secret if the domain is to remain secure. NEVER provide read-access to all files in a WebLogic domain directory.

Like any WebLogic application, you must create a WAR file with the following components inside:
  • WEB-INF (directory)
  • WEB-INF\web.xml
  • WEB-INF\weblogic.xml
The first step is to create a new directory called 'WEB-INF'.

The web.xml descriptor

Inside the WEB-INF directory, create a new file called web.xml, and paste the following XML code:


<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
</web-app>


I won't explain this XML code, but it is essentially an empty deployment descriptor.

The weblogic.xml descriptor

Create another file in the WEB-INF directory called weblogic.xml, and paste the following into it:


<!DOCTYPE weblogic-web-app PUBLIC "-//BEA Systems, Inc.//DTD Web Application 8.1//EN" "http://www.bea.com/servers/wls810/dtd/we
blogic810-web-jar.dtd">

<weblogic-web-app>

<container-descriptor>
<index-directory-enabled>true</index-directory-enabled>
</container-descriptor>

<virtual-directory-mapping>
<local-path>../../../logs/</local-path>
<url-pattern>*</url-pattern>
</virtual-directory-mapping>

<context-root>/getlogs</context-root>

</weblogic-web-app>


XML Elements Explained

<index-directory-enabled> If true, WebLogic will display a list of directories and files. Unfortunately, you cannot filter this list. WebLogic does not provide much in the way of directory/file listing functionality - either you list them or don't.

<local-path> The path component of a URL where you will find the application. For example, the XML code above will make the application accessible at http://localhost:7001/getlogs/

<url-pattern> Specifies the local directory that will be displayed when a user navigates to the context-root.

<context-root> A pattern that URLs must match in order for them to be mapped to the local directory.

Sample Applications (update)

The sample apps from this article can be downloaded here. These sample applications are NOT SAFE for a production environment as they could potentially expose sensitive information.

Deployment Notes:
  • For WebLogic 8.1 or older, when deploying you must select "Copy this Web Application module onto every target for me".
  • For WebLogic 9 or newer, deploy with default options.