In one of my earlier jobs, I once encountered a peculiar problem with WebLogic's SerializedSystemIni.dat file. Upon restarting a WebLogic admin or managed server, we would encounter an exception like:
<1/06/2009 02:23:35 PM EST> <Warning> <Security> <BEA-090066> <Problem handling boot identity. The following exception was generated: weblogic.security.internal.SerializedSystemIniException: [Security:090208]Corrupt SerializedSystemIni.dat>
<1/06/2009 02:23:35 PM EST> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:
We quickly noticed that the SerializedSystemIni.dat file was 0 bytes in size. As any good admin would do, we blamed the developers for corrupting this file. We then restored the file from backup, and everything went smoothly... for a while.
Unfortunately for us, the problem occurred again sometime later in a restricted test environment. This time we knew it was a WebLogic quirk of some sort.
SerializedSystemIni.dat Explained
If you've ever looked through config.xml, you may have noticed password values that look like "{3DES}bOH6MEd8S82sxg2Nk=". This is how WebLogic stores passwords encrypted with the "Triple DES" algorithm (see here for a description of 3DES).
When WebLogic stores or retrieves one of these passwords, it uses a hash key which is stored in the SerializedSystemIni.dat file.
That's right, the key to all your encrypted admin and DB passwords is right there on your file system. Read access is all that is needed to compromise a WebLogic domain. Perhaps worse, you could be making DB passwords available.
Take a look at your WebLogic servers... who has read access to your WebLogic files?
If you're wondering where this file is located, its in the domain directory for WebLogic 8 (or earlier) and in the 'security' directory for WebLogic 9 or later.
Solving the 0-Byte SerializedSystemIni.dat Problem
It wasn't until I understood the purpose of this file that I was able to solve the 0-byte problem.
Some time after we noticed SerializedSystemIni.dat was being corrupted, we realized the problem was occurring after the server had run out of disk space (sometimes several weeks later and the disk was no longer full).
After a lot of googling and well-intended but limited help from BEA, we discovered that SerializedSystemIni.dat is periodically re-written by the WebLogic Admin server. If the server is unable to write the contents of the file (disk is full, etc), WebLogic will lose the hash key and will no longer be able to decrypt any passwords. I.e. if you're the admin, you'd better have backups or personal indemnity insurance.
So if you are wondering why your WebLogic servers won't start, and SerializedSystemIni.dat is 0 bytes in size, you now know why.
The solution? Don't let your server's disk fill up.
<1/06/2009 02:23:35 PM EST> <Warning> <Security> <BEA-090066> <Problem handling boot identity. The following exception was generated: weblogic.security.internal.SerializedSystemIniException: [Security:090208]Corrupt SerializedSystemIni.dat>
<1/06/2009 02:23:35 PM EST> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:
We quickly noticed that the SerializedSystemIni.dat file was 0 bytes in size. As any good admin would do, we blamed the developers for corrupting this file. We then restored the file from backup, and everything went smoothly... for a while.
Unfortunately for us, the problem occurred again sometime later in a restricted test environment. This time we knew it was a WebLogic quirk of some sort.
SerializedSystemIni.dat Explained
If you've ever looked through config.xml, you may have noticed password values that look like "{3DES}bOH6MEd8S82sxg2Nk=". This is how WebLogic stores passwords encrypted with the "Triple DES" algorithm (see here for a description of 3DES).
When WebLogic stores or retrieves one of these passwords, it uses a hash key which is stored in the SerializedSystemIni.dat file.
That's right, the key to all your encrypted admin and DB passwords is right there on your file system. Read access is all that is needed to compromise a WebLogic domain. Perhaps worse, you could be making DB passwords available.
Take a look at your WebLogic servers... who has read access to your WebLogic files?
If you're wondering where this file is located, its in the domain directory for WebLogic 8 (or earlier) and in the 'security' directory for WebLogic 9 or later.
Solving the 0-Byte SerializedSystemIni.dat Problem
It wasn't until I understood the purpose of this file that I was able to solve the 0-byte problem.
Some time after we noticed SerializedSystemIni.dat was being corrupted, we realized the problem was occurring after the server had run out of disk space (sometimes several weeks later and the disk was no longer full).
After a lot of googling and well-intended but limited help from BEA, we discovered that SerializedSystemIni.dat is periodically re-written by the WebLogic Admin server. If the server is unable to write the contents of the file (disk is full, etc), WebLogic will lose the hash key and will no longer be able to decrypt any passwords. I.e. if you're the admin, you'd better have backups or personal indemnity insurance.
So if you are wondering why your WebLogic servers won't start, and SerializedSystemIni.dat is 0 bytes in size, you now know why.
The solution? Don't let your server's disk fill up.
Hi Casey
ReplyDeleteI experienced a similar problem in my WebLogic domain and came across your tips in this post. However, my SerializedSystemIni.dat was not zero-byte, but it was corrupted. So, I investigated further, raised an SR with Oracle, obtained a patch and documented my experience with this issue in my blog (http://itsupport.mrkips.com/2009/06/24/weblogic-serializedsystemini-dat), which I launched only a couple of days ago. Thank you for your tips in this post.