Wednesday, October 7, 2009

BAD_CERTIFICATE - A corrupt or unuseable certificate...

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


In wls_utc, when trying to test a webservice using SSL, the following error message is received: FATAL Alert:BAD_CERTIFICATE - A corrupt or unuseable certificate was received.

If SSL debugging is enabled, the following error also appears in the logs:
ExecuteThread: '4' for queue: 'weblogic.kernel.Default
(self-tuning)' <1254822672320>>
verification failed because RSA key public exponent [3] is too small


The certificate encryption is of a weaker strength than expected by newer versions of Java.


Add the flag "" to the server startup parameters.



Wednesday, July 29, 2009

WebLogic 10 Active Directory Authentication Provider Bug

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


With an Active Directory Authenticator configured, if a user logs in once with incorrect credentials further attempts to log in will fail, even with the correct username and password (until the server is restarted). If the user continues to login with correct credentials, WebLogic will eventually lockout the account.


This is a known bug for WebLogic 10 MP1.
During authentication the AD provider binds twice using the same LDAP connection, once with the username password being authenticated, and once with the credentials supplied when you configure the LDAP provider. If authentication fails, the second binding doesn’t happen, and the unauthenticated LDAP connection is returned to the internal LDAP connection pool. This poses a problem when later trying to authenticate and the unauthenticated LDAP connection is retrieved from the pool...
-Cobbie Behrend (Source: Bastion)


Contact Oracle for a WebLogic patch, or upgrade to a later service pack.

Note to Vignette users: If you encounter this problem with VCM 7.6, Vignette will supply SP1 to fix the issue.


Tuesday, July 28, 2009

WebLogic Server Connection Refused

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


The WebLogic admin or managed server is running, but telnetting to it fails, with an error like:
telnet: connect to address Connection refused

You may be using a development/limited license. These restrict WebLogic to accept connections from up to 5 different IP addresses and then stop accepting connections from any other IP.

If you have multiple interfaces on the server, WebLogic may be listening on one of the others. If you do not specify a listen address for a admin or managed server, it will listen on all interfaces.


Replace the dev license with a purchased one, or restart the server to clear the 5 IPs that it accepts connection from.

Change the listen address, or telnet to the correct address.

Out of Memory: Killed process

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


On Red Hat Enterprise Linux 4 (and possibly other flavours and versions) all weblogic processes are disappearing and there are no errors in the server logs. Upon closer inspection the processes are being killed by the OS. The Linux "dmesg" command shows log messages like:

Out of Memory: Killed process 22043


The processes are using up too much low- (under 640k) or high-memory and the Linux OOM-Killer is killing the processes.


Either disable OOM-killer, or make it work more aggresively.

To disable, run this command:

echo "0" > /proc/sys/vm/oom-kill

To make more aggressive:

echo "250" > /proc/sys/vm/lower_zone_protection


Thursday, July 23, 2009

Connection Server

The Connection Server is the name of this site. That's right, there is no technology officially called a 'connection server'. However it could describe any number of technologies that we use everyday in the corporate sys admin world. Any type of server must accept connections from clients in order to respond, so whether its a HTTP server, JEE server or sql/database server, its pretty much a connection server too. The articles at this site will touch on all these types of technologies.

In case you were wondering...

HTTP Servers, which handle HTTP connections, accept and process requestsand return HTML code which is processed by an internet browser (Eg. Internet Explorer, Firefox, Chrome, Safari and Opera). The internet browser renders the HTML code as formatted text and graphics. Common HTTP servers include Apache and IIS.

Application Servers, such as WebLogic, WebSphere, JBoss and Tomcat, usually include a basic HTTP server, but are also able to run JEE (Java Enterprise Edition) applications which typically perform some business logic/rules and respond with HTML code to the internet browser.

Database Servers, accept SQL connections and process SQL queries to manipulate or return (to an application) data in database tables. Common database servers include Oracle, Microsoft SQL Server, MySQL, PostgreSQL and Pointbase.

Wednesday, July 22, 2009

WebLogic ProtocolException: HTTP tunneling...

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


Getting the following exception (regularly) in the Admin server logs.

HTTPClntLogin: Login rejected with code: 'Failed', reason: HTTP tunneling is disabled

at weblogic.rjvm.http.HTTPServerJVMConnection.acceptJVMConnection(
at weblogic.rjvm.http.TunnelLoginServlet.service(
at javax.servlet.http.HttpServlet.service(
at weblogic.servlet.internal.StubSecurityHelper$
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(
at weblogic.servlet.internal.ServletStubImpl.execute(
at weblogic.servlet.internal.ServletStubImpl.execute(
at weblogic.servlet.internal.WebAppServletContext$
at Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(
at weblogic.servlet.internal.WebAppServletContext.execute(


If you're accessing the WebLogic admin console via a SSH tunnel, or via a network interface that the admin server is not listening on, weblogic will throw this exception.


In the left-hand navigation, click on the domain then 'Servers'. In the list of servers that is displayed, click your admin server. Click on the Protocols tab, then HTTP sub-tab. Ensure 'Enable Tunneling' is checked. Activate your changes and restart the admin server.


More information about the ProtocalException can be found at You can see how the exception is implemented and everything.

The Connection Server (About)

The Connection Server is an informal blog about a range of professional server-related topics, with a focus on the everyday tasks of a corporate IT infrastructure administrator. Articles range from hard-core technical tips to more business-oriented discussions.
What technologies do I cover? Potentially anything used in the corporate IT world, but to name a few big ones:
  • Java (J2EE, J2SE and J2ME)
  • Oracle WebLogic (previously BEA WebLogic)
  • WebSphere and JBoss
  • Oracle DB
  • Apache and IIS
  • Windows, AIX, Solaris and Red Hat
Although a few articles are aimed at helping beginners become more advanced administrators
or developers, most articles assume at least a medium level of technical knowledge.

WebLogic 10.3 Licensing

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


There is no license.bea, UpdateLicense.cmd or files in the BEA_HOME directory of WebLogic 10.3.


When Oracle bought BEA they decided to do away with this approach to licensing.


When you download WebLogic 10.3, you are getting the full version. That is, you do not need to do anything to fully enable the product. Although technically you do not need to do anything to use this product, legally you will need to purchase a license from Oracle. Contact an Oracle sales person for this.

Previous versions of WebLogic, including 10.0, have not been changed, so you still require the license.bea file for these.

Tuesday, July 21, 2009

SocketException: Too many open files

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


On Unix-based systems, log message like:

Failed to listen on port 8081, failure count: 1, failing for 0 seconds, Too many open files


The value of the "Maximum Open Socket" WebLogic setting is set higher than the the Unix limit (obtained by running the "ulimit -a" command).


Either decrease the value of "Maximum Open Socket" (located on the managed server's Tuning tab) to be less than the Unix limit, or increase the Unix limit. Investigate why there are so many requests being made.


Although this particular issue doesn't seem to be documented, you can see many other SocketException scenarios at It's a very helpful resource.

WebLogic Operator group has no effect

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


In WebLogic 9.2, adding an account to the Operator group has no effect if the account is also added to another group.


This is a known bug in WebLogic 9.2.


A patch is provided with the 9.2 installation. Ensure %BEA_HOME%\patch_weblogic920\patch_jars\CR285163_920GA.jar; is on the classpath.



DeleteService failed

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


After removing a windows service, when trying to recreate it, the following error may be displayed:

DeleteService failed - The specified service has been marked for deletion. (0x430)


The service is still in-use (at least to some degree).


Close any Microsoft Management Console (MMC) sessions, kill any processes, etc. Note the service can be accessed remotely, so also consider that remote users may be viewing the service.



NullPointerException at FileUtils.remove

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


at weblogic.utils.FileUtils.remove(
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at javax.servlet.http.HttpServlet.service(
at javax.servlet.http.HttpServlet.service(
at weblogic.servlet.internal.ServletStubImpl$
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(
at weblogic.servlet.internal.WebAppServletContext$
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(
at weblogic.servlet.internal.ServletRequestImpl.execute(
at weblogic.kernel.ExecuteThread.execute(


Something seems to be preventing temporary data from being deleted (i.e. extracted application files, etc).


Delete managed server directories and restart. Applications may also need to be uninstalled and redeployed (not sure).


There are a number of other NullPointerException scenarios here:

User does not have permission on weblogic...

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.


javax.naming.NoPermissionException: User does not have permission on to perform lookup operation.
at weblogic.jndi.internal.ServerNamingNode.checkPermission(
at weblogic.jndi.internal.ServerNamingNode.checkLookup(
at weblogic.jndi.internal.ServerNamingNode.lookupHere(
at weblogic.jndi.internal.BasicNamingNode.lookup(
at weblogic.jndi.internal.BasicNamingNode.lookup(
at weblogic.jndi.internal.BasicNamingNode.lookup(
at weblogic.jndi.internal.WLEventContextImpl.lookup(
at weblogic.jndi.internal.WLContextImpl.lookup(
at javax.naming.InitialContext.lookup(


In releases prior to WebLogic 8.1 sp5, the default value of "AnonymousAdminLookupEnabled" attribute of the domain is "true". But from WebLogic 8.1 sp5, its default value is "false". Hence this exception occurs.


To change this,in the weblogic console click on the domain. Select "View Domain-wide Security Settings" and set the "Anonymous Admin Lookup Enabled" checkbox. This will solve your problem.



WebLogic Books Reviewed

Looking at learning more about WebLogic? In this article I will give my opinions on the books I have encountered throughout my career.

If you're considering buying any of these books, then use the amazon links to support this site (and probably get the best available price).

BEA WebLogic Server 8.1: Unleashed

About four years ago I decided on a career change, and made the switch from Java developer to WebLogic Admin. On my first day on the job I was told that this book would be my bible, and to this day I still think it is the best WebLogic book available for learning WebLogic 8.1. In fact it is the only book I truly recommend, unless you wanting something updated for WebLogic 9 and later.

WebLogic: The Definitive Guide

I know this book gets reasonable reviews, but for a beginner or even every-day-admin I really wouldn't bother. There are definitely topics that are well covered and if its the topic you're after, you're in luck... The problem with this book is it doesn't cover the right topics for someone who has to look after WebLogic environments on a daily basis.

If you're looking at developing for WebLogic or becoming a hardcore WebLogic admin then you should consider this book. Otherwise spend your money on the BEA WebLogic 8.1 Unleashed.

Mastering BEA WebLogic Server

Mastering BEA WebLogic Server is aimed mostly at developers, but does contain some useful information for administrators. If you're interested in getting inside WebLogic then this is a good book. It teaches you to develop good WebLogic applications and then to build good WebLogic environments. The clustering and load-balancing topics would be informative to those new to an WebLogic admin role.

Unfortunately only the last 100 pages or so deal with admin topics, so you'd need to get it fairly cheap if you're not interested in the developer topics. If you pick up a cheap second-hand copy from Amazon, those last 100 pages are worth reading.

Professional Oracle WebLogic Server
Its been quite a while since a book on WebLogic has been published, so this soon to be released title is one I'm looking forward to getting my hands on. According to Amazon, this book will cover WebLogic 11g (I.e. WebLogic Server 10.3).

I'll update the article as soon as I get my hands on it.

Friday, July 17, 2009

Writing to WebLogic Server Logs

For J2EE developers who are not familiar with WebLogic, you can write debug and error messages to the server logs very easily.

When to Write to WebLogic Server Logs

Any debug, warning or error messages relating to the server or infrastructure. For example, information or errors with the following:
  • Connecting to other components/systems.
  • Database access, setup, etc.
  • LDAP
  • HTTP request/response
  • Deployment dependencies
Do not write to the server logs if you are handling the following:
  • Unexpected user input
  • Application debug/info/warning messages. (Write these to your own application logs.)

How to do it

Writing to the WebLogic server logs is simple. Just use LoggingHelper to get a Logger object and print messages to your heart's content. Here is some example code:

Put this in the class declaration:

private static java.util.logging.Logger serverLogger = LoggingHelper.getServerLogger();

Use these throughout your code:

serverLogger.log(Level.INFO, "Hello World!");
serverLogger.warning("This is a warning.");
serverLogger.severe("Something bad has happened!");

If you need more information about these classes, try these links:

BEA provides quite a lot of information about logging here:

Thursday, July 16, 2009

Quotes for the Corporate Sys Admin

Every now and then my job gets so boring that I feel like a robot. Today was one of those days, so I decided I needed a laugh and went in search of funny quotes relevant to working as a corporate sys admin... Hopefully there's one or two new ones for readers.

The man who smiles when things go wrong has thought of someone to blame it on.
-Robert Bloch
UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity.
-Dennis Ritchie
Doing it right is no excuse for not meeting the schedule.
-Plant Manager, Delco Corporation
Some things Man was never meant to know. For everything else, there's Google.
To err is human... to really foul up requires the root password.
Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots...
-Rich Cook
I had a fortune cookie the other day and it said: 'Outlook not so good'. I said: 'Sure, but Microsoft ships it anyway'.
Failure is not an option - it comes bundled with Windows.
Who the hell is General Failure? And why is he trying to read my hard disk?
This week I mapped and gapped the requirements to consolidate everything into a program of work... to maximize synergy, capture and optimize our resource utilization. If any of that sounded like work, I'll do some more of it next week.
-Wally, From Dilbert
Have a great day :)

Wednesday, June 17, 2009

Recovering WebLogic Passwords

In one of my previous articles (here) I explained that the SerializedSystemIni.dat file in WebLogic contains the key used to encrypt and decrypt passwords. If you're not currently keeping this file secure I suggest you do, as with it someone can (to name a few things):
  • Decrypt the WebLogic admin username and password from
  • Recover database passwords, if JDBC Connection pools are configured, from config.xml.
  • Recover the keystore passwords from config.xml and obtain SSL certificates stored in the jks keystores.
Essentially, they can do whatever they want, so if you don't know who can read your SerializedSystemIni.dat files, look... now.

In this article I will show how easy it is for this file to be used to recover lost passwords via a simple WLST script.

The Script

The script I use to decrypt passwords is incredibly short, and it works with WebLogic 8, 9 and 10 (probably for version 7 too). To use it, just create a new file called and paste the following code into it:

from import *
from import *

# Remind user about how to use
raw_input("Please ensure SerializedSystemIni.dat is in the current directory now, and press ENTER to continue.")

# Get encryption service
encryptionService = SerializedSystemIni.getEncryptionService(".")
clearOrEncryptService = ClearOrEncryptedService(encryptionService)

# Get user to enter password
pwd = raw_input("Enter encrypted password (Eg. {3DES}Y1fA34S...): ")

# Remove unnecessary escape characters
preppwd = pwd.replace("\\", "")

# Decrypt the password
print "Recovered password is: " + clearOrEncryptService.decrypt(preppwd)

You can download the script ( here.

Running the Script

To run this script, execute the WLST environment and provide it with the script to run. If you're using a version of WebLogic older than 8.1.6 you'll need to install a newer version which has WLST.

Use the following command (replace %WL_HOME% with the correct directory) to run the script.


Once the WLST environment has started up, the script will remind you to copy the SerializedSystemIni.dat file into the current directory. Press enter and paste your encrypted password string. That's it. It will spit out the password in plain-text.

Understanding the Script

The script is so short and simple it is barely worth explaining, but here are a few things worth mentioning.

You'll notice that the first two lines import packages. These are largely undocumented, so don't expect to find much information on them.

The next significant piece of code instantiates a ClearOrEncryptedService object which will use the SerializedSystemIni.dat file located in the current directory (".").

The final line of code decrypts the password and prints it in plain text.

Monday, June 1, 2009

BEASVC.EXE - WebLogic as a Windows Service

I remember the first time I had to work out why WebLogic wouldn't run as a service. It was a frustrating experience. There were no error messages. No Windows error dialog. No console output.

How do you troubleshoot something like this??!!

This short article will show you. For simplicity I'll talk about the node manager, but the same principles apply for running an admin or managed server as a service.

First Steps

First, you still have your server logs. Sure the console output is better, but it's a starting point. Check this log for errors and especially take note of the start up variables such as PATH and CLASSPATH. If the server log isn't being created, that tells you WebLogic probably isn't even being started. (Check that you have a license file if you haven't already.)

Ok, so the logs were no help. The next step is to look at how the service is trying to start WebLogic.

When you install WebLogic as a service, you're really setting up beasvc.exe as the service. This is a tool that BEA/Oracle supplies with your WebLogic installation, and all it does is runs WebLogic using parameters stored in the Windows Registry.

To find out what beasvc is doing, you can either:
  • Use the beasvc '-debug' mode; or
  • Simulate the way beasvc starts WebLogic.
BEASVC Debug Mode

beasvc.exe provides an easy debug mode. To use it, open a command-line console (cmd.exe) and change to the directory where all your domains are located (eg c:\bea\user_projects\). Then run the following command:

WL_HOME\server\bin\beasvc.exe -debug "SVC_NAME"

Where WL_HOME is where weblogic server is installed to (Eg.
c:\bea\weblogic92) and SVC_NAME is the name of the service.

Running beasvc with this option will give you a lot of useful information. You can find out more about the '-debug' option here.

Simulating BEASVC.EXE

If the '-debug' option isn't helping, or you need to test something specific, you can always simulate how beasvc starts WebLogic.
Normally to start WebLogic, you'd go to the startWebLogic.cmd (or other) script, but since you're starting WebLogic as a service this script is useless.

What we need to do is get all the environment variables and start-up parameters and then run WebLogic at the command-line.

To find the start-up parameters in the Windows Registry, first obtain the full service name from the Windows Services management console. To do this, right-click on the service you are troubleshooting, and click Properties. Copy the 'Display Name' field.

In regedit, search for the service using this display name. (On the Find dialog, select 'Look at: Keys' only.)

When you've found the node, expand it and select 'Parameters'. This contains all the important start-up parameters and environment variables.

Now that we have the start-up parameters used by beasvc, we can easily simulate it.

First, open a command console (cmd.exe) and change to the directory specified by the ExecDir registry entry.

Set the PATH variable to the value of the Path registry entry. Now run java.exe with no options. If it does not execute, you know that the registry's Path entry is incorrect. You can either edit it with regedit, or fix your installsvc.cmd script and run it again.

Finally, run java.exe again, this time followed by the CmdLine registry entry. This should run WebLogic as beasvc.exe would, except any errors will be printed to the console window.

Common Problems

With services, I frequently encounter various problems caused by incorrect CLASSPATH or PATH values - usually directory names with spaces that aren't surrounded by quotes. Java can interpret these as separate command line parameters, rather than a single file path.

As a general rule, avoid installing WebLogic to a directory that has a space in the name. If you do, use the old 8dot3 path name (use the DIR /X command to obtain these).


If you're having trouble working out why an exception is being thrown, you can check if this site covers your scenario: It's a good resource for administrators who understand java.

Friday, May 29, 2009

WebLogic's SerializedSystemIni.dat

In one of my earlier jobs, I once encountered a peculiar problem with WebLogic's SerializedSystemIni.dat file. Upon restarting a WebLogic admin or managed server, we would encounter an exception like:

<1/06/2009 02:23:35 PM EST> <Warning> <Security> <BEA-090066> <Problem handling boot identity. The following exception was generated: [Security:090208]Corrupt SerializedSystemIni.dat>
<1/06/2009 02:23:35 PM EST> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:

We quickly noticed that the SerializedSystemIni.dat file was 0 bytes in size. As any good admin would do, we blamed the developers for corrupting this file. We then restored the file from backup, and everything went smoothly... for a while.

Unfortunately for us, the problem occurred again sometime later in a restricted test environment. This time we knew it was a WebLogic quirk of some sort.

SerializedSystemIni.dat Explained

If you've ever looked through config.xml, you may have noticed password values that look like "{3DES}bOH6MEd8S82sxg2Nk=". This is how WebLogic stores passwords encrypted with the "Triple DES" algorithm (see here for a description of 3DES).

When WebLogic stores or retrieves one of these passwords, it uses a hash key which is stored in the SerializedSystemIni.dat file.

That's right, the key to all your encrypted admin and DB passwords is right there on your file system. Read access is all that is needed to compromise a WebLogic domain. Perhaps worse, you could be making DB passwords available.

Take a look at your WebLogic servers... who has read access to your WebLogic files?

If you're wondering where this file is located, its in the domain directory for WebLogic 8 (or earlier) and in the 'security' directory for WebLogic 9 or later.

Solving the 0-Byte SerializedSystemIni.dat Problem

It wasn't until I understood the purpose of this file that I was able to solve the 0-byte problem.

Some time after we noticed SerializedSystemIni.dat was being corrupted, we realized the problem was occurring after the server had run out of disk space (sometimes several weeks later and the disk was no longer full).

After a lot of googling and well-intended but limited help from BEA, we discovered that SerializedSystemIni.dat is periodically re-written by the WebLogic Admin server. If the server is unable to write the contents of the file (disk is full, etc), WebLogic will lose the hash key and will no longer be able to decrypt any passwords. I.e. if you're the admin, you'd better have backups or personal indemnity insurance.

So if you are wondering why your WebLogic servers won't start, and SerializedSystemIni.dat is 0 bytes in size, you now know why.

The solution? Don't let your server's disk fill up.

Directory Listing - A Simple WebLogic Application

In this article I will show how to create a simple WebLogic application that lets you navigate directories and files on a server. This can be one of the handiest utilities. I use it all the time to give developers access to logs in test and pre-production environments.

But first, a warning... Be careful what files you make available to others. Some files, especially within a WebLogic domain directory, contain information that MUST be kept secret if the domain is to remain secure. NEVER provide read-access to all files in a WebLogic domain directory.

Like any WebLogic application, you must create a WAR file with the following components inside:
  • WEB-INF (directory)
  • WEB-INF\web.xml
  • WEB-INF\weblogic.xml
The first step is to create a new directory called 'WEB-INF'.

The web.xml descriptor

Inside the WEB-INF directory, create a new file called web.xml, and paste the following XML code:

<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "">

I won't explain this XML code, but it is essentially an empty deployment descriptor.

The weblogic.xml descriptor

Create another file in the WEB-INF directory called weblogic.xml, and paste the following into it:

<!DOCTYPE weblogic-web-app PUBLIC "-//BEA Systems, Inc.//DTD Web Application 8.1//EN" "






XML Elements Explained

<index-directory-enabled> If true, WebLogic will display a list of directories and files. Unfortunately, you cannot filter this list. WebLogic does not provide much in the way of directory/file listing functionality - either you list them or don't.

<local-path> The path component of a URL where you will find the application. For example, the XML code above will make the application accessible at http://localhost:7001/getlogs/

<url-pattern> Specifies the local directory that will be displayed when a user navigates to the context-root.

<context-root> A pattern that URLs must match in order for them to be mapped to the local directory.

Sample Applications (update)

The sample apps from this article can be downloaded here. These sample applications are NOT SAFE for a production environment as they could potentially expose sensitive information.

Deployment Notes:
  • For WebLogic 8.1 or older, when deploying you must select "Copy this Web Application module onto every target for me".
  • For WebLogic 9 or newer, deploy with default options.