Skip to main content

Posts

Showing posts from 2009

BAD_CERTIFICATE - A corrupt or unuseable certificate...

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
In wls_utc, when trying to test a webservice using SSL, the following error message is received: javax.net.ssl.SSLKeyException: FATAL Alert:BAD_CERTIFICATE - A corrupt or unuseable certificate was received.
If SSL debugging is enabled, the following error also appears in the logs: ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' <1254822672320>>
verification failed because RSA key public exponent [3] is too small
Cause
The certificate encryption is of a weaker strength than expected by newer versions of Java.
Solution
Add the flag "-Dweblogic.security.SSL.allowSmallRSAExponent=true" to the server startup parameters.
References


None.

WebLogic 10 Active Directory Authentication Provider Bug

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
With an Active Directory Authenticator configured, if a user logs in once with incorrect credentials further attempts to log in will fail, even with the correct username and password (until the server is restarted). If the user continues to login with correct credentials, WebLogic will eventually lockout the account.
Cause
This is a known bug for WebLogic 10 MP1. During authentication the AD provider binds twice using the same LDAP connection, once with the username password being authenticated, and once with the credentials supplied when you configure the LDAP provider. If authentication fails, the second binding doesn’t happen, and the unauthenticated LDAP connection is returned to the internal LDAP connection pool. This poses a problem when later trying to authenticate and the unauthenticated LDAP connection is retrieved from the pool... -Cobbie Behrend (Source: Basti…

WebLogic Server Connection Refused

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
The WebLogic admin or managed server is running, but telnetting to it fails, with an error like: Trying 10.123.123.123... telnet: connect to address 10.123.123.123: Connection refused Cause
You may be using a development/limited license. These restrict WebLogic to accept connections from up to 5 different IP addresses and then stop accepting connections from any other IP.
If you have multiple interfaces on the server, WebLogic may be listening on one of the others. If you do not specify a listen address for a admin or managed server, it will listen on all interfaces.
Solution
Replace the dev license with a purchased one, or restart the server to clear the 5 IPs that it accepts connection from.
Change the listen address, or telnet to the correct address.

Out of Memory: Killed process

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
On Red Hat Enterprise Linux 4 (and possibly other flavours and versions) all weblogic processes are disappearing and there are no errors in the server logs. Upon closer inspection the processes are being killed by the OS. The Linux "dmesg" command shows log messages like:
Out of Memory: Killed process 22043
Cause
The processes are using up too much low- (under 640k) or high-memory and the Linux OOM-Killer is killing the processes.
Solution
Either disable OOM-killer, or make it work more aggresively.
To disable, run this command:
echo "0" > /proc/sys/vm/oom-kill
To make more aggressive:
echo "250" > /proc/sys/vm/lower_zone_protection
References
http://en.wikipedia.org/wiki/Out_of_memory http://www.linux-archive.org/red-hat-linux/39907-out-memory-issue.html

Connection Server

The Connection Server is the name of this site. That's right, there is no technology officially called a 'connection server'. However it could describe any number of technologies that we use everyday in the corporate sys admin world. Any type of server must accept connections from clients in order to respond, so whether its a HTTP server, JEE server or sql/database server, its pretty much a connection server too. The articles at this site will touch on all these types of technologies.
In case you were wondering...
HTTP Servers, which handle HTTP connections, accept and process requestsand return HTML code which is processed by an internet browser (Eg. Internet Explorer, Firefox, Chrome, Safari and Opera). The internet browser renders the HTML code as formatted text and graphics. Common HTTP servers include Apache and IIS.
Application Servers, such as WebLogic, WebSphere, JBoss and Tomcat, usually include a basic HTTP server, but are also able to run JEE (Java Enterprise Editi…

WebLogic ProtocolException: HTTP tunneling...

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
Getting the following exception (regularly) in the Admin server logs.
HTTPClntLogin: Login rejected with code: 'Failed', reason: java.net.ProtocolException: HTTP tunneling is disabled
at weblogic.rjvm.http.HTTPServerJVMConnection.acceptJVMConnection(HTTPServerJVMConnection.java:88) at weblogic.rjvm.http.TunnelLoginServlet.service(TunnelLoginServlet.java:80) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.…

The Connection Server (About)

The Connection Server is an informal blog about a range of professional server-related topics, with a focus on the everyday tasks of a corporate IT infrastructure administrator. Articles range from hard-core technical tips to more business-oriented discussions.
What technologies do I cover? Potentially anything used in the corporate IT world, but to name a few big ones: Java (J2EE, J2SE and J2ME)Oracle WebLogic (previously BEA WebLogic)WebSphere and JBossOracle DBApache and IISWindows, AIX, Solaris and Red HatAlthough a few articles are aimed at helping beginners become more advanced administrators or developers, most articles assume at least a medium level of technical knowledge.

WebLogic 10.3 Licensing

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
There is no license.bea, UpdateLicense.cmd or UpdateLicense.sh files in the BEA_HOME directory of WebLogic 10.3.
Cause
When Oracle bought BEA they decided to do away with this approach to licensing.
Solution
When you download WebLogic 10.3, you are getting the full version. That is, you do not need to do anything to fully enable the product. Although technically you do not need to do anything to use this product, legally you will need to purchase a license from Oracle. Contact an Oracle sales person for this.
Previous versions of WebLogic, including 10.0, have not been changed, so you still require the license.bea file for these.

SocketException: Too many open files

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
On Unix-based systems, log message like:
Failed to listen on port 8081, failure count: 1, failing for 0 seconds, java.net.SocketException: Too many open files
Cause
The value of the "Maximum Open Socket" WebLogic setting is set higher than the the Unix limit (obtained by running the "ulimit -a" command).
Solution
Either decrease the value of "Maximum Open Socket" (located on the managed server's Tuning tab) to be less than the Unix limit, or increase the Unix limit. Investigate why there are so many requests being made.
References
Although this particular issue doesn't seem to be documented, you can see many other SocketException scenarios at http://www.insideexceptions.com/en/jdk-1-5-0/java-net-SocketException.html. It's a very helpful resource.

WebLogic Operator group has no effect

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
In WebLogic 9.2, adding an account to the Operator group has no effect if the account is also added to another group.
Cause
This is a known bug in WebLogic 9.2.
Solution
A patch is provided with the 9.2 installation. Ensure %BEA_HOME%\patch_weblogic920\patch_jars\CR285163_920GA.jar; is on the classpath.
References
None.

DeleteService failed

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
After removing a windows service, when trying to recreate it, the following error may be displayed:
DeleteService failed - The specified service has been marked for deletion. (0x430)
Cause
The service is still in-use (at least to some degree).
Solution
Close any Microsoft Management Console (MMC) sessions, kill any processes, etc. Note the service can be accessed remotely, so also consider that remote users may be viewing the service.
References
None.

NullPointerException at FileUtils.remove

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.
Symptom
java.lang.NullPointerException at weblogic.utils.FileUtils.remove(FileUtils.java:203) at weblogic.management.mbeans.custom.Component.removeTempModule(Component.java:353) at weblogic.management.mbeans.custom.Component.initializeTwoPhase(Component.java:435) at weblogic.management.mbeans.custom.Component.initialize(Component.java:282) at weblogic.management.mbeans.custom.EJBComponent.findOrCreateEJBDescriptor(EJBComponent.java:220) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at weblogic.management.internal.DynamicMBeanImpl.invokeLocally(DynamicMBeanImpl.java:755) at weblogic.management.internal.DynamicMBeanImpl.invoke(DynamicMBeanImpl.j…

User does not have permission on weblogic...

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

javax.naming.NoPermissionException: User does not have permission on weblogic.management.home to perform lookup operation. at weblogic.jndi.internal.ServerNamingNode.checkPermission(ServerNamingNode.java:365) at weblogic.jndi.internal.ServerNamingNode.checkLookup(ServerNamingNode.java:329) at weblogic.jndi.internal.ServerNamingNode.lookupHere(ServerNamingNode.java:153) at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:188) at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:196) at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:196) at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:258) at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:363) at javax.naming.InitialContext.lookup(InitialContext.java:361)
Cause
In releases prior to WebLogic 8.1 sp5, the default value of "…

WebLogic Books Reviewed

Looking at learning more about WebLogic? In this article I will give my opinions on the books I have encountered throughout my career.
If you're considering buying any of these books, then use the amazon links to support this site (and probably get the best available price).





BEA WebLogic Server 8.1: Unleashed

About four years ago I decided on a career change, and made the switch from Java developer to WebLogic Admin. On my first day on the job I was told that this book would be my bible, and to this day I still think it is the best WebLogic book available for learning WebLogic 8.1. In fact it is the only book I truly recommend, unless you wanting something updated for WebLogic 9 and later.









WebLogic: The Definitive Guide

I know this book gets reasonable reviews, but for a beginner or even every-day-admin I really wouldn't bother. There are definitely topics that are well covered and if its the topic you're after, you're in luck... The problem with this book is it doesn't…

Writing to WebLogic Server Logs

For J2EE developers who are not familiar with WebLogic, you can write debug and error messages to the server logs very easily.
When to Write to WebLogic Server Logs
Any debug, warning or error messages relating to the server or infrastructure. For example, information or errors with the following: Connecting to other components/systems.Database access, setup, etc.LDAPHTTP request/responseDeployment dependencies Do not write to the server logs if you are handling the following: Unexpected user inputApplication debug/info/warning messages. (Write these to your own application logs.)
How to do it
Writing to the WebLogic server logs is simple. Just use LoggingHelper to get a Logger object and print messages to your heart's content. Here is some example code:
Put this in the class declaration:
private static java.util.logging.Logger serverLogger = LoggingHelper.getServerLogger();
Use these throughout your code:
serverLogger.log(Level.INFO, "Hello World!"); serverLogger.warning("This…

Quotes for the Corporate Sys Admin

Every now and then my job gets so boring that I feel like a robot. Today was one of those days, so I decided I needed a laugh and went in search of funny quotes relevant to working as a corporate sys admin... Hopefully there's one or two new ones for readers.
The man who smiles when things go wrong has thought of someone to blame it on. -Robert Bloch UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity. -Dennis Ritchie Doing it right is no excuse for not meeting the schedule. -Plant Manager, Delco Corporation Some things Man was never meant to know. For everything else, there's Google. -Unknown To err is human... to really foul up requires the root password. -Unknown Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots... -Rich Cook I had a fortune cookie the other day and it said: 'Outlook not so good'. I said: …

Recovering WebLogic Passwords

In one of my previous articles (here) I explained that the SerializedSystemIni.dat file in WebLogic contains the key used to encrypt and decrypt passwords. If you're not currently keeping this file secure I suggest you do, as with it someone can (to name a few things): Decrypt the WebLogic admin username and password from boot.properties.Recover database passwords, if JDBC Connection pools are configured, from config.xml.Recover the keystore passwords from config.xml and obtain SSL certificates stored in the jks keystores. Essentially, they can do whatever they want, so if you don't know who can read your SerializedSystemIni.dat files, look... now.
In this article I will show how easy it is for this file to be used to recover lost passwords via a simple WLST script.

The Script
The script I use to decrypt passwords is incredibly short, and it works with WebLogic 8, 9 and 10 (probably for version 7 too). To use it, just create a new file called decryptpwd.py and paste the following c…

BEASVC.EXE - WebLogic as a Windows Service

I remember the first time I had to work out why WebLogic wouldn't run as a service. It was a frustrating experience. There were no error messages. No Windows error dialog. No console output.

How do you troubleshoot something like this??!!

This short article will show you. For simplicity I'll talk about the node manager, but the same principles apply for running an admin or managed server as a service.

First Steps

First, you still have your server logs. Sure the console output is better, but it's a starting point. Check this log for errors and especially take note of the start up variables such as PATH and CLASSPATH. If the server log isn't being created, that tells you WebLogic probably isn't even being started. (Check that you have a license file if you haven't already.)

Ok, so the logs were no help. The next step is to look at how the service is trying to start WebLogic.

When you install WebLogic as a service, you're really setting up beasvc.exe as the service.…

WebLogic's SerializedSystemIni.dat

In one of my earlier jobs, I once encountered a peculiar problem with WebLogic's SerializedSystemIni.dat file. Upon restarting a WebLogic admin or managed server, we would encounter an exception like:

<1/06/2009 02:23:35 PM EST> <Warning> <Security> <BEA-090066> <Problem handling boot identity. The following exception was generated: weblogic.security.internal.SerializedSystemIniException: [Security:090208]Corrupt SerializedSystemIni.dat>
<1/06/2009 02:23:35 PM EST> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:

We quickly noticed that the SerializedSystemIni.dat file was 0 bytes in size. As any good admin would do, we blamed the developers for corrupting this file. We then restored the file from backup, and everything went smoothly... for a while.

Unfortunately for us, the problem occurred again sometime later in a restricted test environment. This time we knew it was …

Directory Listing - A Simple WebLogic Application

In this article I will show how to create a simple WebLogic application that lets you navigate directories and files on a server. This can be one of the handiest utilities. I use it all the time to give developers access to logs in test and pre-production environments.


But first, a warning... Be careful what files you make available to others. Some files, especially within a WebLogic domain directory, contain information that MUST be kept secret if the domain is to remain secure. NEVER provide read-access to all files in a WebLogic domain directory.

Like any WebLogic application, you must create a WAR file with the following components inside:
WEB-INF (directory)
WEB-INF\web.xml
WEB-INF\weblogic.xml
The first step is to create a new directory called 'WEB-INF'.

The web.xml descriptor

Inside the WEB-INF directory, create a new file called web.xml, and paste the following XML code:


<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://…