Wednesday, July 29, 2009

WebLogic 10 Active Directory Authentication Provider Bug


This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

With an Active Directory Authenticator configured, if a user logs in once with incorrect credentials further attempts to log in will fail, even with the correct username and password (until the server is restarted). If the user continues to login with correct credentials, WebLogic will eventually lockout the account.

Cause

This is a known bug for WebLogic 10 MP1.
During authentication the AD provider binds twice using the same LDAP connection, once with the username password being authenticated, and once with the credentials supplied when you configure the LDAP provider. If authentication fails, the second binding doesn’t happen, and the unauthenticated LDAP connection is returned to the internal LDAP connection pool. This poses a problem when later trying to authenticate and the unauthenticated LDAP connection is retrieved from the pool...
-Cobbie Behrend (Source: Bastion)

Solution

Contact Oracle for a WebLogic patch, or upgrade to a later service pack.

Note to Vignette users: If you encounter this problem with VCM 7.6, Vignette will supply SP1 to fix the issue.

References

Tuesday, July 28, 2009

WebLogic Server Connection Refused


This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

The WebLogic admin or managed server is running, but telnetting to it fails, with an error like:
Trying 10.123.123.123...
telnet: connect to address 10.123.123.123: Connection refused
Cause

You may be using a development/limited license. These restrict WebLogic to accept connections from up to 5 different IP addresses and then stop accepting connections from any other IP.

If you have multiple interfaces on the server, WebLogic may be listening on one of the others. If you do not specify a listen address for a admin or managed server, it will listen on all interfaces.

Solution

Replace the dev license with a purchased one, or restart the server to clear the 5 IPs that it accepts connection from.

Change the listen address, or telnet to the correct address.

Out of Memory: Killed process


This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

On Red Hat Enterprise Linux 4 (and possibly other flavours and versions) all weblogic processes are disappearing and there are no errors in the server logs. Upon closer inspection the processes are being killed by the OS. The Linux "dmesg" command shows log messages like:

Out of Memory: Killed process 22043

Cause

The processes are using up too much low- (under 640k) or high-memory and the Linux OOM-Killer is killing the processes.

Solution

Either disable OOM-killer, or make it work more aggresively.

To disable, run this command:

echo "0" > /proc/sys/vm/oom-kill

To make more aggressive:

echo "250" > /proc/sys/vm/lower_zone_protection

References

http://en.wikipedia.org/wiki/Out_of_memory
http://www.linux-archive.org/red-hat-linux/39907-out-memory-issue.html

Thursday, July 23, 2009

Connection Server

The Connection Server is the name of this site. That's right, there is no technology officially called a 'connection server'. However it could describe any number of technologies that we use everyday in the corporate sys admin world. Any type of server must accept connections from clients in order to respond, so whether its a HTTP server, JEE server or sql/database server, its pretty much a connection server too. The articles at this site will touch on all these types of technologies.

In case you were wondering...

HTTP Servers, which handle HTTP connections, accept and process requestsand return HTML code which is processed by an internet browser (Eg. Internet Explorer, Firefox, Chrome, Safari and Opera). The internet browser renders the HTML code as formatted text and graphics. Common HTTP servers include Apache and IIS.

Application Servers, such as WebLogic, WebSphere, JBoss and Tomcat, usually include a basic HTTP server, but are also able to run JEE (Java Enterprise Edition) applications which typically perform some business logic/rules and respond with HTML code to the internet browser.

Database Servers, accept SQL connections and process SQL queries to manipulate or return (to an application) data in database tables. Common database servers include Oracle, Microsoft SQL Server, MySQL, PostgreSQL and Pointbase.

Wednesday, July 22, 2009

WebLogic ProtocolException: HTTP tunneling...


This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

Getting the following exception (regularly) in the Admin server logs.

HTTPClntLogin: Login rejected with code: 'Failed', reason: java.net.ProtocolException: HTTP tunneling is disabled

at weblogic.rjvm.http.HTTPServerJVMConnection.acceptJVMConnection(HTTPServerJVMConnection.java:88)
at weblogic.rjvm.http.TunnelLoginServlet.service(TunnelLoginServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3395)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)

Cause

If you're accessing the WebLogic admin console via a SSH tunnel, or via a network interface that the admin server is not listening on, weblogic will throw this exception.

Solution

In the left-hand navigation, click on the domain then 'Servers'. In the list of servers that is displayed, click your admin server. Click on the Protocols tab, then HTTP sub-tab. Ensure 'Enable Tunneling' is checked. Activate your changes and restart the admin server.

References

More information about the ProtocalException can be found at http://www.insideexceptions.com/en/jdk-1-5-0/java-net-ProtocolException.html. You can see how the exception is implemented and everything.

The Connection Server (About)


The Connection Server is an informal blog about a range of professional server-related topics, with a focus on the everyday tasks of a corporate IT infrastructure administrator. Articles range from hard-core technical tips to more business-oriented discussions.
What technologies do I cover? Potentially anything used in the corporate IT world, but to name a few big ones:
  • Java (J2EE, J2SE and J2ME)
  • Oracle WebLogic (previously BEA WebLogic)
  • WebSphere and JBoss
  • Oracle DB
  • Apache and IIS
  • Windows, AIX, Solaris and Red Hat
Although a few articles are aimed at helping beginners become more advanced administrators
or developers, most articles assume at least a medium level of technical knowledge.

WebLogic 10.3 Licensing



This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

There is no license.bea, UpdateLicense.cmd or UpdateLicense.sh files in the BEA_HOME directory of WebLogic 10.3.

Cause

When Oracle bought BEA they decided to do away with this approach to licensing.

Solution

When you download WebLogic 10.3, you are getting the full version. That is, you do not need to do anything to fully enable the product. Although technically you do not need to do anything to use this product, legally you will need to purchase a license from Oracle. Contact an Oracle sales person for this.

Previous versions of WebLogic, including 10.0, have not been changed, so you still require the license.bea file for these.

Tuesday, July 21, 2009

SocketException: Too many open files


This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

On Unix-based systems, log message like:

Failed to listen on port 8081, failure count: 1, failing for 0 seconds, java.net.SocketException: Too many open files

Cause

The value of the "Maximum Open Socket" WebLogic setting is set higher than the the Unix limit (obtained by running the "ulimit -a" command).

Solution

Either decrease the value of "Maximum Open Socket" (located on the managed server's Tuning tab) to be less than the Unix limit, or increase the Unix limit. Investigate why there are so many requests being made.

References

Although this particular issue doesn't seem to be documented, you can see many other SocketException scenarios at http://www.insideexceptions.com/en/jdk-1-5-0/java-net-SocketException.html. It's a very helpful resource.

WebLogic Operator group has no effect


This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

In WebLogic 9.2, adding an account to the Operator group has no effect if the account is also added to another group.

Cause

This is a known bug in WebLogic 9.2.

Solution

A patch is provided with the 9.2 installation. Ensure %BEA_HOME%\patch_weblogic920\patch_jars\CR285163_920GA.jar; is on the classpath.

References

None.

DeleteService failed


This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

After removing a windows service, when trying to recreate it, the following error may be displayed:

DeleteService failed - The specified service has been marked for deletion. (0x430)

Cause

The service is still in-use (at least to some degree).

Solution

Close any Microsoft Management Console (MMC) sessions, kill any processes, etc. Note the service can be accessed remotely, so also consider that remote users may be viewing the service.

References

None.

NullPointerException at FileUtils.remove


This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

java.lang.NullPointerException
at weblogic.utils.FileUtils.remove(FileUtils.java:203)
at weblogic.management.mbeans.custom.Component.removeTempModule(Component.java:353)
at weblogic.management.mbeans.custom.Component.initializeTwoPhase(Component.java:435)
at weblogic.management.mbeans.custom.Component.initialize(Component.java:282)
at weblogic.management.mbeans.custom.EJBComponent.findOrCreateEJBDescriptor(EJBComponent.java:220)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at weblogic.management.internal.DynamicMBeanImpl.invokeLocally(DynamicMBeanImpl.java:755)
at weblogic.management.internal.DynamicMBeanImpl.invoke(DynamicMBeanImpl.java:734)
at weblogic.management.internal.ConfigurationMBeanImpl.invoke(ConfigurationMBeanImpl.java:516)
at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1557)
at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1525)
at weblogic.management.internal.RemoteMBeanServerImpl.private_invoke(RemoteMBeanServerImpl.java:990)
at weblogic.management.internal.RemoteMBeanServerImpl.invoke(RemoteMBeanServerImpl.java:948)
at weblogic.management.internal.MBeanProxy.invoke(MBeanProxy.java:948)
at weblogic.management.internal.MBeanProxy.invokeForCachingStub(MBeanProxy.java:475)
at weblogic.management.configuration.EJBComponentMBean_Stub.findOrCreateEJBDescriptor(EJBComponentMBean_Stub.java:1853)
at weblogic.management.console.actions.mbean.DoDeleteMBeanAction.perform(DoDeleteMBeanAction.java:203)
at weblogic.management.console.actions.internal.ActionServlet.doAction(ActionServlet.java:182)
at weblogic.management.console.actions.internal.ActionServlet.doPost(ActionServlet.java:86)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1072)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6981)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3892)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2766)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

Cause

Something seems to be preventing temporary data from being deleted (i.e. extracted application files, etc).

Solution

Delete managed server directories and restart. Applications may also need to be uninstalled and redeployed (not sure).

References

There are a number of other NullPointerException scenarios here: http://www.insideexceptions.com/en/jdk-1-5-0/java-lang-NullPointerException.html

User does not have permission on weblogic...

This is a simple symptom-cause-solution blog entry only. I hope these blogs will help fellow administrators.

Symptom

javax.naming.NoPermissionException: User does not have permission on weblogic.management.home to perform lookup operation.
at weblogic.jndi.internal.ServerNamingNode.checkPermission(ServerNamingNode.java:365)
at weblogic.jndi.internal.ServerNamingNode.checkLookup(ServerNamingNode.java:329)
at weblogic.jndi.internal.ServerNamingNode.lookupHere(ServerNamingNode.java:153)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:188)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:196)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:196)
at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:258)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:363)
at javax.naming.InitialContext.lookup(InitialContext.java:361)

Cause

In releases prior to WebLogic 8.1 sp5, the default value of "AnonymousAdminLookupEnabled" attribute of the domain is "true". But from WebLogic 8.1 sp5, its default value is "false". Hence this exception occurs.

Solution

To change this,in the weblogic console click on the domain. Select "View Domain-wide Security Settings" and set the "Anonymous Admin Lookup Enabled" checkbox. This will solve your problem.

References

None


WebLogic Books Reviewed

Looking at learning more about WebLogic? In this article I will give my opinions on the books I have encountered throughout my career.

If you're considering buying any of these books, then use the amazon links to support this site (and probably get the best available price).





BEA WebLogic Server 8.1: Unleashed

About four years ago I decided on a career change, and made the switch from Java developer to WebLogic Admin. On my first day on the job I was told that this book would be my bible, and to this day I still think it is the best WebLogic book available for learning WebLogic 8.1. In fact it is the only book I truly recommend, unless you wanting something updated for WebLogic 9 and later.









WebLogic: The Definitive Guide

I know this book gets reasonable reviews, but for a beginner or even every-day-admin I really wouldn't bother. There are definitely topics that are well covered and if its the topic you're after, you're in luck... The problem with this book is it doesn't cover the right topics for someone who has to look after WebLogic environments on a daily basis.

If you're looking at developing for WebLogic or becoming a hardcore WebLogic admin then you should consider this book. Otherwise spend your money on the BEA WebLogic 8.1 Unleashed.







Mastering BEA WebLogic Server

Mastering BEA WebLogic Server is aimed mostly at developers, but does contain some useful information for administrators. If you're interested in getting inside WebLogic then this is a good book. It teaches you to develop good WebLogic applications and then to build good WebLogic environments. The clustering and load-balancing topics would be informative to those new to an WebLogic admin role.

Unfortunately only the last 100 pages or so deal with admin topics, so you'd need to get it fairly cheap if you're not interested in the developer topics. If you pick up a cheap second-hand copy from Amazon, those last 100 pages are worth reading.





Professional Oracle WebLogic Server
Its been quite a while since a book on WebLogic has been published, so this soon to be released title is one I'm looking forward to getting my hands on. According to Amazon, this book will cover WebLogic 11g (I.e. WebLogic Server 10.3).

I'll update the article as soon as I get my hands on it.








Friday, July 17, 2009

Writing to WebLogic Server Logs

For J2EE developers who are not familiar with WebLogic, you can write debug and error messages to the server logs very easily.

When to Write to WebLogic Server Logs

Any debug, warning or error messages relating to the server or infrastructure. For example, information or errors with the following:
  • Connecting to other components/systems.
  • Database access, setup, etc.
  • LDAP
  • HTTP request/response
  • Deployment dependencies
Do not write to the server logs if you are handling the following:
  • Unexpected user input
  • Application debug/info/warning messages. (Write these to your own application logs.)

How to do it

Writing to the WebLogic server logs is simple. Just use LoggingHelper to get a Logger object and print messages to your heart's content. Here is some example code:

Put this in the class declaration:

private static java.util.logging.Logger serverLogger = LoggingHelper.getServerLogger();

Use these throughout your code:

serverLogger.log(Level.INFO, "Hello World!");
serverLogger.warning("This is a warning.");
serverLogger.severe("Something bad has happened!");

If you need more information about these classes, try these links:



BEA provides quite a lot of information about logging here:


Thursday, July 16, 2009

Quotes for the Corporate Sys Admin

Every now and then my job gets so boring that I feel like a robot. Today was one of those days, so I decided I needed a laugh and went in search of funny quotes relevant to working as a corporate sys admin... Hopefully there's one or two new ones for readers.

The man who smiles when things go wrong has thought of someone to blame it on.
-Robert Bloch
UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity.
-Dennis Ritchie
Doing it right is no excuse for not meeting the schedule.
-Plant Manager, Delco Corporation
Some things Man was never meant to know. For everything else, there's Google.
-Unknown
To err is human... to really foul up requires the root password.
-Unknown
Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots...
-Rich Cook
I had a fortune cookie the other day and it said: 'Outlook not so good'. I said: 'Sure, but Microsoft ships it anyway'.
-Unknown
Failure is not an option - it comes bundled with Windows.
-R_A_Z_N
Who the hell is General Failure? And why is he trying to read my hard disk?
-Unknown
This week I mapped and gapped the requirements to consolidate everything into a program of work... to maximize synergy, capture and optimize our resource utilization. If any of that sounded like work, I'll do some more of it next week.
-Wally, From Dilbert
Have a great day :)