Websphere deployment

[geofs]

Hi,

I've just installed jolokia's .war version 0.80 on a test websphere application server v7.0.0.11 and a WebSphere admin complains about "memory leaks" and MBeans that are not unregistered properly when application stops.

I tried filling a bug report in github, but I'm a bit confused, so let's report this here.

[roland]

Thanks for the report. In fact, the Jolokia agent uses two MBeans for configuration and history management. However, it does it very best to unregister them during shutdown or undeployment (HttpServlet.destroy())

I will investigate this issue. Well, I'm not a dedicated websphere expert, where on the Admin Console is is possible to detect Memory Leaks and unregistered MBeans ?

[roland]

Sorry, I probably mis-understood you. It is probably not a memory detection feature of the admin console what you meant, but a 'real' administrator ;-)

Of course, I would be curious to know a bit more about the technique with which the  Memory-Leak has been detected and kind of Memory Leak it is supposed to be (PermGen, Heap, ....)

* With Websphere 7.0.0.0.0 I could deploy, undeploy and deploy again the agent without problem. This indicates, that during undeployment the MBeans are properly unregistered, otherwise the second deployment would have failed since it registers the same MBeans again (on the same MBeanServer, which is a singleton for the JVM). This is true for start, stop, all kind of combinations.

* I used the new Tomcat 7 memory leak detector after several times of deployment and undeployment. Nothings shows up here, too. Also, we are using the agent in really large enviroments (500+ server), over a very long time. A memory leak didn't occur there, either.

So, I'm really curious about your findings, so that we can improve Jolokia even further.

[geofs]

Thanks for you reply.  Indeed, it's a real guy ;-) not the admin console.

He uses his own tool: http://code.google.com/p/arit

His tool detects application instances that have been stopped but for which the classloader can't be garbage-collected because the application didn't free all its resources (that's the "memory leak").

[roland]

Thanks for the pointer. I tried it also on Websphere 7.0.0.0.11 and I get a similar
result. However, there are some part of the report, which I don't buy.

I attached a screenshot of the result after deploying/undeploying jolokia
multiple times.

First of all it reports, that the MBean jolokia:type=Config still exists twice.
For the same MBeanServer (and Jolokia registers its MBean always on the same existing
MBeanServer), this cannot happen, since the name needs to be unique.

Also, as I said yesterday, Jolokia register the MBean with the same time each time it
starts. If such an MBean is already registered, Websphere would have croaked (I already saw
this sort of symptoms earlier).

Also, I wonder where the 'hanging' MemoryThread should come from. For sure, Jolokia
never obtained actively a reference to this Websphere thread.

So, my conclusion so far is: Either arit has some false detections or Websphere indeed has itself
a memory leak when dealing with custom MBeans.

However, I will try arit and Jolokia with different JEE Servers (jboss, weblogic) and will tell you the results.

Jolokia really doesn't something spectacular when it comes to MBean registration: It registers MBeans
on Servlet.init() and unregister them on Servlet.destroy().  Nothing uncommon.

I will keep you informed when I have new results.

[roland]

Just even stranger, I just tried it with the jmx4perl agent. Look at
the screenshot, arit reports two MBeans registered twice each (with
the same name !). I can gurantee, that each MBean is registered
only once during startup.

[roland]

Well, I think I found a hook. Need some time for investigation though (i.e. some remote debugging sessions).

[roland]

Short version:

I found the issue and fixed it. Please download the latest Snapshot war from  http://labs.consol.de/maven/snapshots-repository/org/jolokia/jolokia-war/0.81-SNAPSHOT/jolokia-war-0.81-20101024.035943-3.war
and try it again with Arit. All MBeans should now be deregister properly.

Long version:

In fact, it was a bug in WAS 7 which bites here. When registering an MBean via the MBeanRegistration-Interface mechanism (i.e. letting the MBean itself provide its name), then a NullPointerException is raised by WAS (that's the bug), but it already registered the MBean. Jolokia tries multiple MBeanServers for registering its own MBeans before giving up. So, after the Exception occured it assumes that the registration of the MBean has failed (which IMO is a fair assumption) and registers it with the second MBeanServer (in this case, the PlatformMBeanServer). Hence, there exists indeed 2 MBeans which identical names in the system (but for different MBeanServer). Arit was right.

The solution was to switch for WAS 7 to an explicite MBean registration with a given name. For WAS 6.1 I I still need the other indirect registration because only in this case, WAS doesn't add it server id as part of the MBean-Name (and I need a fixed MBean name so that I can predictably access this MBean).

Thanks for spotting this issue.

[geofs]

Thanks for the fix, I will try it now.

In fact, WebSphere adds some IDs to the MBean name (process, node and cell name) so the Deployment Manager knows where to contact the MBean.  So when registering, we must extract the name returned by WebSphere and use it when unregistering.  (Note that I'm not an expert so I'm just reporting what I understood).

-- geof

[geofs]

Ok, I can confirm the MBean are unregistered.

Thanks!