Websphere deployment

classic Classic list List threaded Threaded
10 messages Options
geofs geofs
Reply | Threaded
Open this post in threaded view
|

Websphere deployment

Hi,

I've just installed jolokia's .war version 0.80 on a test websphere application server v7.0.0.11 and a WebSphere admin complains about "memory leaks" and MBeans that are not unregistered properly when application stops.

I tried filling a bug report in github, but I'm a bit confused, so let's report this here.
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: Websphere deployment

Thanks for the report. In fact, the Jolokia agent uses two MBeans for configuration and history management. However, it does it very best to unregister them during shutdown or undeployment (HttpServlet.destroy())

I will investigate this issue. Well, I'm not a dedicated websphere expert, where on the Admin Console is is possible to detect Memory Leaks and unregistered MBeans ?
... roland
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: Websphere deployment

In reply to this post by geofs
Sorry, I probably mis-understood you. It is probably not a memory detection feature of the admin console what you meant, but a 'real' administrator ;-)

Of course, I would be curious to know a bit more about the technique with which the  Memory-Leak has been detected and kind of Memory Leak it is supposed to be (PermGen, Heap, ....)

* With Websphere 7.0.0.0.0 I could deploy, undeploy and deploy again the agent without problem. This indicates, that during undeployment the MBeans are properly unregistered, otherwise the second deployment would have failed since it registers the same MBeans again (on the same MBeanServer, which is a singleton for the JVM). This is true for start, stop, all kind of combinations.

* I used the new Tomcat 7 memory leak detector after several times of deployment and undeployment. Nothings shows up here, too. Also, we are using the agent in really large enviroments (500+ server), over a very long time. A memory leak didn't occur there, either.

So, I'm really curious about your findings, so that we can improve Jolokia even further.
... roland
geofs geofs
Reply | Threaded
Open this post in threaded view
|

Re: Websphere deployment

Thanks for you reply.  Indeed, it's a real guy ;-) not the admin console.

He uses his own tool: http://code.google.com/p/arit

His tool detects application instances that have been stopped but for which the classloader can't be garbage-collected because the application didn't free all its resources (that's the "memory leak").

roland roland
Reply | Threaded
Open this post in threaded view
|

Re: Websphere deployment

Thanks for the pointer. I tried it also on Websphere 7.0.0.0.11 and I get a similar
result. However, there are some part of the report, which I don't buy.

I attached a screenshot of the result after deploying/undeploying jolokia
multiple times.

First of all it reports, that the MBean jolokia:type=Config still exists twice.
For the same MBeanServer (and Jolokia registers its MBean always on the same existing
MBeanServer), this cannot happen, since the name needs to be unique.

Also, as I said yesterday, Jolokia register the MBean with the same time each time it
starts. If such an MBean is already registered, Websphere would have croaked (I already saw
this sort of symptoms earlier).

Also, I wonder where the 'hanging' MemoryThread should come from. For sure, Jolokia
never obtained actively a reference to this Websphere thread.

So, my conclusion so far is: Either arit has some false detections or Websphere indeed has itself
a memory leak when dealing with custom MBeans.

However, I will try arit and Jolokia with different JEE Servers (jboss, weblogic) and will tell you the results.

Jolokia really doesn't something spectacular when it comes to MBean registration: It registers MBeans
on Servlet.init() and unregister them on Servlet.destroy().  Nothing uncommon.

I will keep you informed when I have new results.

arit and Jolokia

... roland
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: Websphere deployment

In reply to this post by geofs
Just even stranger, I just tried it with the jmx4perl agent. Look at
the screenshot, arit reports two MBeans registered twice each (with
the same name !). I can gurantee, that each MBean is registered
only once during startup.

... roland
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: Websphere deployment

Well, I think I found a hook. Need some time for investigation though (i.e. some remote debugging sessions).
... roland
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: Websphere deployment

Short version:

I found the issue and fixed it. Please download the latest Snapshot war from  http://labs.consol.de/maven/snapshots-repository/org/jolokia/jolokia-war/0.81-SNAPSHOT/jolokia-war-0.81-20101024.035943-3.war
and try it again with Arit. All MBeans should now be deregister properly.

Long version:

In fact, it was a bug in WAS 7 which bites here. When registering an MBean via the MBeanRegistration-Interface mechanism (i.e. letting the MBean itself provide its name), then a NullPointerException is raised by WAS (that's the bug), but it already registered the MBean. Jolokia tries multiple MBeanServers for registering its own MBeans before giving up. So, after the Exception occured it assumes that the registration of the MBean has failed (which IMO is a fair assumption) and registers it with the second MBeanServer (in this case, the PlatformMBeanServer). Hence, there exists indeed 2 MBeans which identical names in the system (but for different MBeanServer). Arit was right.

The solution was to switch for WAS 7 to an explicite MBean registration with a given name. For WAS 6.1 I I still need the other indirect registration because only in this case, WAS doesn't add it server id as part of the MBean-Name (and I need a fixed MBean name so that I can predictably access this MBean).

Thanks for spotting this issue.
... roland
geofs geofs
Reply | Threaded
Open this post in threaded view
|

Re: Websphere deployment

Thanks for the fix, I will try it now.

In fact, WebSphere adds some IDs to the MBean name (process, node and cell name) so the Deployment Manager knows where to contact the MBean.  So when registering, we must extract the name returned by WebSphere and use it when unregistering.  (Note that I'm not an expert so I'm just reporting what I understood).

-- geof

geofs geofs
Reply | Threaded
Open this post in threaded view
|

Re: Websphere deployment

In reply to this post by roland
Ok, I can confirm the MBean are unregistered.

Thanks!