Trouble retrieving runtime information from WebLogic 10.3

classic Classic list List threaded Threaded
5 messages Options
Maciej Maciej
Reply | Threaded
Open this post in threaded view
|

Trouble retrieving runtime information from WebLogic 10.3

Hi,

first of all kudos for developing Jolokia!

I am using jmx4perl on the client side. Jolokia is deployed in agent mode on 2 WebLogic Servers: one AdminServer and one Managed Server. I was able to successfully gain access to a lot of mbean attributes using jmx4perl.

My goal is to monitor the actual states of my deployments. Not just the fact, that they are deployed, but also if they are active or not.

As I understand, Jolokia "merges" all MBean Servers into one virtual MBeanServer (http://labs.consol.de/blog/jmx4perl/configuring-remote-jmx-access-for-weblogic/)

To find out all the possible MBeans I can access I did two things:
1) deployed wlnav.war, which is a MBean browser
2) did an ls() from j4psh and stored the output in a file

I believe there are differences. For example: I couldn't access any MBean from the domain runtime on the Admin Server

I cannot access some runtime information, like
com.bea:Name=AppRuntimeStateRuntime,Type=AppRuntimeStateRuntime
com.bea:Name=my_domain,Type=DomainRuntime

which I have found using wlnav
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: Trouble retrieving runtime information from WebLogic 10.3

Hi Maciej,
Maciej wrote
As I understand, Jolokia "merges" all MBean Servers into one virtual MBeanServer (http://labs.consol.de/blog/jmx4perl/configuring-remote-jmx-access-for-weblogic/)

Yes, that's true. It tries all MBeanServer it can find in turn and uses the first MBean found. However, the MBeanServer detection is product specific, so it could be easily the case that I 'missed' and MBeanServer.

To find out all the possible MBeans I can access I did two things: 1) deployed wlnav.war, which is a MBean browser 2) did an ls() from j4psh and stored the output in a file

Hint: You can also use 'jmx4perl http://.... list' which is an easier way to get the list of available MBeans. Or even 'jmx4perl http://... attributes' which will also try to fetch all attributes of all MBeans. (Please be aware that this can be A LOT of data, > 200 MB or so).

I believe there are differences. For example: I couldn't access any MBean from the domain runtime on the Admin Server I cannot access some runtime information, like com.bea:Name=AppRuntimeStateRuntime,Type=AppRuntimeStateRuntime com.bea:Name=my_domain,Type=DomainRuntime which I have found using wlnav

For Weblogic, the following MBeanServers are examined:

  • The MBeanServer found under the JNDI name "java:comp/env/jmx/runtime"
  • All MBeanServers returned from MBeanServerFactory.findMBeanServer(null);
  • ManagementFactory.getPlatformMBeanServer()

So, I really didn't merged in all MBeanServer available as listed in my blog post. I will check this out and look for the missing MBean's there (and if so, adapt the MBeanServer detector for Weblogic accordingly).

Is there a way to get this 'wlnav' app (maybe even with source) so that I can verify and examine the situation in more depth ?

... roland
Maciej Maciej
Reply | Threaded
Open this post in threaded view
|

Re: Trouble retrieving runtime information from WebLogic 10.3

In reply to this post by Maciej
Thanks for the hint. The 'list' command makes it a lot easier.

1. WLNav hasn't been updated since 2007, but according to the information on the site it works for 10.x as well:
http://wlnav.open.collab.net/
After deployment you just go to <a href="http://host:port/wlnav">http://host:port/wlnav to browse MBeans.

2. When I inspect my ManagedServer I see 3 Tabs: Server Runtime, Server Config and Custom. I assume each tab is an access to a different MBean server. On the AdminServer I see additional tabs: Edit, Domain Runtime, Domain Config. I use the MBean description displayed after "CMO: [MBeanServerInvocationHandler]" to determine the MBean name which I use in the jmx4perl call.

3. When I navigate to the Domain Runtime tab I see some MBeans I would like to read. When I execute:
jmx4perl http://host:7001/jolokia read com.bea:Name=my_domain,Type=DomainRuntime
I get:
ERROR: javax.management.InstanceNotFoundException : No MBean with ObjectName com.bea:Name=my_domain,Type=DomainRuntime found in any MBeanServer

That happens of course to any MBean in the hierarchy below, like the AppRuntimeStateRuntime MBean. Also, there is no AppRuntimeStateRuntime or DomainRuntime in the output retrieved from 'jmx4perl list' command.

You can find the DomainRuntime MBean in the reference manual (Runtime MBeans -> DomainRuntimeMBean -> AppRuntimeStateRuntime)
http://download.oracle.com/docs/cd/E12840_01/wls/docs103/wlsmbeanref/core/index.html

However, given the fact that the AdminServer is not required to run in production, I am going to setup my monitoring for each managed server separately. Therefore access to these MBeans is not critical at the moment. It would be better to see them though.

3. I can retrieve information from Domain Config (as opposed to Domain Runtime), but only after leaving out the "Location=" part of the MBean description.
OK:            com.bea:Name=my_domain,Type=Domain
Not found: com.bea:Name=my_domain,Location=my_domain,Type=Domain

4. When running jmx4perl without any commands jolokia wrongly identifies the server as Glassfish, but after adding '-product weblogic' it correctly identifies the server version. I am using WebLogic 10.3.5 for reference sake.

Thanks
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: Trouble retrieving runtime information from WebLogic 10.3

Hi Maciej,
Maciej wrote
3. When I navigate to the Domain Runtime tab I see some MBeans I would like to read. When I execute: jmx4perl http://host:7001/jolokia read com.bea:Name=my_domain,Type=DomainRuntime I get: ERROR: javax.management.InstanceNotFoundException : No MBean with ObjectName com.bea:Name=my_domain,Type=DomainRuntime found in any MBeanServer
Yes, that's true. As explained in my post there are three kinds of MBeanServer available. Unfortunately, only the Runtime MBeanServer can be accessed locally:
JMX clients running within a WebLogic Server JVM can access the server’s Runtime MBean Server directly through JNDI and must be authenticated to do so. This is the only WebLogic Server MBean server that allows local access. When accessed from a local client, the Runtime MBean Server returns its javax.management.MBeanServer interface, which enables clients to access WebLogic Server Means and to create, register, and access custom MBeans.
(from the Oracle documentation)

wlnav is using a different way for getting to the MBean (via wlst as far as I've seen). I will look into its source code of wlnav, but for now, the Runtime MBean Server is all the Jolokia agent can access for now.

You can find the DomainRuntime MBean in the reference manual (Runtime MBeans -> DomainRuntimeMBean -> AppRuntimeStateRuntime) http://download.oracle.com/docs/cd/E12840_01/wls/docs103/wlsmbeanref/core/index.html
As this MBean is registered at the Domain Runtime MBeanServer, it is not available locally as I would need it for the agent.
3. I can retrieve information from Domain Config (as opposed to Domain Runtime), but only after leaving out the "Location=" part of the MBean description. OK: com.bea:Name=my_domain,Type=Domain Not found: com.bea:Name=my_domain,Location=my_domain,Type=Domain
Probably the reason for this explained somewhat is also explained in the document mentioned above:
When you access runtime MBeans or configuration MBeans through the Domain Runtime MBean Server, the MBean object names include a Location=servername key property which specifies the name of the server instance on which that MBean is located. See MBean Servers. Singleton MBeans, such as DomainRuntimeMBean and ServerLifeCycleRuntimeMBean exist only on the Administration Server and do not need to include this key property.
Next point:
4. When running jmx4perl without any commands jolokia wrongly identifies the server as Glassfish, but after adding '-product weblogic' it correctly identifies the server version. I am using WebLogic 10.3.5 for reference sake.
That's a bug in jmx4perl, indeed. If you point your browser to http://..../jolokia, you will see, that the agent itself recognizes the server properly. I will fix this for jmx4perl, thanks for spotting this.
... roland
santony santony
Reply | Threaded
Open this post in threaded view
|

Re: Trouble retrieving runtime information from WebLogic 10.3

In reply to this post by Maciej
Hi

Don't know if it can help :

We were trying to use jolokia war agent 1.3.7 on weblogic 12.1.3 servers, to replace a custom web-app (working the same way finally to get mbeans informations).
With our custom web-app we were able to get from AdminServer informations from all managed servers int the domain, throught the DomainRuntime MBeanServer.

After some investigations on jolokia, we found that we have access only to the platform mbean server and not to the servers defined for domain by weblogic... It finally lead us to this post.

As said in previous post :
For Weblogic, the following MBeanServers are examined:

The MBeanServer found under the JNDI name "java:comp/env/jmx/runtime"
All MBeanServers returned from MBeanServerFactory.findMBeanServer(null);
ManagementFactory.getPlatformMBeanServer()

In fact with these informations we get the sources, and only add a little fix to add a lookup for the domain runtime (java:comp/env/jmx/domainRuntime), in agent-core project -> WeblogicDetector and replace the jar obtained by the new one :
try {
                ctx = new InitialContext();
                MBeanServer serverDomainRuntime = (MBeanServer) ctx.lookup("java:comp/env/jmx/domainRuntime");
                if (serverDomainRuntime != null) {
                    servers.add(serverDomainRuntime);
                }
            } catch (NamingException e) {
                // expected and can happen on non-Weblogic platforms
            }

After that, we can have access to mbeans from the domain runtime (with Location=...) from the AdminServer, giving us the same centralized view of all our managed servers too. On ManagedServers, this mbean server is not available and we still have basics mbeans (as on AdminServer if we want).

Hope will help