WebSphere javax.management.InstanceNotFoundException

classic Classic list List threaded Threaded
11 messages Options
Mark Mark
Reply | Threaded
Open this post in threaded view
|

WebSphere javax.management.InstanceNotFoundException

Hi,
I made a list of all mbeans with:

  jmx4perl https://xxx-wps01.xxx/jolokia list >mbean.list

This works:

  jmx4perl https://xxx-wps01.xxx/jolokia read "java.lang:type=Memory"

But nothing at all in the 'WebSphere:' mbean list works, E.g.

  jmx4perl https://xxx-wps01.xxx/jolokia read "WebSphere:J2EEServer=RMRS.Support.wps2node1.0,Server=RMRS.Support.wps2node1.0,cell=wpscell01,j2eeType=JCAResource,mbeanIdentifier=cells/wpscell01/clusters/RMRS.Support/resources.xml#J2CResourceAdapter_1295960256813,name=SIB JMS Resource Adapter,node=wps2node1,platform=dynamicproxy,process=RMRS.Support.wps2node1.0,spec=1.0,type=J2CResourceAdapter,version=7.0.0.19"

ERROR: javax.management.InstanceNotFoundException : WebSphere:J2EEServer=RMRS.Support.wps2node1.0,Server=RMRS.Support.wps2node1.0,cell=wpscell01,j2eeType=JCAResource,mbeanIdentifier=cells

Any ideas?

Cheers
Mark.
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Hi Mark,

can you please tell me, what version of jmx4perl (jmx4perl --version) and Jolokia (point browser to https://xxx-wps01.xxx/jolokia) you are using ? There has been a switch for the escaping algorithm for / from pre-1.0 to 1.0 in Jolokia.

If you use jmx4perl >= 1.0 with jolokia < 1.0 you can use the '--legacy-escape' flag for the jmx4perl call.

If you jolokia >= 1.0 you should use jmx4perl >= 1.0.

Said this, you can always circumvent escaping by using POST as HTTP method with '--method POST' as extra option. Unfortunately, this option is undocumented in jmx4perl (which I'm just about to fix).
... roland
Mark Mark
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Hi Roland,
Thanks, it actually was initially an escaping issue and then I upgraded jmx4perl, so:

# jmx4perl --version
jmx4perl 1.04

# curl -kn https://xxx-wps01.xxx/jolokia 
{"value":{"agent":"1.0.2","protocol":"6.1","info":{"version":"7.0.0.19","vendor":"IBM","extraInfo":{"buildDate":"8\/13\/11"},"product":"websphere"}},"status":200,"timestamp":1330512979,"request":{"type":"version"}}

It looks like it might be some sort of permission issue within WebSphere but I can't see how to fix it. Getting the list of mbeans works but accessing attributes doesn't, E.g.

[root@ldld3v-mon01 ~]# jmx4perl https://xxx-wps01.xxx/jolokia search '*:j2eeType=J2EEServer,*'
WebSphere:cell=wpscell01,j2eeType=J2EEServer,mbeanIdentifier=cells/wpscell01/nodes/wps1node2/servers/RMRS.Support.wps1node2.0/server.xml#Server_1295974375714,name=RMRS.Support.wps1node2.0,node=wps1node2,platform=proxy,process=RMRS.Support.wps1node2.0,processType=ManagedProcess,spec=1.0,type=Server,version=7.0.0.19

# jmx4perl https://xxx-wps01.xxx/jolokia read '*:j2eeType=J2EEServer,*'
ERROR: javax.management.RuntimeOperationsException : Exception occured trying to invoke the getter on the MBean

Any ideas?
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Can you please call jmx4perl with '--verbose'  and post the output ?

thx ...
... roland
Mark Mark
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Ah! :) Is this the problem?

Caused by: javax.management.JMRuntimeException: ADMN0022E: Access is denied for the getName operation on Server MBean because of insufficient or empty credentials.

But where do I put the credentials? Full output follows:

ERROR: javax.management.RuntimeOperationsException : Exception occured trying to invoke the getter on the MBean
Server Stacktrace:
javax.management.RuntimeOperationsException: Exception occured trying to invoke the getter on the MBean
        at com.ibm.ws.management.PlatformMBeanServer.getAttribute(PlatformMBeanServer.java:682)
        at org.jolokia.handler.ReadHandler.getAttribute(ReadHandler.java:223)
        at org.jolokia.handler.ReadHandler.fetchAttributes(ReadHandler.java:162)
        at org.jolokia.handler.ReadHandler.fetchAttributesForMBeanPattern(ReadHandler.java:104)
        at org.jolokia.handler.ReadHandler.doHandleRequest(ReadHandler.java:89)
        at org.jolokia.handler.ReadHandler.doHandleRequest(ReadHandler.java:34)
        at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:157)
        at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:99)
        at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:83)
        at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:197)
        at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:175)
        at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:148)
        at org.jolokia.http.HttpRequestHandler.handleGetRequest(HttpRequestHandler.java:78)
        at org.jolokia.http.AgentServlet$3.handleRequest(AgentServlet.java:269)
        at org.jolokia.http.AgentServlet.handle(AgentServlet.java:213)
        at org.jolokia.http.AgentServlet.doGet(AgentServlet.java:195)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
        at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
        at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3935)
        at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
        at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:931)
        at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
        at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
        at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1048)
        at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:642)
        at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
        at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
        at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
        at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
        at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
        at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
        at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
        at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Caused by: javax.management.JMRuntimeException: ADMN0022E: Access is denied for the getName operation on Server MBean because of insufficient or empty credentials.
        at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2380)
        at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2187)
        at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2081)
        at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2054)
        at com.ibm.ws.management.AdminServiceImpl.getAttribute(AdminServiceImpl.java:853)
        at com.ibm.ws.management.PlatformMBeanServer.getAttribute(PlatformMBeanServer.java:679)
        ... 41 more
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Hi Mark, things a bit complicated with Websphere. Although I'm not really a Websphere expert, I tracked this security issue down somehwat for Websphere 6.1.0.33 (should be true for Websphere 7 as well). What version of Websphere are you using ?

Here's the drilldown:

  • If you have global security switched off, it should always work
  • If global security is swtichedn on, that several things needs to be done described below
First you need to enable security in the Jolokia agent's web.xml by either using the command line tool 'jolokia' which comes with 'jmx4perl' or by editing manually the web.xml found in jolokia.war. I recommend using the 'jolokia' CLI command, please refer to its documentation for details. Here is an example:
13:49 [~] $ jolokia
* Loading Jolokia meta data from http://www.jolokia.org/jolokia.meta
* Good PGP signature, signed by Roland Huss <roland@jmx4perl.org> (EF101165)
* Using Jolokia 1.0.2 for Jmx4Perl 1.04
* Downloading war agent version 1.0.2 from repository http://labs.consol.de/maven/repository
* Saved ./jolokia.war
* Good PGP signature, signed by Roland Huss <roland@jmx4perl.org> (EF101165)                                                                        

13:50 [~] $ jolokia repack --security --security-role JMX jolokia.war
* Added security mapping for role JMX
* Updated web.xml for jolokia.war
Alternatively you could add it manually to web.xml:
 <login-config>
   <auth-method>BASIC</auth-method>
   <realm-name>WSRealm</realm-name>
  </login-config>

 <security-constraint>
   <web-resource-collection>
     <web-resource-name>jolokia-Agent Access</web-resource-name>
     <url-pattern>/*</url-pattern>
   </web-resource-collection>
   <auth-constraint>
     <role-name>JMX</role-name>
   </auth-constraint>
 </security-constraint>

 <security-role>
   <role-name>JMX</role-name>
 </security-role>
Next, the user needs to be setup in the WAS Admin-Console:
  • A Administrative user should be added with at least the role 'monitor'
  • Under "Manage Users" should be a password set for user 'roland'
  • The role "JMX" referenced in the web.xml must be mapped to this user (Enterprise Applications -> jolokia.war -> Security role to user/group mapping)
Also be sure, that you have 'Application Security' enabled in WebSphere ("Security" --> "Secure administration, applications, and infrastructure" --> "Enable application security")

With this setup, I was able to connect to the WAS with global security switched on. Note, that the 'Java 2 Security' option won't work, since I would have to make some changes to the codebase.

Please try it out and let me know whether this work for you.

... roland
Mark Mark
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Hi Roland,
Many thanks for your detailed reply.

This works perfectly for WebSphere 7 - (WAS versions 7.0.0.17 and 7.0.0.19).

After websphere configuration I just needed to add '--user' and '--password' when querying with jmx4perl and check_jmx4perl.

Thanks again!
Mark Clarkson
Bala Bala
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Hi,
The same issue we are facing for accessing MBEAN(by SNMP request from ireasoning MIB browser). We are not able to access the MBEAN. How can we send the username and password in SNMP request. How to add -username, -password in the request. or is there anyway to disable the MBean security in Websphere 7 application server?
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Sorry, accessing JMX via SNMP is a complete different beast and has nothing to do with Jolokia.
... roland
Nirav Shah Nirav Shah
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Hi Roland,

I am net to jolokia.I am currently trying to setup Jolokia for Websphere liberty Profile and trying to access same using just web URL.

Based on your comment I have added necessary entry in jolokia.war and also made necessary changes in server.xml .

However I am still not ablet to access websphere JMX details like connectionPool and other details.

I am using Websphere Liberty 8.5.5.1. Added Appsecuity and Servlet 3.0 featuer for enabling java security.

 <featureManager>
        <feature>monitor-1.0</feature>
        <feature>jsp-2.2</feature>
        <feature>jdbc-4.0</feature>
        <feature>appSecurity-2.0</feature>
        <feature>servlet-3.0</feature>
    </featureManager>

Following entry to server.xml .

<basicRegistry id="basic" realm="Jolokia">
                <user name="JolokiaUser" password="Passw0rd" />
</basicRegistry>

<administrator-role>
     <user>JolokiaUser</user>
 </administrator-role>

<application id="jolokia" location="${shared.app.dir}/jolokia-war-1.2.0.war"
       name="jolokia" type="war" context-root="/jolokia">
        <application-bnd>
                         <security-role name="Jolokia">
                                        <user name="JolokiaUser" />
                                </security-role>
                </application-bnd>
    </application>

 I am using HTTP mode to connect. while I am getting console to enter the user details and able to login successfully . not able to get websphere details using that.


I am only getting JVM stats nothing else.

http://XX.XX.XX.XX:9085/jolokia/read/WebSphere:type=*

{"timestamp":1399857175,"status":200,"request":{"mbean":"WebSphere:type=*","type":"read"},"value":{"WebSphere:type=JvmStats":{"GcCount":11,"ProcessCPU":-1.0,"Heap":1073676288,"UsedMemory":171549848,"FreeMemory":902126440,"UpTime":4246792,"GcTime":531}}}


If you can help that would be great.

Thanks & regards
Nirav Shah
roland roland
Reply | Threaded
Open this post in threaded view
|

Re: WebSphere javax.management.InstanceNotFoundException

Sorry, there is not much I can do for you, since I'm really not a WebSphere expert and do not have much experience with WebSphere MBeans. Since you don't get an error when accessing the Jolokia Agent I guess it's not really an Jolokia issue.

However, I would try the following:

* What shows up when you access JMX via a "native" JMX console (like jconsole or maybe Websphere internal view). Are there any other MBean in the realm 'Websphere:' showing up ?
* What do you receive when you are doing a "list" command (http://.../jolokia/list) Are there any mising MBeans contained ?

It might be, that you have to enable extra MBeans. I know for Glassfish you have to "boot" the AMX subsystem to get any substantial information. Maybe that's the same for WebSphere ?

Sorry, but you better probably ask the IBM community how to enable extra MBeans ?

... roland
... roland