Support Migration Notice: To update migrated JIRA cases click here to open a new case use www.vmware.com/go/sr | vFabric Hyperic 5.7.0 is Now Available

Hyperic HQ

JMX plugin continues to reconnect jmx monitored app even when its down

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Cannot Reproduce
  • Affects Version/s: 3.2.5, 3.2.6, 4.0.1, 4.0.2, 4.0.3, 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.2.0
  • Fix Version/s: None
  • Component/s: Plugins

Description

1. Agent monitoring a JMX resource with many metrics
2. Disable JMX on the resource
3. Agent will attempt to retry JMX connections repeatedly using up the bulk of the system resources

To reproduce the issue of JMX plugin never giving up to reconnect to the JMX app thats not running use attached sample app.

1. Download mbeanExample.jar
2. Run it as follows (update port/path as needed) on a machine running HQ agent
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2121 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -jar mbeanExample.jar
3. Add this app to HQ as Sun JVM server (use jmx.url = service:jmx:rmi:///jndi/rmi://localhost:2121/jmxrmi under inventory -> configuration priperties)
4. Let it collect for few minutes and then kill the jmx app
5. Notice that agent will continue to connect to port 2121 once every minute

I see following in agent.log immediately after killing the jmx app. But these connection errors are not repeated after first time.
2009-07-21 10:46:04,859 ERROR [ScheduleThread] [ScheduleThread] Metric unreachable: Can't connect to MBeanServer [{jmx.url=service:jmx:rmi:///jndi/rmi://localhost:2121/jmxrmi}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.Ser
viceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]
2009-07-21 10:46:04,859 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11186
2009-07-21 10:46:06,859 ERROR [ScheduleThread] [ScheduleThread] Metric unreachable: Can't connect to MBeanServer [{jmx.url=service:jmx:rmi:///jndi/rmi://localhost:2121/jmxrmi}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.Ser
viceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]
2009-07-21 10:46:06,859 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 2:10308
2009-07-21 10:46:09,875 ERROR [ScheduleThread] [ScheduleThread] Metric unreachable: Can't connect to MBeanServer [{jmx.url=service:jmx:rmi:///jndi/rmi://localhost:2121/jmxrmi}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.Ser
viceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]
2009-07-21 10:46:09,875 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11185
2009-07-21 10:46:11,890 ERROR [ScheduleThread] [ScheduleThread] Metric unreachable: Can't connect to MBeanServer [{jmx.url=service:jmx:rmi:///jndi/rmi://localhost:2121/jmxrmi}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.Ser
viceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]
2009-07-21 10:46:11,890 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11189
2009-07-21 10:46:13,906 ERROR [ScheduleThread] [ScheduleThread] Metric unreachable: Can't connect to MBeanServer [{jmx.url=service:jmx:rmi:///jndi/rmi://localhost:2121/jmxrmi}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.Ser
viceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]
2009-07-21 10:46:13,906 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11191
2009-07-21 10:46:15,921 ERROR [ScheduleThread] [ScheduleThread] Metric unreachable: Can't connect to MBeanServer [{jmx.url=service:jmx:rmi:///jndi/rmi://localhost:2121/jmxrmi}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.Ser
viceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]
2009-07-21 10:46:15,921 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11190
2009-07-21 10:46:17,921 ERROR [ScheduleThread] [ScheduleThread] Metric unreachable: Can't connect to MBeanServer [{jmx.url=service:jmx:rmi:///jndi/rmi://localhost:2121/jmxrmi}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.Ser
viceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]
2009-07-21 10:46:17,921 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11187
2009-07-21 10:46:19,843 ERROR [ScheduleThread] [ScheduleThread] Metric unreachable: Can't connect to MBeanServer [{jmx.url=service:jmx:rmi:///jndi/rmi://localhost:2121/jmxrmi}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.Ser
viceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]
2009-07-21 10:46:19,843 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11188
2009-07-21 10:52:00,000 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11186

I continue to see enabling/disabling metris INFO/WARN messages. These are metrics for JMX app which is down.

2009-07-21 16:10:11,031 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11191
2009-07-21 16:10:11,156 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11190
2009-07-21 16:10:13,156 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11190
2009-07-21 16:10:13,156 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11187
2009-07-21 16:10:15,156 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11187
2009-07-21 16:10:15,171 INFO [ScheduleThread] [ScheduleThread] Re-enabling metrics for: 3:11188
2009-07-21 16:10:17,171 WARN [ScheduleThread] [ScheduleThread] Disabling metrics for: 3:11188

Activity

Hide
Kashyap Parikh added a comment -

CF42

Show
Kashyap Parikh added a comment - CF42
Hide
David Wiener added a comment -

Closed due to being outdated

Show
David Wiener added a comment - Closed due to being outdated

People

Vote (1)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved:
    Last comment:
    1 year, 37 weeks, 2 days ago