Support Migration Notice: To update migrated JIRA cases click here to open a new case use www.vmware.com/go/sr | vFabric Hyperic 5.7.0 is Now Available

Hyperic HQ

ObjectNotFoundException from MetricsNotComingInDiagnostic@ on deleting a resource that was not reporting metrics

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Deferred
  • Affects Version/s: 4.5
  • Fix Version/s: None
  • Component/s: None
  • Environment:
    MySQL, CentOS, build #105, 1342 platforms
  • Case Links:
    none
  • Regression:
    Yes

Description

Scenario:

Hq-server was upgraded on the server host where an agent is deployed. Hence, the previous ActiveMQ 4.3 server was listed as down and not reporting metrics in the past 8 hours.

Steps:
1.Login to the HQ UI and go to Resource Hub Page
2.Select older hq-server and delete the resource

The delete is successful as seen from the UI

In the server log, the following exception is thrown :

2010-09-16 09:56:38,841 INFO [ControlEventListener1] [org.hyperic.hq.control.server.session.ControlEventListener@77] Removing scheduled jobs for 3:41410
2010-09-16 09:56:38,935 DEBUG [AgentScheduler2] [org.hyperic.hq.measurement.server.session.MeasurementProcessorImpl@273] unschedule agentId=10100, numOfResources=6
2010-09-16 09:56:38,998 INFO [ControlEventListener1] [org.hyperic.hq.control.server.session.ControlEventListener@77] Removing scheduled jobs for 3:41411
2010-09-16 09:56:39,044 INFO [ControlEventListener1] [org.hyperic.hq.control.server.session.ControlEventListener@77] Removing scheduled jobs for 3:41412
2010-09-16 09:56:39,083 INFO [ControlEventListener1] [org.hyperic.hq.control.server.session.ControlEventListener@77] Removing scheduled jobs for 3:41413
2010-09-16 09:56:39,141 INFO [ControlEventListener1] [org.hyperic.hq.control.server.session.ControlEventListener@77] Removing scheduled jobs for 3:41414
2010-09-16 09:56:39,176 INFO [ControlEventListener1] [org.hyperic.hq.control.server.session.ControlEventListener@77] Removing scheduled jobs for 3:41415
2010-09-16 09:58:58,813 ERROR [Thread-3] [org.hyperic.hq.measurement.MetricsNotComingInDiagnostic@126] org.springframework.orm.hibernate3.HibernateSystemException: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: org.hyperic.hq.authz.server.session.Resource#50803; nested exception is org.hibernate.HibernateException: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: org.hyperic.hq.authz.server.session.Resource#50803
org.springframework.orm.hibernate3.HibernateSystemException: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: org.hyperic.hq.authz.server.session.Resource#50803; nested exception is org.hibernate.HibernateException: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: org.hyperic.hq.authz.server.session.Resource#50803
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:679)
at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:412)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
at org.hyperic.hq.hibernate.SessionManager.runInSessionInternal(SessionManager.java:83)
at org.hyperic.hq.hibernate.SessionManager.runInSession(SessionManager.java:67)
at org.hyperic.hq.measurement.MetricsNotComingInDiagnostic.getStatus(MetricsNotComingInDiagnostic.java:116)
at org.hyperic.hq.common.DiagnosticsLogger.run(DiagnosticsLogger.java:104)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.hibernate.HibernateException: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: org.hyperic.hq.authz.server.session.Resource#50803
at org.hyperic.hq.hibernate.SessionManager$1.doInHibernate(SessionManager.java:88)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
... 6 more
Caused by: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: org.hyperic.hq.authz.server.session.Resource#50803
at org.hibernate.impl.SessionFactoryImpl$1.handleEntityNotFound(SessionFactoryImpl.java:377)
at org.hibernate.proxy.AbstractLazyInitializer.checkTargetState(AbstractLazyInitializer.java:79)
at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:68)
at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:111)
at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.invoke(CGLIBLazyInitializer.java:150)
at org.hyperic.hq.authz.server.session.Resource$$EnhancerByCGLIB$$800ad285.hashCode(<generated>)
at java.util.HashMap.put(HashMap.java:372)
at org.hyperic.hq.measurement.MetricsNotComingInDiagnostic.getChildren(MetricsNotComingInDiagnostic.java:217)
at org.hyperic.hq.measurement.MetricsNotComingInDiagnostic.setStatusBuf(MetricsNotComingInDiagnostic.java:139)
at org.hyperic.hq.measurement.MetricsNotComingInDiagnostic$1.run(MetricsNotComingInDiagnostic.java:118)
at org.hyperic.hq.hibernate.SessionManager$1.doInHibernate(SessionManager.java:86)
... 7 more
2010-09-16 09:58:58,834 INFO [Thread-3] [org.hyperic.hq.common.DiagnosticsLogger@104] [org.hyperic.hq.measurement.MetricsNotComingInDiagnostic@b9f7607]

Filing as major since it's an exception but does not impact the result of the action itself
Have not seen this behavior in 4.3 and did not find such a bug logged in 4.3 earlier, hence marking as regression.

  1. hqstats-09-16-16.44.34.csv
    17/Sep/10 3:11 PM
    1.19 MB
    Dharma Srinivasan
  2. server.09-15.server105.log.gz
    16/Sep/10 10:42 AM
    211 kB
    Dharma Srinivasan
  1. cache_misses.png
    75 kB
    17/Sep/10 3:11 PM

Issue Links

Activity

Hide
Kashyap Parikh added a comment -

Dharma, can you link the metrics not reachable diagnostic report performance issue bug with this one. We need to take another look at this report in 4.5. It's not only causing performance issue but annoyances like above.

Does the report run successfully next tenth of an hour?

Show
Kashyap Parikh added a comment - Dharma, can you link the metrics not reachable diagnostic report performance issue bug with this one. We need to take another look at this report in 4.5. It's not only causing performance issue but annoyances like above. Does the report run successfully next tenth of an hour?
Hide
Dharma Srinivasan added a comment -

Kashyap,

The next metrics not collecting diagnostics were every 18 mins in the log (after that msg was at 10:16 am (error above was 9:58), next 10:34, then 10:52)
Next day's log shows metricsnotcollecting report every 16 mins

2010-09-16 10:16:53,677 INFO [Thread-3] [org.hyperic.hq.common.DiagnosticsLogger@104] [org.hyperic.hq.measurement.MetricsNotComingInDiagnostic@b9f7607]
Enabled metrics not reported in for 60 minutes (by platform hierarchy)
------------------------------------------------------------------------

Show
Dharma Srinivasan added a comment - Kashyap, The next metrics not collecting diagnostics were every 18 mins in the log (after that msg was at 10:16 am (error above was 9:58), next 10:34, then 10:52) Next day's log shows metricsnotcollecting report every 16 mins 2010-09-16 10:16:53,677 INFO [Thread-3] [org.hyperic.hq.common.DiagnosticsLogger@104] [org.hyperic.hq.measurement.MetricsNotComingInDiagnostic@b9f7607] Enabled metrics not reported in for 60 minutes (by platform hierarchy) ------------------------------------------------------------------------
Hide
Dharma Srinivasan added a comment -

Other related stats:

scenario:
resource listed in "metrics not collecting" report was deleted, resulted in exception in server log

stats:
at that time, the jdbc connection pool count spiked to 289, 245 (those are the values collected by hq-stats in 15 second intervals, so the spike stayed for ~30 seconds)

Hibernate_2nd_level_cache_misses spiked also during this time (chart attached)

Show
Dharma Srinivasan added a comment - Other related stats: scenario: resource listed in "metrics not collecting" report was deleted, resulted in exception in server log stats: at that time, the jdbc connection pool count spiked to 289, 245 (those are the values collected by hq-stats in 15 second intervals, so the spike stayed for ~30 seconds) Hibernate_2nd_level_cache_misses spiked also during this time (chart attached)
Hide
Idan Hod added a comment -

As part of our continuous effort to improve product quality, The Hyperic product team has decided to implement a "zero bug policy" methodology.

Following this methodology, only defects that are planned to be handled in the near future will remain open. Any other defect will be deferred, with the option to be reevaluated if the need arises, or if changes to the Hyperic road-map make such defect a candidate for a fix.

We believe this new process will help create clarity and focus in the Hyperic road-map and ultimately benefit our customer base.

This bug has been deferred as part of the new policy.

We appreciate your cooperation and continues contribution to the improvement of Hyperic.

Show
Idan Hod added a comment - As part of our continuous effort to improve product quality, The Hyperic product team has decided to implement a "zero bug policy" methodology. Following this methodology, only defects that are planned to be handled in the near future will remain open. Any other defect will be deferred, with the option to be reevaluated if the need arises, or if changes to the Hyperic road-map make such defect a candidate for a fix. We believe this new process will help create clarity and focus in the Hyperic road-map and ultimately benefit our customer base. This bug has been deferred as part of the new policy. We appreciate your cooperation and continues contribution to the improvement of Hyperic.

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved:
    Last comment:
    40 weeks, 6 days ago