Support Migration Notice: To update migrated JIRA cases click here to open a new case use www.vmware.com/go/sr | vFabric Hyperic 5.7.0 is Now Available

Hyperic HQ

Exceptions in AvailabilityManagerEJBImpl.sendDataToEventHandlers can cause availability to not be stored

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Deferred
  • Affects Version/s: 4.2.0
  • Fix Version/s: None
  • Component/s: Core Monitoring
  • Case Links:
    none
  • Regression:
    No

Description

If an exception occurs in AvailabilityManagerEJBImpl.sendDataToEventHandlers it would appear the transaction is rolled back and no availability data is stored. Regular metric data continues to come in as expected. This particular case was due to an alert firing on a resource that was in async delete state. I've been told this condition has been fixed (if not we need another bug here) but in any case if an exception happens here it should not cause availability to not be stored.

In this particular case, we were able to work around this condition by manually modifying the EAM_RESOURCE table setting resource_type_id for the resources in which alerts were firing. Once that was cleaned up the exception went away and availability started getting reported as expected.

Here's the stacktrace:

2010-04-09 08:28:29,125 ERROR [pool-10-thread-1] [org.jboss.ejb.plugins.LogInterceptor@308] TransactionRolledbackLocalException in method: public abstract java.util.Map org.hyperic.hq.events.shared.EventLogManagerLocal.findLastUnfixedAlertFiredEvents(java.util.List), causedBy:
java.lang.IllegalArgumentException: m0000719 PPS Filter Status does not have a Resource Type
at org.hyperic.hq.appdef.shared.AppdefEntityID.<init>(AppdefEntityID.java:140)
at org.hyperic.hq.events.server.session.EventLogDAO.createAlertFiredEvent(EventLogDAO.java:356)
at org.hyperic.hq.events.server.session.EventLogDAO.findUnfixedAlertFiredEventLogs(EventLogDAO.java:346)
at org.hyperic.hq.events.server.session.EventLogManagerEJBImpl.findLastUnfixedAlertFiredEvents(EventLogManagerEJBImpl.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:528)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:601)
at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:528)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:601)
at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
at org.jboss.ejb.Container.invoke(Container.java:960)
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
at $Proxy298.findLastUnfixedAlertFiredEvents(Unknown Source)
at org.hyperic.hq.events.server.session.RegisteredTriggerManagerEJBImpl.findLastUnfixedAlertFiredEvents(RegisteredTriggerManagerEJBImpl.java:323)
at org.hyperic.hq.events.server.session.RegisteredTriggerManagerEJBImpl.initializeTriggers(RegisteredTriggerManagerEJBImpl.java:190)
at org.hyperic.hq.events.server.session.RegisteredTriggerManagerEJBImpl.initializeTriggers(RegisteredTriggerManagerEJBImpl.java:145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:528)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:601)
at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:528)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:601)
at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
at org.jboss.ejb.Container.invoke(Container.java:960)
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
at $Proxy236.initializeTriggers(Unknown Source)
at org.hyperic.hq.events.ext.RegisteredTriggers.init(RegisteredTriggers.java:79)
at org.hyperic.hq.events.ext.RegisteredTriggers.getAndInitialize(RegisteredTriggers.java:203)
at org.hyperic.hq.events.ext.RegisteredTriggers.isTriggerInterested(RegisteredTriggers.java:241)
at org.hyperic.hq.measurement.server.session.AvailabilityManagerEJBImpl.sendDataToEventHandlers(AvailabilityManagerEJBImpl.java:1316)
at org.hyperic.hq.measurement.server.session.AvailabilityManagerEJBImpl.addData(AvailabilityManagerEJBImpl.java:773)
at org.hyperic.hq.measurement.server.session.AvailabilityManagerEJBImpl.addData(AvailabilityManagerEJBImpl.java:710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:528)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:601)
at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:528)
at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:601)
at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
at org.jboss.ejb.Container.invoke(Container.java:960)
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
at $Proxy269.addData(Unknown Source)
at org.hyperic.hq.measurement.server.session.SynchronousAvailDataInserter.insertMetrics(SynchronousAvailDataInserter.java:45)
at com.hyperic.hq.measurement.server.session.BatchAggregateDataInserter$Worker.insertData(BatchAggregateDataInserter.java:477)
at com.hyperic.hq.measurement.server.session.BatchAggregateDataInserter$Worker.run(BatchAggregateDataInserter.java:506)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

Issue Links

Activity

Hide
Kashyap Parikh added a comment -

This issue is fixed as part of HHQ-3534

Show
Kashyap Parikh added a comment - This issue is fixed as part of HHQ-3534
Hide
Ryan Morgan added a comment -

I see the 'fix' here is to add:

if (log.getResource().isInAsyncDeleteState()) { continue; }

to EventLogDAO to prevent the creation of an alert fired event for resources in async delete state. While that's OK for this particular stack trace, it's missing the actual intent of this bug. The intent is to not have this coupling such that an exception in initializing the triggers would cause availability data to not be inserted.

Show
Ryan Morgan added a comment - I see the 'fix' here is to add: if (log.getResource().isInAsyncDeleteState()) { continue; } to EventLogDAO to prevent the creation of an alert fired event for resources in async delete state. While that's OK for this particular stack trace, it's missing the actual intent of this bug. The intent is to not have this coupling such that an exception in initializing the triggers would cause availability data to not be inserted.
Hide
Yoav Epelman added a comment -

Bulk change to new components

Show
Yoav Epelman added a comment - Bulk change to new components
Hide
Idan Hod added a comment -

As part of our continuous effort to improve product quality, The Hyperic product team has decided to implement a "zero bug policy" methodology.

Following this methodology, only defects that are planned to be handled in the near future will remain open. Any other defect will be deferred, with the option to be reevaluated if the need arises, or if changes to the Hyperic road-map make such defect a candidate for a fix.

We believe this new process will help create clarity and focus in the Hyperic road-map and ultimately benefit our customer base.

This bug has been deferred as part of the new policy.

We appreciate your cooperation and continues contribution to the improvement of Hyperic.

Show
Idan Hod added a comment - As part of our continuous effort to improve product quality, The Hyperic product team has decided to implement a "zero bug policy" methodology. Following this methodology, only defects that are planned to be handled in the near future will remain open. Any other defect will be deferred, with the option to be reevaluated if the need arises, or if changes to the Hyperic road-map make such defect a candidate for a fix. We believe this new process will help create clarity and focus in the Hyperic road-map and ultimately benefit our customer base. This bug has been deferred as part of the new policy. We appreciate your cooperation and continues contribution to the improvement of Hyperic.

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved:
    Last comment:
    41 weeks, 1 day ago