History | Log In     View a printable version of the current page.  
   HOME       BROWSE PROJECT       FIND ISSUES   
    QUICK SEARCH:  
Hyperic HQ 3.2.5-EE Maintenance Release is Now Available
Issue Details (XML | Word | Printable)

Key: HHQ-1960
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Critical Critical
Assignee: Ryan Morgan
Reporter: Kashyap Parikh
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Hyperic HQ

Hibernate locking up on getting DerivedMeasurementDAO.get

Created: 11/Apr/08 08:24 AM   Updated: 14/Apr/08 04:52 PM
Component/s: Server
Affects Version/s: 3.2.2
Fix Version/s: 3.2.3

Environment: 3.2.2 #650 with mysql backend

Verify By: Kashyap Parikh
Last comment: 26 weeks, 1 day ago
Resolution Date: 11/Apr/08 10:22 AM


 Description  « Hide
When many agents are connected we see lots of server thread locked up at following location. Because of this none of the metric data from any agent it getting inserted. MySQL is idle and server load avg is above 10 with high cpu usage.

This is a hibernate bug -> http://opensource.atlassian.com/projects/hibernate/browse/HHH-2645 and a patch is readily available.
Scott and I tested the patch yesterday and it resolbing the issue we are experiencing. It requires a patched hibernate3.jar as this Hibernate did not backport the fix to 3.2.3.

"Thread-70083" daemon prio=1 tid=0x00002aab8a4b1440 nid=0x493e waiting for monitor entry [0x00002aab97c6d000..0x00002aab97c6fb80]
        at java.util.Collections$SynchronizedMap.get(Unknown Source)
        - waiting to lock <0x00002aaaf4236408> (a java.util.Collections$SynchronizedMap)
        at org.hibernate.tuple.EntityModeToTuplizerMapping.getTuplizerOrNull(EntityModeToTuplizerMapping.java:53)
        at org.hibernate.tuple.EntityModeToTuplizerMapping.getTuplizer(EntityModeToTuplizerMapping.java:66)
        at org.hibernate.tuple.entity.EntityMetamodel.getTuplizer(EntityMetamodel.java:105)
        at org.hibernate.persister.entity.AbstractEntityPersister.getTuplizer(AbstractEntityPersister.java:3191)
        at org.hibernate.persister.entity.AbstractEntityPersister.instantiate(AbstractEntityPersister.java:3604)
        at org.hibernate.impl.SessionImpl.instantiate(SessionImpl.java:1275)
        at org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:538)
        at org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:508)
        at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:357)
        at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139)
        at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195)
        at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
        at org.hibernate.impl.SessionImpl.get(SessionImpl.java:815)
        at org.hibernate.impl.SessionImpl.get(SessionImpl.java:808)
        at org.hyperic.hq.dao.HibernateDAO.get(HibernateDAO.java:72)
        at org.hyperic.hq.measurement.server.session.DerivedMeasurementDAO.get(DerivedMeasurementDAO.java:60)
        at org.hyperic.hq.measurement.server.session.DerivedMeasurementManagerEJBImpl.getMeasurement(DerivedMeasurementManagerEJBImpl.java:603)
        at sun.reflect.GeneratedMethodAccessor419.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
        at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
        at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)
        at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)
        at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:54)
        at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:416)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:468)
        at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
        at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)
        at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
        at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:416)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:468)
        at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
        at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)
        at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
        at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
        at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
        at org.jboss.ejb.Container.invoke(Container.java:873)
        at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:415)
        at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)
        at $Proxy315.getMeasurement(Unknown Source)
        at org.hyperic.hq.measurement.server.session.ReportProcessorEJBImpl.handleMeasurementReport(ReportProcessorEJBImpl.java:174)
        at sun.reflect.GeneratedMethodAccessor486.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
        at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
        at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)
        at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)
        at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:54)
        at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:416)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:468)
        at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
        at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)
        at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
        at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:416)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:468)
        at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
        at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)
        at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
        at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
        at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
        at org.jboss.ejb.Container.invoke(Container.java:873)
        at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:415)
        at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)
        at $Proxy319.handleMeasurementReport(Unknown Source)
        at org.hyperic.hq.bizapp.server.session.LatherDispatcher.cmdMeasurementSendReport(LatherDispatcher.java:491)
        at org.hyperic.hq.bizapp.server.session.LatherDispatcher._dispatch(LatherDispatcher.java:750)
        at org.hyperic.hq.bizapp.server.session.LatherDispatcher.dispatch(LatherDispatcher.java:734)
        at com.hyperic.hq.bizapp.server.session.LatherDispatcher.dispatch(LatherDispatcher.java:148)
        at com.hyperic.hq.bizapp.server.session.EELatherBossEJBImpl.dispatchWithoutTx(EELatherBossEJBImpl.java:86)
        at com.hyperic.hq.bizapp.server.session.EELatherBossEJBImpl.dispatchWithTx(EELatherBossEJBImpl.java:74)
        at com.hyperic.hq.bizapp.server.session.EELatherBossEJBImpl.dispatch(EELatherBossEJBImpl.java:59)
        at sun.reflect.GeneratedMethodAccessor480.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
        at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
        at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)
        at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)
        at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:54)
        at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:416)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:468)
        at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
        at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)
        at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:300)
        at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:416)
        at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:468)
        at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
        at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)
        at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
        at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
        at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
        at org.jboss.ejb.Container.invoke(Container.java:873)
        at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:415)
        at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)
        at $Proxy329.dispatch(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor997.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.hyperic.lather.jboss.JBossLatherServlet$EJBCaller.doInvoke(JBossLatherServlet.java:318)
        at org.hyperic.lather.jboss.JBossLatherServlet$EJBCaller.run(JBossLatherServlet.java:354)

 All   Comments   Change History      Sort Order: Ascending order - Click to sort in descending order
[ Permlink | « Hide ]
Ryan Morgan - 11/Apr/08 08:42 AM

I can upgrade us to Hibernate 3.2.6 which contains this fix. (We are running 3.2.5 currently). I have been holding off on this upgrade until 3.2.7 is released, which fixes the non-safe collections cleanup errors we are seeing in the logs on 4.0. My email to QA & Dev on 3/13:

QA,

You may see some messages that look like the following in 4.0:

2008-03-13 18:33:07,539 WARN [Thread-28] [org.hibernate.engine.loading.LoadContexts] fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@c65481<rs=org.postgresql.jdbc3.Jdbc3ResultSet@972d66>

This tends to happen during mass operations. (such as a new agent refreshing it's schedule) This is due to a bug that was introduced in Hibernate 3.2.5. It can be safely ignored and will be fixed when we patch to 3.2.7.

http://opensource.atlassian.com/projects/hibernate/browse/HHH-2936

-Ryan

[ Permlink | « Hide ]
Ryan Morgan - 11/Apr/08 08:44 AM

Actually, it appears this non-safe collections fix is in 3.2.6, I'll upgrade trunk (4.0) with this jar to see if it fixes our problems. If deemed production worthy we can evaluate a backport to HQ 3.2.

[ Permlink | « Hide ]
Ryan Morgan - 11/Apr/08 08:48 AM
Mark 3.2.3.

[ Permlink | « Hide ]
Ryan Morgan - 11/Apr/08 10:22 AM

Upgraded to Hibernate 3.2.6ga on both 3.2 and 4.0.

Fixes in 4.0.0 build #681 and 3.2.3 build #653


Powered by a free Atlassian JIRA non-profit license for Hyperic HQ. Try JIRA - bug tracking software for your team.
Atlassian JIRA the Professional Issue Tracker. (Enterprise Edition, Version: 3.12.3-#302) - Bug/feature request - Atlassian news - Contact Administrators