Support Migration Notice: To update migrated JIRA cases click here to open a new case use www.vmware.com/go/sr | vFabric Hyperic 5.7.0 is Now Available

Hyperic HQ

Unidirectional agent ran out of memory while monitoring 1000 http service checks

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Deferred
  • Affects Version/s: 4.5.1
  • Fix Version/s: None
  • Component/s: Agent / PDK
  • Environment:
    4.5.1
  • Case Links:
    none
  • Regression:
    No

Description

Configure a unidirectional agent to monitor 1000 http service checks and HQ Tomcat server.
Update collection interval for HTTP service availability and Response Code to 1 minute (this effectively makes the agent collect ~1500 metrics/minute)

After a while (minutes or hours) the agent throws OOM error in agent.log and JSW ends up restarting agent. This keeps on repeating every few minutes. The stack in agent log with OOM varies. Following are 2 versions I have seen.

Note: This issue is not reproducible on bi-directional agent. It only happens on uni-directional agent.

java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:746)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.hyperic.hq.plugin.netservices.HTTPCollector.collect(HTTPCollector.java:320)
at org.hyperic.hq.product.Collector.run(Collector.java:563)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:619)

2010-11-30 13:26:30,497 ERROR [pool-1-thread-26] [Collector] Error running http collector: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:87)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.hyperic.hq.plugin.netservices.HTTPCollector.collect(HTTPCollector.java:320)
at org.hyperic.hq.product.Collector.run(Collector.java:563)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:619)

  1. agent.log
    30/Nov/10 1:51 PM
    1.11 MB
    Kashyap Parikh
  2. wrapper.log
    30/Nov/10 1:51 PM
    153 kB
    Kashyap Parikh

Activity

Hide
Yoav Epelman added a comment -

Bulk change to new components

Show
Yoav Epelman added a comment - Bulk change to new components
Hide
Idan Hod added a comment -

As part of our continuous effort to improve product quality, The Hyperic product team has decided to implement a "zero bug policy" methodology.

Following this methodology, only defects that are planned to be handled in the near future will remain open. Any other defect will be deferred, with the option to be reevaluated if the need arises, or if changes to the Hyperic road-map make such defect a candidate for a fix.

We believe this new process will help create clarity and focus in the Hyperic road-map and ultimately benefit our customer base.

This bug has been deferred as part of the new policy.

We appreciate your cooperation and continues contribution to the improvement of Hyperic.

Show
Idan Hod added a comment - As part of our continuous effort to improve product quality, The Hyperic product team has decided to implement a "zero bug policy" methodology. Following this methodology, only defects that are planned to be handled in the near future will remain open. Any other defect will be deferred, with the option to be reevaluated if the need arises, or if changes to the Hyperic road-map make such defect a candidate for a fix. We believe this new process will help create clarity and focus in the Hyperic road-map and ultimately benefit our customer base. This bug has been deferred as part of the new policy. We appreciate your cooperation and continues contribution to the improvement of Hyperic.

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved:
    Last comment:
    41 weeks, 2 days ago