Support Migration Notice: To update migrated JIRA cases click here to open a new case use www.vmware.com/go/sr | vFabric Hyperic 5.7.0 is Now Available

Hyperic HQ

ScheduleThread class can incorrectly end up creating many more executors on startup

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 4.6.5
  • Component/s: Deprecated: Agent:Core
  • Case Links:
    none
  • Regression:
    No
  • Tags:

Description

When using the automatic Service type creator for spring applications, it is possible that on agent startup, each service type will create its own executor in ScheduleThread and service type should not have their own executor. This is due to the schedule thread starting before the plugins that are using the spring auto service type creator) are initialized.

After some investigation, it appears that the best resolution is to throw out the unknown plugin types. This would only ignore the scheduling of the plugins on restart of the agent. So the metrics will be correctly scheduled on the next scheduling cycle, once all the plugins are fully initialized.

Activity

Hide
Jason Konicki added a comment -

Modification: If the plugin is not found for a template, it will ignore it, since it is likely that it is waiting for its associated plugin to be fully initialized. The measurement will be scheduled during the next interval.

Show
Jason Konicki added a comment - Modification: If the plugin is not found for a template, it will ignore it, since it is likely that it is waiting for its associated plugin to be fully initialized. The measurement will be scheduled during the next interval.
Hide
Zvika Messing added a comment - - edited

Hi Jason.

How can I verify this change? what exactly is a template without a plugin?
can you add testing recommendations to this?

thanks

Show
Zvika Messing added a comment - - edited Hi Jason. How can I verify this change? what exactly is a template without a plugin? can you add testing recommendations to this? thanks
Hide
Jason Konicki added a comment -

Zvika,

Quick summary of issue:
Hyperic is able to create dynamic services via the mbeans of a Spring Application with instrumentation. The problem encountered here was that, on restart, the ScheduleThread tries to start scheduling metrics, before the dynamic services are loaded.

Originally, this caused the ScheduleThread to create a new executor thread for each service type (creating unnecessary and never used threads) that it considered "not found" (assuming it was a proxied plugin).
Now we no longer create the executor in this case until the service is found, which it will be once the dynamic service is loaded.

How to test:
The work is internal, so it might be difficult to verify. The way I tested was the following:

  • Created a tc Server runtime instance
  • Discovered the instance in Hyperic
  • Deployed a spring instrumented war file to the instance from Hyperic tc Server plugin.
    • This will create many dynamic services.
  • After all the dynamic services are created, I would restart the server.
  • Before the change I would see the creation of a large amount of Executors in the log file, now you will no longer see those created.
  • Once everything is fully initialized the measurements should be scheduled as expected and the services should show they are available.

If you need a spring instrumented war file to deploy, I can see about getting you one.

Show
Jason Konicki added a comment - Zvika, Quick summary of issue: Hyperic is able to create dynamic services via the mbeans of a Spring Application with instrumentation. The problem encountered here was that, on restart, the ScheduleThread tries to start scheduling metrics, before the dynamic services are loaded. Originally, this caused the ScheduleThread to create a new executor thread for each service type (creating unnecessary and never used threads) that it considered "not found" (assuming it was a proxied plugin). Now we no longer create the executor in this case until the service is found, which it will be once the dynamic service is loaded. How to test: The work is internal, so it might be difficult to verify. The way I tested was the following:
  • Created a tc Server runtime instance
  • Discovered the instance in Hyperic
  • Deployed a spring instrumented war file to the instance from Hyperic tc Server plugin.
    • This will create many dynamic services.
  • After all the dynamic services are created, I would restart the server.
  • Before the change I would see the creation of a large amount of Executors in the log file, now you will no longer see those created.
  • Once everything is fully initialized the measurements should be scheduled as expected and the services should show they are available.
If you need a spring instrumented war file to deploy, I can see about getting you one.
Hide
Zvika Messing added a comment -

hi Jason. thank you for the detailed answer.

one question about the test sceario - does it neccesery to deploy the war using the the tc server plugin and not manually deploy it on the tcserver instance? (hot deployment of the war to the tcserver webapps)

thanks again

Show
Zvika Messing added a comment - hi Jason. thank you for the detailed answer. one question about the test sceario - does it neccesery to deploy the war using the the tc server plugin and not manually deploy it on the tcserver instance? (hot deployment of the war to the tcserver webapps) thanks again
Hide
Jason Konicki added a comment -

Yes and no.

If you drop the war into the webapps directory manually, it will load it correctly, but it will not kick off the service discovery.

When you deploy through the tc Server plugin it will check off a scan.

What you can do is, restart the agent after the manual deployment, that will kick off a scan.

Show
Jason Konicki added a comment - Yes and no. If you drop the war into the webapps directory manually, it will load it correctly, but it will not kick off the service discovery. When you deploy through the tc Server plugin it will check off a scan. What you can do is, restart the agent after the manual deployment, that will kick off a scan.
Hide
Zvika Messing added a comment -

verified in hyperic 4.6.5 RC2 according to jason testing recommendations. saw that executors are not created (in the log file) and that the tcserver services are collecting metrics as expected.

Show
Zvika Messing added a comment - verified in hyperic 4.6.5 RC2 according to jason testing recommendations. saw that executors are not created (in the log file) and that the tcserver services are collecting metrics as expected.

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved:
    Last comment:
    2 years, 8 weeks ago