Support Migration Notice: To update migrated JIRA cases click here to open a new case use www.vmware.com/go/sr | vFabric Hyperic 5.7.0 is Now Available

Hyperic HQ

Removing multiple VMs from vCenter doesn't delete VMs from HQ inventory for a long period

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 4.5.1
  • Fix Version/s: 4.5.1.3, 4.x Sprint 7
  • Component/s: Agent / PDK
  • Environment:
    4.5.1
  • Case Links:
    none
  • Regression:
    No
  • Story Points:
    5
  • Tags:

Description

To reproduce:

Configure and monitor vCenter server in Hyperic
Power off and delete 5 VMs from disk in vCenter Server
It takes more then an hour to remove all 5 VMs from Hyperic inventory
Based on server.log it looks like only 1 VM gets deleted per default scan (15 minutes)

Here's snip of server.log 6 hours after deleting 325 VMs from vCenter Server. In Hyperic UI 300 VMs are still showing as unavailable. Full server.log and all agent logs are attached

2010-12-16 13:19:03,070 INFO [tomcat-http--4] [ResourceController@?] Invoking method: delete with [id:[12464]]
2010-12-16 13:49:00,046 INFO [tomcat-http--33] [ResourceController@?] Invoking method: delete with [id:[12466]]
2010-12-16 14:03:12,086 INFO [tomcat-http--16] [ResourceController@?] Invoking method: delete with [id:[11480]]
2010-12-16 14:17:31,650 INFO [tomcat-http--27] [ResourceController@?] Invoking method: delete with [id:[12467]]
2010-12-16 14:29:28,185 INFO [tomcat-http--19] [ResourceController@?] Invoking method: delete with [id:[12468]]
2010-12-16 14:47:44,239 INFO [tomcat-http--3] [ResourceController@?] Invoking method: delete with [id:[12465]]
2010-12-16 14:58:37,423 INFO [tomcat-http--24] [ResourceController@?] Invoking method: delete with [id:[11459]]

Activity

Hide
Kashyap Parikh added a comment -

As a result of these VMs not getting deleted, agent keeps trying to collect metrics for these VMs and fills up agent.log with following type messages:

agent.log.3:2010-12-16 18:30:20,546 ERROR [pool-1-thread-62] [org.hyperic.hq.plugin.vsphere.VSphereHostCollector@150] Error setting availability for HQ-4.5-Agent-VM-E-100: VirtualMachine/HQ-4.5-Agent-VM-E-100: not found
agent.log.3:2010-12-16 18:30:54,145 ERROR [pool-1-thread-19] [org.hyperic.hq.plugin.vsphere.VSphereHostCollector@150] Error setting availability for HQ-4.5-Agent-VM-E-100: VirtualMachine/HQ-4.5-Agent-VM-E-100: not found
agent.log.3:2010-12-16 18:32:23,642 ERROR [pool-1-thread-73] [org.hyperic.hq.plugin.vsphere.VSphereHostCollector@150] Error setting availability for HQ-4.5-Agent-VM-E-100: VirtualMachine/HQ-4.5-Agent-VM-E-100: not found

Show
Kashyap Parikh added a comment - As a result of these VMs not getting deleted, agent keeps trying to collect metrics for these VMs and fills up agent.log with following type messages: agent.log.3:2010-12-16 18:30:20,546 ERROR [pool-1-thread-62] [org.hyperic.hq.plugin.vsphere.VSphereHostCollector@150] Error setting availability for HQ-4.5-Agent-VM-E-100: VirtualMachine/HQ-4.5-Agent-VM-E-100: not found agent.log.3:2010-12-16 18:30:54,145 ERROR [pool-1-thread-19] [org.hyperic.hq.plugin.vsphere.VSphereHostCollector@150] Error setting availability for HQ-4.5-Agent-VM-E-100: VirtualMachine/HQ-4.5-Agent-VM-E-100: not found agent.log.3:2010-12-16 18:32:23,642 ERROR [pool-1-thread-73] [org.hyperic.hq.plugin.vsphere.VSphereHostCollector@150] Error setting availability for HQ-4.5-Agent-VM-E-100: VirtualMachine/HQ-4.5-Agent-VM-E-100: not found
Hide
Ryan Morgan added a comment -

Moving issues to 4.5.1.2.

Show
Ryan Morgan added a comment - Moving issues to 4.5.1.2.
Hide
Todd Rader added a comment -

It would be very helpful to have DEBUG for VSphereHostCollector turned on here.

Show
Todd Rader added a comment - It would be very helpful to have DEBUG for VSphereHostCollector turned on here.
Hide
Patrick Nguyen added a comment -

this command on the agent.log files:

grep "Removing from HQ inventory" .log > patrick.txt

returned results very similar to the server.log log statements in the description of the bug. the delete statements in the agent.log and server.log are within a few seconds of each other (and very reasonable considering the agent-to-server communication latency). so, when a resource is marked for deletion by the agent, it is deleted by the server very quickly.

however, will need to add more debugging to determine why vms/hosts are not being marked for deletion during the autodiscovery (default and runtime) process more frequently as expected.

partial results listed below:

agent.log.4:2010-12-16 13:19:01,544 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-1 {4228ed3c-beb5-824a-bd68-2699372e244f}) no longer exists in vCenter. Removing from HQ inventory.

agent.log.4:2010-12-16 13:48:58,756 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-0 {4228a1e6-e4e3-a985-2010-1a2c693394bb}) no longer exists in vCenter. Removing from HQ inventory.

agent.log.4:2010-12-16 14:03:10,890 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (NapaServer {4228e1a3-3845-6307-6ce0-3f7a46495af2}) no longer exists in vCenter. Removing from HQ inventory.

agent.log.4:2010-12-16 14:17:30,587 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-0 {42280b98-2e1a-a5e9-45bb-61db712580c1}) no longer exists in vCenter. Removing from HQ inventory.

agent.log.4:2010-12-16 14:29:27,212 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-0 {4228bd68-b239-a7a2-1ddd-1121c5fbdc0a}) no longer exists in vCenter. Removing from HQ inventory.

agent.log.4:2010-12-16 14:47:43,407 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-2 {422874cb-9a4b-20e5-1aa6-90b6acbe7715}) no longer exists in vCenter. Removing from HQ inventory.

agent.log.4:2010-12-16 14:58:36,668 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (NapaServer {4228fcc4-a5d1-314f-479b-f5428690caa9}) no longer exists in vCenter. Removing from HQ inventory.

Show
Patrick Nguyen added a comment - this command on the agent.log files: grep "Removing from HQ inventory" .log > patrick.txt returned results very similar to the server.log log statements in the description of the bug. the delete statements in the agent.log and server.log are within a few seconds of each other (and very reasonable considering the agent-to-server communication latency). so, when a resource is marked for deletion by the agent, it is deleted by the server very quickly. however, will need to add more debugging to determine why vms/hosts are not being marked for deletion during the autodiscovery (default and runtime) process more frequently as expected. partial results listed below: agent.log.4:2010-12-16 13:19:01,544 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-1 {4228ed3c-beb5-824a-bd68-2699372e244f}) no longer exists in vCenter. Removing from HQ inventory. agent.log.4:2010-12-16 13:48:58,756 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-0 {4228a1e6-e4e3-a985-2010-1a2c693394bb}) no longer exists in vCenter. Removing from HQ inventory. agent.log.4:2010-12-16 14:03:10,890 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (NapaServer {4228e1a3-3845-6307-6ce0-3f7a46495af2}) no longer exists in vCenter. Removing from HQ inventory. agent.log.4:2010-12-16 14:17:30,587 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-0 {42280b98-2e1a-a5e9-45bb-61db712580c1}) no longer exists in vCenter. Removing from HQ inventory. agent.log.4:2010-12-16 14:29:27,212 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-0 {4228bd68-b239-a7a2-1ddd-1121c5fbdc0a}) no longer exists in vCenter. Removing from HQ inventory. agent.log.4:2010-12-16 14:47:43,407 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (vm-2 {422874cb-9a4b-20e5-1aa6-90b6acbe7715}) no longer exists in vCenter. Removing from HQ inventory. agent.log.4:2010-12-16 14:58:36,668 DEBUG [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@748] Managed entity (NapaServer {4228fcc4-a5d1-314f-479b-f5428690caa9}) no longer exists in vCenter. Removing from HQ inventory.
Hide
Patrick Nguyen added a comment -

analyzing the agent.logs more, it looks like this exception might be one of the root causes:

2010-12-16 13:34:01,808 ERROR [Thread-2] [org.hyperic.hq.plugin.vsphere.VCenterDetector@148] Could not discover platforms during the default scan: VirtualMachine/4228a1e6-e4e3-a985-2010-1a2c693394bb: java.lang.RuntimeException: com.vmware.vim25.ManagedObjectNotFound
org.hyperic.hq.product.PluginException: VirtualMachine/4228a1e6-e4e3-a985-2010-1a2c693394bb: java.lang.RuntimeException: com.vmware.vim25.ManagedObjectNotFound
at org.hyperic.hq.plugin.vsphere.VSphereUtil.findByUuid(VSphereUtil.java:178)
at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.removeVM(VMAndHostVCenterPlatformDetector.java:733)
at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.removePlatformsFromInventory(VMAndHostVCenterPlatformDetector.java:626)
at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.discoverPlatforms(VMAndHostVCenterPlatformDetector.java:483)
at org.hyperic.hq.plugin.vsphere.VCenterDetector.discoverPlatforms(VCenterDetector.java:119)
at org.hyperic.hq.plugin.vsphere.VCenterDetector.discoverPlatforms(VCenterDetector.java:144)
at org.hyperic.hq.plugin.vsphere.VCenterDetector.getServerResources(VCenterDetector.java:164)
at org.hyperic.hq.autoinventory.scanimpl.NullScan.scan(NullScan.java:83)
at org.hyperic.hq.autoinventory.Scanner.start(Scanner.java:198)
at org.hyperic.hq.autoinventory.ScanManager.mainRunLoop(ScanManager.java:141)
at org.hyperic.hq.autoinventory.ScanManager.access$000(ScanManager.java:41)
at org.hyperic.hq.autoinventory.ScanManager$1.run(ScanManager.java:107)
Caused by: java.lang.RuntimeException: com.vmware.vim25.ManagedObjectNotFound
at com.vmware.vim25.mo.ManagedObject.retrieveObjectProperties(ManagedObject.java:158)
at com.vmware.vim25.mo.ManagedObject.getCurrentProperty(ManagedObject.java:179)
at com.vmware.vim25.mo.VirtualMachine.getConfig(VirtualMachine.java:55)
at org.hyperic.hq.plugin.vsphere.VSphereUtil.getUuid(VSphereUtil.java:262)
at org.hyperic.hq.plugin.vsphere.VSphereUtil.findByUuid(VSphereUtil.java:158)
... 11 more

Show
Patrick Nguyen added a comment - analyzing the agent.logs more, it looks like this exception might be one of the root causes: 2010-12-16 13:34:01,808 ERROR [Thread-2] [org.hyperic.hq.plugin.vsphere.VCenterDetector@148] Could not discover platforms during the default scan: VirtualMachine/4228a1e6-e4e3-a985-2010-1a2c693394bb: java.lang.RuntimeException: com.vmware.vim25.ManagedObjectNotFound org.hyperic.hq.product.PluginException: VirtualMachine/4228a1e6-e4e3-a985-2010-1a2c693394bb: java.lang.RuntimeException: com.vmware.vim25.ManagedObjectNotFound at org.hyperic.hq.plugin.vsphere.VSphereUtil.findByUuid(VSphereUtil.java:178) at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.removeVM(VMAndHostVCenterPlatformDetector.java:733) at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.removePlatformsFromInventory(VMAndHostVCenterPlatformDetector.java:626) at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.discoverPlatforms(VMAndHostVCenterPlatformDetector.java:483) at org.hyperic.hq.plugin.vsphere.VCenterDetector.discoverPlatforms(VCenterDetector.java:119) at org.hyperic.hq.plugin.vsphere.VCenterDetector.discoverPlatforms(VCenterDetector.java:144) at org.hyperic.hq.plugin.vsphere.VCenterDetector.getServerResources(VCenterDetector.java:164) at org.hyperic.hq.autoinventory.scanimpl.NullScan.scan(NullScan.java:83) at org.hyperic.hq.autoinventory.Scanner.start(Scanner.java:198) at org.hyperic.hq.autoinventory.ScanManager.mainRunLoop(ScanManager.java:141) at org.hyperic.hq.autoinventory.ScanManager.access$000(ScanManager.java:41) at org.hyperic.hq.autoinventory.ScanManager$1.run(ScanManager.java:107) Caused by: java.lang.RuntimeException: com.vmware.vim25.ManagedObjectNotFound at com.vmware.vim25.mo.ManagedObject.retrieveObjectProperties(ManagedObject.java:158) at com.vmware.vim25.mo.ManagedObject.getCurrentProperty(ManagedObject.java:179) at com.vmware.vim25.mo.VirtualMachine.getConfig(VirtualMachine.java:55) at org.hyperic.hq.plugin.vsphere.VSphereUtil.getUuid(VSphereUtil.java:262) at org.hyperic.hq.plugin.vsphere.VSphereUtil.findByUuid(VSphereUtil.java:158) ... 11 more
Hide
Patrick Nguyen added a comment -

there's another NPE that needs to be fixed:

2010-12-16 14:04:50,190 ERROR [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@424]
java.lang.NullPointerException
at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.discoverVM(VMAndHostVCenterPlatformDetector.java:182)
at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.discoverHosts(VMAndHostVCenterPlatformDetector.java:413)
at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.discoverPlatforms(VMAndHostVCenterPlatformDetector.java:451)
at org.hyperic.hq.plugin.vsphere.VCenterDetector.discoverPlatforms(VCenterDetector.java:119)
at org.hyperic.hq.plugin.vsphere.VCenterDetector.discoverPlatforms(VCenterDetector.java:144)
at org.hyperic.hq.plugin.vsphere.VCenterDetector.getServerResources(VCenterDetector.java:164)
at org.hyperic.hq.autoinventory.scanimpl.NullScan.scan(NullScan.java:83)
at org.hyperic.hq.autoinventory.Scanner.start(Scanner.java:198)
at org.hyperic.hq.autoinventory.ScanManager.mainRunLoop(ScanManager.java:141)
at org.hyperic.hq.autoinventory.ScanManager.access$000(ScanManager.java:41)
at org.hyperic.hq.autoinventory.ScanManager$1.run(ScanManager.java:107)

Show
Patrick Nguyen added a comment - there's another NPE that needs to be fixed: 2010-12-16 14:04:50,190 ERROR [Thread-2] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@424] java.lang.NullPointerException at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.discoverVM(VMAndHostVCenterPlatformDetector.java:182) at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.discoverHosts(VMAndHostVCenterPlatformDetector.java:413) at org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector.discoverPlatforms(VMAndHostVCenterPlatformDetector.java:451) at org.hyperic.hq.plugin.vsphere.VCenterDetector.discoverPlatforms(VCenterDetector.java:119) at org.hyperic.hq.plugin.vsphere.VCenterDetector.discoverPlatforms(VCenterDetector.java:144) at org.hyperic.hq.plugin.vsphere.VCenterDetector.getServerResources(VCenterDetector.java:164) at org.hyperic.hq.autoinventory.scanimpl.NullScan.scan(NullScan.java:83) at org.hyperic.hq.autoinventory.Scanner.start(Scanner.java:198) at org.hyperic.hq.autoinventory.ScanManager.mainRunLoop(ScanManager.java:141) at org.hyperic.hq.autoinventory.ScanManager.access$000(ScanManager.java:41) at org.hyperic.hq.autoinventory.ScanManager$1.run(ScanManager.java:107)
Hide
Kashyap Parikh added a comment -

Regarding ManagedObjectNotFound exeption I thought it was a side effect of VM being deleted from VCS and agent still trying to collect data for it.
NPE issue is logged here HHQ-4481

Show
Kashyap Parikh added a comment - Regarding ManagedObjectNotFound exeption I thought it was a side effect of VM being deleted from VCS and agent still trying to collect data for it. NPE issue is logged here HHQ-4481
Hide
Patrick Nguyen added a comment -

the ManagedObjectNotFound appears in 2 scenarios, the one that you described (agent still trying to collect data for deleted resources) and the one i that mentioned above during the default scan ("Could not discover platforms during the default scan...ManagedObjectNotFound..."). this second scenario is causing the autodiscovery and delete process to stop prematurely.

Show
Patrick Nguyen added a comment - the ManagedObjectNotFound appears in 2 scenarios, the one that you described (agent still trying to collect data for deleted resources) and the one i that mentioned above during the default scan ("Could not discover platforms during the default scan...ManagedObjectNotFound..."). this second scenario is causing the autodiscovery and delete process to stop prematurely.
Hide
Patrick Nguyen added a comment -

FIX 1: ManagedObjectNotFound can be thrown while getting the UUID for a deleted resource, so handle this use case.

Show
Patrick Nguyen added a comment - FIX 1: ManagedObjectNotFound can be thrown while getting the UUID for a deleted resource, so handle this use case.
Hide
Kashyap Parikh added a comment -

ManagedObjectNotFound exception is no longer thrown during auto-discovery but I still see the issue of VMs not getting deleted for long time. In this test run I deleted 5 VMs in VCS. First one was deleted from HQ roughly 5 minutes after but others were deleted anywhere from 30 minutes to hour. See time stamps for individual VM delete messaged from agent log below. Full agent logs are attached (agent-logs-trial-2.zip) with DEBUG log on VSphereHostCollector and VMAndHostVCenterPlatformDetector.

bash-3.00$ grep 'Removing from HQ inventory' agent.log.1 | grep B-
2011-02-02 17:35:03,684 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@782] Managed entity (HQ-4.5-Agent-VM-B-0 {4228e48c-dae3-e4e0-e5e2-87575e44cc6c}) no longer exists in vCenter. Removing from HQ inventory.
2011-02-02 17:50:15,437 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@782] Managed entity (HQ-4.5-Agent-VM-B-1 {42281ecf-a05b-6ab3-6eb4-4f3fe7e57f25}) no longer exists in vCenter. Removing from HQ inventory.
2011-02-02 18:05:21,375 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@782] Managed entity (HQ-4.5-Agent-VM-B-2 {42281928-d094-c7f3-de23-66afd80c455d}) no longer exists in vCenter. Removing from HQ inventory.
2011-02-02 18:20:11,775 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@782] Managed entity (HQ-4.5-Agent-VM-B-3 {4228ae31-e3b8-adbc-d726-7748a13e838f}) no longer exists in vCenter. Removing from HQ inventory.

Show
Kashyap Parikh added a comment - ManagedObjectNotFound exception is no longer thrown during auto-discovery but I still see the issue of VMs not getting deleted for long time. In this test run I deleted 5 VMs in VCS. First one was deleted from HQ roughly 5 minutes after but others were deleted anywhere from 30 minutes to hour. See time stamps for individual VM delete messaged from agent log below. Full agent logs are attached (agent-logs-trial-2.zip) with DEBUG log on VSphereHostCollector and VMAndHostVCenterPlatformDetector. bash-3.00$ grep 'Removing from HQ inventory' agent.log.1 | grep B- 2011-02-02 17:35:03,684 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@782] Managed entity (HQ-4.5-Agent-VM-B-0 {4228e48c-dae3-e4e0-e5e2-87575e44cc6c}) no longer exists in vCenter. Removing from HQ inventory. 2011-02-02 17:50:15,437 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@782] Managed entity (HQ-4.5-Agent-VM-B-1 {42281ecf-a05b-6ab3-6eb4-4f3fe7e57f25}) no longer exists in vCenter. Removing from HQ inventory. 2011-02-02 18:05:21,375 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@782] Managed entity (HQ-4.5-Agent-VM-B-2 {42281928-d094-c7f3-de23-66afd80c455d}) no longer exists in vCenter. Removing from HQ inventory. 2011-02-02 18:20:11,775 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@782] Managed entity (HQ-4.5-Agent-VM-B-3 {4228ae31-e3b8-adbc-d726-7748a13e838f}) no longer exists in vCenter. Removing from HQ inventory.
Hide
Patrick Nguyen added a comment -

as a final validation check before deleting a vm or host, we verify to see if the resource still exists in vCenter. originally, this validation check went directly to vCenter. however, we later added caching to improve performance of general lookups. unfortunately, because of this caching, a deleted resource will remain in the cache until it has expired (10 minutes). this means resources are not deleted as timely as desired.

agent.log:2011-02-02 17:50:20,675 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11411, name=HQ-4.5-Agent-VM-B-2 {42281928-d094-c7f3-de23-66afd80c455d}] exists in vCenter. Not removing from HQ.

agent.log:2011-02-02 17:50:20,677 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11412, name=HQ-4.5-Agent-VM-B-3 {4228ae31-e3b8-adbc-d726-7748a13e838f}] exists in vCenter. Not removing from HQ.
agent.log:2011-02-02 17:50:20,679 DEBUG [Thread-3]

[org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11413, name=HQ-4.5-Agent-VM-B-4 {42282fb8-9c75-48ae-dab9-bd6db0d0690b}] exists in vCenter. Not removing from HQ.
agent.log:2011-02-02 17:50:20,680 DEBUG [Thread-3]

[org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11414, name=HQ-4.5-Agent-VM-B-5 {4228abb0-08df-b397-2a86-cc90de9630d5}] exists in vCenter. Not removing from HQ.
agent.log:2011-02-02 17:50:20,700 DEBUG [Thread-3]

[org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11099, name=vm-2 {422805df-4511-d597-f014-542facae9aa7}] exists in vCenter. Not removing from HQ.
agent.log:2011-02-02 18:05:26,789 DEBUG [Thread-3]

[org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11412, name=HQ-4.5-Agent-VM-B-3 {4228ae31-e3b8-adbc-d726-7748a13e838f}]
exists in vCenter. Not removing from HQ.
agent.log:2011-02-02 18:05:26,792 DEBUG [Thread-3]

[org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11413, name=HQ-4.5-Agent-VM-B-4 {42282fb8-9c75-48ae-dab9-bd6db0d0690b}] exists in vCenter. Not removing from HQ.
agent.log:2011-02-02 18:05:26,794 DEBUG [Thread-3]

[org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11414, name=HQ-4.5-Agent-VM-B-5 {4228abb0-08df-b397-2a86-cc90de9630d5}] exists in vCenter. Not removing from HQ.
agent.log:2011-02-02 18:05:26,817 DEBUG [Thread-3]

[org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11099, name=vm-2 {422805df-4511-d597-f014-542facae9aa7}] exists in vCenter. Not removing from HQ.

Show
Patrick Nguyen added a comment - as a final validation check before deleting a vm or host, we verify to see if the resource still exists in vCenter. originally, this validation check went directly to vCenter. however, we later added caching to improve performance of general lookups. unfortunately, because of this caching, a deleted resource will remain in the cache until it has expired (10 minutes). this means resources are not deleted as timely as desired. agent.log:2011-02-02 17:50:20,675 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11411, name=HQ-4.5-Agent-VM-B-2 {42281928-d094-c7f3-de23-66afd80c455d}] exists in vCenter. Not removing from HQ. agent.log:2011-02-02 17:50:20,677 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11412, name=HQ-4.5-Agent-VM-B-3 {4228ae31-e3b8-adbc-d726-7748a13e838f}] exists in vCenter. Not removing from HQ. agent.log:2011-02-02 17:50:20,679 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11413, name=HQ-4.5-Agent-VM-B-4 {42282fb8-9c75-48ae-dab9-bd6db0d0690b}] exists in vCenter. Not removing from HQ. agent.log:2011-02-02 17:50:20,680 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11414, name=HQ-4.5-Agent-VM-B-5 {4228abb0-08df-b397-2a86-cc90de9630d5}] exists in vCenter. Not removing from HQ. agent.log:2011-02-02 17:50:20,700 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11099, name=vm-2 {422805df-4511-d597-f014-542facae9aa7}] exists in vCenter. Not removing from HQ. agent.log:2011-02-02 18:05:26,789 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11412, name=HQ-4.5-Agent-VM-B-3 {4228ae31-e3b8-adbc-d726-7748a13e838f}] exists in vCenter. Not removing from HQ. agent.log:2011-02-02 18:05:26,792 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11413, name=HQ-4.5-Agent-VM-B-4 {42282fb8-9c75-48ae-dab9-bd6db0d0690b}] exists in vCenter. Not removing from HQ. agent.log:2011-02-02 18:05:26,794 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11414, name=HQ-4.5-Agent-VM-B-5 {4228abb0-08df-b397-2a86-cc90de9630d5}] exists in vCenter. Not removing from HQ. agent.log:2011-02-02 18:05:26,817 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@765] VMware vSphere VM[id=11099, name=vm-2 {422805df-4511-d597-f014-542facae9aa7}] exists in vCenter. Not removing from HQ.
Hide
Patrick Nguyen added a comment -

there are a couple of options:

1) remove the final validation check when deleting vms/hosts

2) update the final validation check to bypass the cache and go directly to vCenter to verify inventory.

Show
Patrick Nguyen added a comment - there are a couple of options: 1) remove the final validation check when deleting vms/hosts 2) update the final validation check to bypass the cache and go directly to vCenter to verify inventory.
Hide
Patrick Nguyen added a comment -

FIX: Before deleting, perform final validation check by bypassing the cache and going directly to vCenter to verify inventory.

Show
Patrick Nguyen added a comment - FIX: Before deleting, perform final validation check by bypassing the cache and going directly to vCenter to verify inventory.
Hide
Patrick Nguyen added a comment -

Backport to 4.5.1.3

Show
Patrick Nguyen added a comment - Backport to 4.5.1.3
Hide
Kashyap Parikh added a comment -

Looks much better.

5 VM deleted at 18:18 were removed from Hyperic inventory at 18:25

2011-02-18 18:25:17,366 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12307, name=HQ-4.5-Agent-VM-B-6 {4228aecb-df82-25f1-7a3c-e365a1d76615}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:25:22,252 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12311, name=HQ-4.5-Agent-VM-B-10 {422810c9-b849-ec2c-c114-372d5b7cdaaa}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:25:27,175 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12309, name=HQ-4.5-Agent-VM-B-8 {4228425c-3e8f-ce1e-f7af-13a3d0a18e1a}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:25:33,756 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12308, name=HQ-4.5-Agent-VM-B-7 {422816ca-b31f-c846-c3ef-a6e8dda16706}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:25:37,093 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12310, name=HQ-4.5-Agent-VM-B-9 {42284e04-6ba5-b7d3-5269-12c30576000b}] no longer exists in vCenter. Removing from HQ inventory.

Similar result with 10 VMs, in that case time difference between VM delete from VC and HQ was 14 minutes

2011-02-18 18:54:56,204 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12313, name=HQ-4.5-Agent-VM-B-12 {422812b7-8954-b58c-a327-10baf36ac578}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:55:01,469 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12320, name=HQ-4.5-Agent-VM-B-19 {42283d43-2cef-934f-3ce3-c850a324816f}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:55:06,093 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12321, name=HQ-4.5-Agent-VM-B-20 {42289ce2-7618-c883-c5e3-f501e0e9cac0}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:55:11,126 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12315, name=HQ-4.5-Agent-VM-B-14 {422899a5-f890-5975-d281-b59627edf885}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:55:16,586 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12318, name=HQ-4.5-Agent-VM-B-17 {4228a004-51d1-1396-eef3-ad194bb0b2d9}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:55:21,274 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12316, name=HQ-4.5-Agent-VM-B-15 {42282df7-0d6a-a477-83e5-e6575a40f405}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:55:24,525 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12312, name=HQ-4.5-Agent-VM-B-11 {42289fec-a505-9b83-1820-cd4b7052eb4b}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:55:27,752 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12314, name=HQ-4.5-Agent-VM-B-13 {4228ef8f-38b7-fefa-682a-36844c1e2fe8}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:55:30,955 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12317, name=HQ-4.5-Agent-VM-B-16 {4228dbc2-4ab3-f898-e3e8-19e45770751f}] no longer exists in vCenter. Removing from HQ inventory.
2011-02-18 18:55:34,196 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12319, name=HQ-4.5-Agent-VM-B-18 {422863bc-4610-7169-3101-da339dce30d9}] no longer exists in vCenter. Removing from HQ inventory.

Show
Kashyap Parikh added a comment - Looks much better. 5 VM deleted at 18:18 were removed from Hyperic inventory at 18:25 2011-02-18 18:25:17,366 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12307, name=HQ-4.5-Agent-VM-B-6 {4228aecb-df82-25f1-7a3c-e365a1d76615}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:25:22,252 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12311, name=HQ-4.5-Agent-VM-B-10 {422810c9-b849-ec2c-c114-372d5b7cdaaa}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:25:27,175 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12309, name=HQ-4.5-Agent-VM-B-8 {4228425c-3e8f-ce1e-f7af-13a3d0a18e1a}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:25:33,756 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12308, name=HQ-4.5-Agent-VM-B-7 {422816ca-b31f-c846-c3ef-a6e8dda16706}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:25:37,093 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12310, name=HQ-4.5-Agent-VM-B-9 {42284e04-6ba5-b7d3-5269-12c30576000b}] no longer exists in vCenter. Removing from HQ inventory. Similar result with 10 VMs, in that case time difference between VM delete from VC and HQ was 14 minutes 2011-02-18 18:54:56,204 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12313, name=HQ-4.5-Agent-VM-B-12 {422812b7-8954-b58c-a327-10baf36ac578}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:55:01,469 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12320, name=HQ-4.5-Agent-VM-B-19 {42283d43-2cef-934f-3ce3-c850a324816f}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:55:06,093 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12321, name=HQ-4.5-Agent-VM-B-20 {42289ce2-7618-c883-c5e3-f501e0e9cac0}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:55:11,126 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12315, name=HQ-4.5-Agent-VM-B-14 {422899a5-f890-5975-d281-b59627edf885}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:55:16,586 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12318, name=HQ-4.5-Agent-VM-B-17 {4228a004-51d1-1396-eef3-ad194bb0b2d9}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:55:21,274 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12316, name=HQ-4.5-Agent-VM-B-15 {42282df7-0d6a-a477-83e5-e6575a40f405}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:55:24,525 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12312, name=HQ-4.5-Agent-VM-B-11 {42289fec-a505-9b83-1820-cd4b7052eb4b}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:55:27,752 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12314, name=HQ-4.5-Agent-VM-B-13 {4228ef8f-38b7-fefa-682a-36844c1e2fe8}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:55:30,955 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12317, name=HQ-4.5-Agent-VM-B-16 {4228dbc2-4ab3-f898-e3e8-19e45770751f}] no longer exists in vCenter. Removing from HQ inventory. 2011-02-18 18:55:34,196 DEBUG [Thread-3] [org.hyperic.hq.plugin.vsphere.VMAndHostVCenterPlatformDetector@788] Resource [id=12319, name=HQ-4.5-Agent-VM-B-18 {422863bc-4610-7169-3101-da339dce30d9}] no longer exists in vCenter. Removing from HQ inventory.

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved:
    Last comment:
    3 years, 9 weeks, 6 days ago