{"id":470,"date":"2023-04-12T20:09:39","date_gmt":"2023-04-12T20:09:39","guid":{"rendered":"https:\/\/systemsmedic.net\/?p=470"},"modified":"2024-10-30T16:52:03","modified_gmt":"2024-10-30T16:52:03","slug":"vcenter-6-5-stopped-working-after-esxi-upgraded-6-5-to-7-0-mistakenly-without-doing-vcenter-upgrade","status":"publish","type":"post","link":"https:\/\/systemsmedic.net\/index.php\/2023\/04\/12\/vcenter-6-5-stopped-working-after-esxi-upgraded-6-5-to-7-0-mistakenly-without-doing-vcenter-upgrade\/","title":{"rendered":"vCenter 6.5 stopped working after ESXi upgraded 6.5 to 7.0 mistakenly without doing vCenter upgrade."},"content":{"rendered":"\n

vCenter 6.5.0 17999427<\/code><\/p>\n\n\n\n

ESXi version 7.0.3 build-20842708<\/code><\/p>\n\n\n\n

I was able to troubleshoot interesting issue today. Have you ever upgrade the ESXi to higher version before the vCenter. This is interesting why ESXi has been upgraded to version 7.0 U3 while vCenter is running with 6.5. There were two hosts in this cluster one was 6.5 where the vCenter is living. <\/p>\n\n\n\n

First start the troubleshooting from the vCenter. access the vCenter via SSH and check the service status. vCenter service is stopped. <\/p>\n\n\n\n

root@vcenter [ ~ ]# service-control --status --all\nRunning:<\/strong>\n applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui\n\nStopped:<\/strong>\n vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-vcha vmware-vpxd<\/strong>\n<\/code><\/pre>\n\n\n\n
First and foremost tried to start it. <\/p>\n\n\n\n
root@vcenter [ ~ ]# service-control --start vmware-vpxd<\/strong>\nPerform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_c                                                                                                             oreossvcs=False, include_leafossvcs=False\n2023-04-12T19:14:58.710Z   Service vpxd state STOPPED\nError executing start on service vpxd. Details {\n    \"resolution\": null,\n    \"detail\": [\n        {\n            \"args\": [\n                \"vpxd\"\n            ],\n            \"id\": \"install.ciscommon.service.failstart\",\n            \"localized\": \"An error occurred while starting service 'vpxd'\",\n            \"translatable\": \"An error occurred while starting service '%(0)s'\"\n        }\n    ],\n    \"componentKey\": null,\n    \"problemId\": null\n}\nService-control failed. Error {\n    \"resolution\": null,\n    \"detail\": [\n        {\n            \"args\": [\n                \"vpxd\"\n            ],\n            \"id\": \"install.ciscommon.service.failstart\",\n            \"localized\": \"An error occurred while starting service 'vpxd'\",\n            \"translatable\": \"An error occurred while starting service '%(0)s'\"\n        }\n    ],\n    \"componentKey\": null,\n    \"problemId\": null\n}<\/code><\/pre>\n\n\n\n
After further investigate through the logs (vpxd.log) found the below entries.<\/p>\n\n\n\n
2023-04-12T19:15:05.810Z warning vpxd[7F5ED15AB700] [Originator@6876 sub=Default opID=1473431e] Default resource used for Solution Manager tab 'extension.com.bitdefender.sve.solutionManagerInfo.tab[0].label'\n2023-04-12T19:15:07.389Z info vpxd[7F5ED3972700] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-43cec339] Certificate not available, starting hostsync for host: host-66487\n2023-04-12T19:15:07.403Z info vpxd[7F5ED3EFD700] [Originator@6876 sub=HostAccess opID=HostSync-host-66487-10b1d72c] Using vpxapi.version.version11 to communicate with vpxa at host hplesx01.hplntstd.com\n2023-04-12T19:15:07.716Z info vpxd[7F5ED3972700] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-43cec339] Certificate not available, starting hostsync for host: host-63038\n2023-04-12T19:15:07.731Z info vpxd[7F5ED19B3700] [Originator@6876 sub=HostAccess opID=HostSync-host-63038-780d04c9] Using vpxapi.version.version11 to communicate with vpxa at host hplesx07.hplntstd.com\n2023-04-12T19:15:08.110Z info vpxd[7F5ED3CF9700] [Originator@6876 sub=vpxLro opID=28a522b2] [VpxLRO] -- BEGIN lro-30 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- 529ff11f-bc7b-4278-90a7-2ca840d2aaeb\n2023-04-12T19:15:08.111Z info vpxd[7F5ED3CF9700] [Originator@6876 sub=vpxLro opID=28a522b2] [VpxLRO] -- FINISH lro-30\n2023-04-12T19:15:08.127Z info vpxd[7F5ED39F3700] [Originator@6876 sub=vpxLro opID=7188e8f6] [VpxLRO] -- BEGIN lro-31 -- SessionManager -- vim.SessionManager.login -- 52781ab5-1cb7-a6e7-4638-d849a7c3142d\n2023-04-12T19:15:08.176Z error vpxd[7F5ED39F3700] [Originator@6876 sub=[SSO] opID=7188e8f6] [UserDirectorySso] AcquireToken exception: N9SsoClient27InvalidCredentialsExceptionE(Authentication failed: Invalid credentials)\n--> [context]zKq7AVECAAAAAK+UEgEWdnB4ZAAAlNIrbGlidm1hY29yZS5zbwAAgicbAN5pGAH\/RwJsaWJzc29jbGllbnQuc28AAUpMAgHLXAIB710CAR4LAgL51P92cHhkAAK61KECFeyigwwPBgFsaWJ2aW0tdHlwZXMuc28AAnA0dQKJK3QC\/iB0ArVFdAL2mHQA8xYkANoZJABLKywE1HMAbGlicHRocmVhZC5zby4wAAWtjg5saWJjLnNvLjYA[\/context]\n2023-04-12T19:15:08.177Z error vpxd[7F5ED39F3700] [Originator@6876 sub=User opID=7188e8f6] Failed to authenticate user <vsphere.local\\administrator>\n2023-04-12T19:15:08.346Z info vpxd[7F5ED3972700] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-43cec339] Certificate not available, starting hostsync for host: host-66132\n2023-04-12T19:15:08.370Z info vpxd[7F5ED35EB700] [Originator@6876 sub=HostAccess opID=HostSync-host-66132-5376a8b5] Using vpxapi.version.version11 to communicate with vpxa at host hplesx09.hplntstd.com\n2023-04-12T19:15:08.541Z warning vpxd[7F5ED35EB700] [Originator@6876 sub=MoTransEVCMgr opID=HostSync-host-66132-5376a8b5] hplesx09.hplntstd.com does not support any EVC mode.\n2023-04-12T19:15:08.647Z info vpxd[7F5ED35EB700] [Originator@6876 sub=Default opID=HostSync-host-66132-5376a8b5] [StringTable::Expand]: nBkts = 256\n-->\n2023-04-12T19:15:08.671Z warning vpxd[7F5ED35EB700] [Originator@6876 sub=OMMOutputCache opID=HostSync-host-66132-5376a8b5] Could not retrieve overhead memory value for (VM [vim.VirtualMachine:vm-66587,ALPUBLIC-0009], Host [vim.HostSystem:host-66132,hplesx09.hplntstd.com]) pair, due to missing host data\n2023-04-12T19:15:08.743Z info vpxd[7F5ED3C78700] [Originator@6876 sub=Default opID=HostSync-host-66132-5376a8b5-SWI-3452778e] [StringTable::Expand]: nBkts = 256\n-->\n2023-04-12T19:15:08.816Z info vpxd[7F5ED3C78700] [Originator@6876 sub=Default opID=HostSync-host-66132-5376a8b5-SWI-3452778e] [StringTable::Expand]: nBkts = 512\n-->\n2023-04-12T19:15:08.820Z warning vpxd[7F5ED3C78700] [Originator@6876 sub=OMM opID=HostSync-host-66132-5376a8b5-SWI-3452778e] Cannot download vm config of [vim.VirtualMachine:vm-66472,WINDOWS-10-TEMPLATE-PUBLIC OR STAFF]: Received SOAP response fault from [<cs p:00007f5e2801ed80, TCP:hplesx09.hplntstd.com:443>]: downloadVMXConfig\n-->\n2023-04-12T19:15:09.074Z info vpxd[7F5ED1932700] [Originator@6876 sub=vpxLro opID=1fb9475b] [VpxLRO] -- BEGIN lro-33 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- 52cb9bd8-069b-459d-4860-d461c57c0500\n2023-04-12T19:15:09.075Z info vpxd[7F5ED1932700] [Originator@6876 sub=vpxLro opID=1fb9475b] [VpxLRO] -- FINISH lro-33\n2023-04-12T19:15:09.079Z info vpxd[7F5ED18B1700] [Originator@6876 sub=vpxLro opID=3cfbb7a2] [VpxLRO] -- BEGIN lro-34 -- ServiceInstance -- vim.ServiceInstance.retrieveInternalContent -- 52e2d9a8-f952-d5cc-7371-4c788a1cd136\n2023-04-12T19:15:09.079Z info vpxd[7F5ED18B1700] [Originator@6876 sub=vpxLro opID=3cfbb7a2] [VpxLRO] -- FINISH lro-34\n2023-04-12T19:15:09.090Z info vpxd[7F5ED81AF700] [Originator@6876 sub=vpxLro opID=31d50bf5] [VpxLRO] -- BEGIN lro-35 -- SessionManager -- vim.SessionManager.login -- 5278be93-b7d4-e554-e9a7-be988f22c8e7\n2023-04-12T19:15:09.133Z info vpxd[7F5ED81AF700] [Originator@6876 sub=vpxLro opID=31d50bf5] [VpxLRO] -- FINISH lro-35\n2023-04-12T19:15:09.140Z info vpxd[7F5ED34E9700] [Originator@6876 sub=vpxLro opID=3a1cc46] [VpxLRO] -- BEGIN lro-36 -- datastore-66465 -- vim.Datastore.refresh -- 5278be93-b7d4-e554-e9a7-be988f22c8e7(521639b0-4fb3-17ef-eb43-e083d9910841)\n2023-04-12T19:15:09.209Z info vpxd[7F5ED19B3700] [Originator@6876 sub=Default opID=HostSync-host-63038-780d04c9] [StringTable::Expand]: nBkts = 256\n-->\n2023-04-12T19:15:09.241Z info vpxd[7F5ED19B3700] [Originator@6876 sub=MoHost opID=HostSync-host-63038-780d04c9] [CheckVersionChange] Version of host h01esx.test.com has changed from 6.5.0 to 7.0.3<\/strong>\n\n2023-04-12T19:25:07.226Z info vpxd[7F72EE6BF700] [Originator@6876 sub=vpxLro opID=406e057c] [VpxLRO] -- FINISH lro-729\n2023-04-12T19:25:07.227Z info vpxd[7F72EE6BF700] [Originator@6876 sub=Default opID=406e057c] [VpxLRO] -- ERROR lro-729 -- SessionManager -- vim.SessionManager.login: vim.fault.InvalidLogin:\n--> Result:\n--> (vim.fault.InvalidLogin) {\n-->    faultCause = (vmodl.MethodFault) null,\n-->    faultMessage = <unset>\n-->    msg = \"\"\n--> }\n--> Args:\n-->\n--> Arg userName:\n--> \"vsphere.local\\administrator\"\n--> Arg password:\n--> (not shown)\n-->\n--> Arg locale:\n-->\n2023-04-12T19:25:09.631Z info vpxd[7F72EF45A700] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-43cec339] Certificate not available, starting hostsync for host: host-63038\n2023-04-12T19:25:09.648Z info vpxd[7F730C0EF700] [Originator@6876 sub=HostAccess opID=HostSync-host-63038-300e51d3] Using vpxapi.version.version11 to communicate with vpxa at host hplesx07.hplntstd.com\n2023-04-12T19:25:10.969Z info vpxd[7F72EEFD1700] [Originator@6876 sub=vpxLro opID=4c68c682] [VpxLRO] -- BEGIN lro-733 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- 522a774b-5823-d9ea-b375-20224ef53b68\n2023-04-12T19:25:10.969Z info vpxd[7F72EEFD1700] [Originator@6876 sub=vpxLro opID=4c68c682] [VpxLRO] -- FINISH lro-733\n2023-04-12T19:25:10.973Z info vpxd[7F72EF6DF700] [Originator@6876 sub=vpxLro opID=72c0a7fe] [VpxLRO] -- BEGIN lro-734 -- ServiceInstance -- vim.ServiceInstance.retrieveInternalContent -- 52adc82e-9e35-2ac4-3609-8f8053aaa78d\n2023-04-12T19:25:10.973Z info vpxd[7F72EF6DF700] [Originator@6876 sub=vpxLro opID=72c0a7fe] [VpxLRO] -- FINISH lro-734\n2023-04-12T19:25:10.976Z info vpxd[7F72EE6BF700] [Originator@6876 sub=vpxLro opID=46554f8d] [VpxLRO] -- BEGIN lro-735 -- SessionManager -- vim.SessionManager.login -- 5230bd7a-0edb-de7b-2949-5e4dd28c6b58\n2023-04-12T19:25:11.018Z info vpxd[7F72EE6BF700] [Originator@6876 sub=vpxLro opID=46554f8d] [VpxLRO] -- FINISH lro-735\n2023-04-12T19:25:11.025Z info vpxd[7F72EE9C5700] [Originator@6876 sub=vpxLro opID=ad9d60e] [VpxLRO] -- BEGIN lro-736 -- SessionManager -- vim.SessionManager.logout -- 5230bd7a-0edb-de7b-2949-5e4dd28c6b58(52b81548-2aae-8895-4320-4206a0ba84ca)\n2023-04-12T19:25:11.028Z info vpxd[7F72EE9C5700] [Originator@6876 sub=vpxLro opID=ad9d60e] [VpxLRO] -- FINISH lro-736\n2023-04-12T19:25:11.126Z info vpxd[7F730C0EF700] [Originator@6876 sub=MoHost opID=HostSync-host-63038-300e51d3] [CheckVersionChange] Version of host h01esx.test.com has changed from 6.5.0 to 7.0.3<\/strong>\n\n<\/code><\/pre>\n\n\n\n
Meantime I was checking the hostd.log and vpxa.log both from the upgraded host. <\/p>\n\n\n\n
2023-04-12T17:00:16.716Z info hostd[2101299] [Originator@6876 sub=Hostsvc.ServiceSystem] TSM running status is false\n2023-04-12T17:00:16.737Z info hostd[2101299] [Originator@6876 sub=Hostsvc.ServiceSystem] TSM-SSH running status is false\n2023-04-12T17:00:16.739Z info hostd[2101299] [Originator@6876 sub=Vsansvc] [HostSystemListener] Refresh vSAN vmknics right after hostd is statrted\n2023-04-12T17:00:16.741Z warning hostd[2101302] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00000050f7fe4ff0, TCP:localhost.localdomain:80>, (null\n), duration: 2msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)\n--> [context]zKq7AVICAgAAANQIPgEKaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAK5ZIQCf0BoA8yEzAE9WLgDMxy0AVAMuAAIRPwE7fQBsaWJwdGhyZWFkLnNvLjAAAm3RDmxpYmMuc28uNgA=[\/context]\n2023-04-12T17:00:16.742Z warning hostd[2102084] [Originator@6876 sub=HttpConnectionPool-000001] Failed to get pooled connection; <cs p:00000050f7fe4ff0, TCP:localhost.localdomain:80>, (null\n), duration: 2msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)\n--> [context]zKq7AVICAgAAANQIPgEKaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAK5ZIQCf0BoA8yEzAE9WLgDMxy0AVAMuAAIRPwE7fQBsaWJwdGhyZWFkLnNvLjAAAm3RDmxpYmMuc28uNgA=[\/context]\n2023-04-12T17:00:16.743Z info hostd[2101302] [Originator@6876 sub=IO.Http] Set user agent error; state: 1, (null), N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)\n\n<\/code><\/pre>\n\n\n\n
Finally I decided to stop the vpxa service on the 7.0 host to stop the communication with vCenter. After vpxd service was started successfully.<\/p>\n\n\n\n
[root@hplesx07:\/var\/log] vmware -vl<\/strong>\nVMware ESXi 7.0.3 build-20842708\nVMware ESXi 7.0 Update 3\n\n[root@hplesx07:\/var\/log] \/etc\/init.d\/vpxa status<\/strong>\nvpxa is running\n\n[root@hplesx07:\/var\/log] \/etc\/init.d\/vpxa stop<\/strong>\nwatchdog-vpxa[2104316]: Terminating watchdog process with PID 2104150\nvpxa stopped.<\/code><\/pre>\n\n\n\n
root@vcenter [ \/var\/log\/vmware\/vpxd ]# service-control --start vmware-vpxd<\/strong>\nPerform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False\n2023-04-12T19:55:33.095Z   Service vpxd state STOPPED\nSuccessfully started service vpxd<\/strong><\/code><\/pre>\n\n\n\n
Thank you for reading. I wish I could help you. Appreciate your comments below. <\/p>\n","protected":false},"excerpt":{"rendered":"
vCenter 6.5.0 17999427 ESXi version 7.0.3 build-20842708 I was able to troubleshoot interesting issue today. Have you ever upgrade the ESXi to higher version before the vCenter. This is interesting why ESXi has been upgraded to version 7.0 U3 while vCenter is running with 6.5. There were two hosts in this cluster one was 6.5 […]<\/p>\n","protected":false},"author":1,"featured_media":249,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18],"tags":[],"class_list":["post-470","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-vmware"],"_links":{"self":[{"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/posts\/470","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/comments?post=470"}],"version-history":[{"count":4,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/posts\/470\/revisions"}],"predecessor-version":[{"id":476,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/posts\/470\/revisions\/476"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/media\/249"}],"wp:attachment":[{"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/media?parent=470"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/categories?post=470"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/tags?post=470"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}