VMware, Microsoft, Linux and Networking

vCenter 6.5 stopped working after ESXi upgraded 6.5 to 7.0 mistakenly without doing vCenter upgrade.

VMware
Administrator

vCenter 6.5.0 17999427

ESXi version 7.0.3 build-20842708

I was able to troubleshoot interesting issue today. Have you ever upgrade the ESXi to higher version before the vCenter. This is interesting why ESXi has been upgraded to version 7.0 U3 while vCenter is running with 6.5. There were two hosts in this cluster one was 6.5 where the vCenter is living.

First start the troubleshooting from the vCenter. access the vCenter via SSH and check the service status. vCenter service is stopped. 

root@vcenter [ ~ ]# service-control --status --all
Running:
 applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui

Stopped:
 vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-vcha vmware-vpxd

First and foremost tried to start it. 

root@vcenter [ ~ ]# service-control --start vmware-vpxd
Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_c                                                                                                             oreossvcs=False, include_leafossvcs=False
2023-04-12T19:14:58.710Z   Service vpxd state STOPPED
Error executing start on service vpxd. Details {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
Service-control failed. Error {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}

After further investigate through the logs (vpxd.log) found the below entries.

2023-04-12T19:15:05.810Z warning vpxd[7F5ED15AB700] [Originator@6876 sub=Default opID=1473431e] Default resource used for Solution Manager tab 'extension.com.bitdefender.sve.solutionManagerInfo.tab[0].label'
2023-04-12T19:15:07.389Z info vpxd[7F5ED3972700] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-43cec339] Certificate not available, starting hostsync for host: host-66487
2023-04-12T19:15:07.403Z info vpxd[7F5ED3EFD700] [Originator@6876 sub=HostAccess opID=HostSync-host-66487-10b1d72c] Using vpxapi.version.version11 to communicate with vpxa at host hplesx01.hplntstd.com
2023-04-12T19:15:07.716Z info vpxd[7F5ED3972700] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-43cec339] Certificate not available, starting hostsync for host: host-63038
2023-04-12T19:15:07.731Z info vpxd[7F5ED19B3700] [Originator@6876 sub=HostAccess opID=HostSync-host-63038-780d04c9] Using vpxapi.version.version11 to communicate with vpxa at host hplesx07.hplntstd.com
2023-04-12T19:15:08.110Z info vpxd[7F5ED3CF9700] [Originator@6876 sub=vpxLro opID=28a522b2] [VpxLRO] -- BEGIN lro-30 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- 529ff11f-bc7b-4278-90a7-2ca840d2aaeb
2023-04-12T19:15:08.111Z info vpxd[7F5ED3CF9700] [Originator@6876 sub=vpxLro opID=28a522b2] [VpxLRO] -- FINISH lro-30
2023-04-12T19:15:08.127Z info vpxd[7F5ED39F3700] [Originator@6876 sub=vpxLro opID=7188e8f6] [VpxLRO] -- BEGIN lro-31 -- SessionManager -- vim.SessionManager.login -- 52781ab5-1cb7-a6e7-4638-d849a7c3142d
2023-04-12T19:15:08.176Z error vpxd[7F5ED39F3700] [Originator@6876 sub=[SSO] opID=7188e8f6] [UserDirectorySso] AcquireToken exception: N9SsoClient27InvalidCredentialsExceptionE(Authentication failed: Invalid credentials)
--> [context]zKq7AVECAAAAAK+UEgEWdnB4ZAAAlNIrbGlidm1hY29yZS5zbwAAgicbAN5pGAH/RwJsaWJzc29jbGllbnQuc28AAUpMAgHLXAIB710CAR4LAgL51P92cHhkAAK61KECFeyigwwPBgFsaWJ2aW0tdHlwZXMuc28AAnA0dQKJK3QC/iB0ArVFdAL2mHQA8xYkANoZJABLKywE1HMAbGlicHRocmVhZC5zby4wAAWtjg5saWJjLnNvLjYA[/context]
2023-04-12T19:15:08.177Z error vpxd[7F5ED39F3700] [Originator@6876 sub=User opID=7188e8f6] Failed to authenticate user <vsphere.local\administrator>
2023-04-12T19:15:08.346Z info vpxd[7F5ED3972700] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-43cec339] Certificate not available, starting hostsync for host: host-66132
2023-04-12T19:15:08.370Z info vpxd[7F5ED35EB700] [Originator@6876 sub=HostAccess opID=HostSync-host-66132-5376a8b5] Using vpxapi.version.version11 to communicate with vpxa at host hplesx09.hplntstd.com
2023-04-12T19:15:08.541Z warning vpxd[7F5ED35EB700] [Originator@6876 sub=MoTransEVCMgr opID=HostSync-host-66132-5376a8b5] hplesx09.hplntstd.com does not support any EVC mode.
2023-04-12T19:15:08.647Z info vpxd[7F5ED35EB700] [Originator@6876 sub=Default opID=HostSync-host-66132-5376a8b5] [StringTable::Expand]: nBkts = 256
-->
2023-04-12T19:15:08.671Z warning vpxd[7F5ED35EB700] [Originator@6876 sub=OMMOutputCache opID=HostSync-host-66132-5376a8b5] Could not retrieve overhead memory value for (VM [vim.VirtualMachine:vm-66587,ALPUBLIC-0009], Host [vim.HostSystem:host-66132,hplesx09.hplntstd.com]) pair, due to missing host data
2023-04-12T19:15:08.743Z info vpxd[7F5ED3C78700] [Originator@6876 sub=Default opID=HostSync-host-66132-5376a8b5-SWI-3452778e] [StringTable::Expand]: nBkts = 256
-->
2023-04-12T19:15:08.816Z info vpxd[7F5ED3C78700] [Originator@6876 sub=Default opID=HostSync-host-66132-5376a8b5-SWI-3452778e] [StringTable::Expand]: nBkts = 512
-->
2023-04-12T19:15:08.820Z warning vpxd[7F5ED3C78700] [Originator@6876 sub=OMM opID=HostSync-host-66132-5376a8b5-SWI-3452778e] Cannot download vm config of [vim.VirtualMachine:vm-66472,WINDOWS-10-TEMPLATE-PUBLIC OR STAFF]: Received SOAP response fault from [<cs p:00007f5e2801ed80, TCP:hplesx09.hplntstd.com:443>]: downloadVMXConfig
-->
2023-04-12T19:15:09.074Z info vpxd[7F5ED1932700] [Originator@6876 sub=vpxLro opID=1fb9475b] [VpxLRO] -- BEGIN lro-33 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- 52cb9bd8-069b-459d-4860-d461c57c0500
2023-04-12T19:15:09.075Z info vpxd[7F5ED1932700] [Originator@6876 sub=vpxLro opID=1fb9475b] [VpxLRO] -- FINISH lro-33
2023-04-12T19:15:09.079Z info vpxd[7F5ED18B1700] [Originator@6876 sub=vpxLro opID=3cfbb7a2] [VpxLRO] -- BEGIN lro-34 -- ServiceInstance -- vim.ServiceInstance.retrieveInternalContent -- 52e2d9a8-f952-d5cc-7371-4c788a1cd136
2023-04-12T19:15:09.079Z info vpxd[7F5ED18B1700] [Originator@6876 sub=vpxLro opID=3cfbb7a2] [VpxLRO] -- FINISH lro-34
2023-04-12T19:15:09.090Z info vpxd[7F5ED81AF700] [Originator@6876 sub=vpxLro opID=31d50bf5] [VpxLRO] -- BEGIN lro-35 -- SessionManager -- vim.SessionManager.login -- 5278be93-b7d4-e554-e9a7-be988f22c8e7
2023-04-12T19:15:09.133Z info vpxd[7F5ED81AF700] [Originator@6876 sub=vpxLro opID=31d50bf5] [VpxLRO] -- FINISH lro-35
2023-04-12T19:15:09.140Z info vpxd[7F5ED34E9700] [Originator@6876 sub=vpxLro opID=3a1cc46] [VpxLRO] -- BEGIN lro-36 -- datastore-66465 -- vim.Datastore.refresh -- 5278be93-b7d4-e554-e9a7-be988f22c8e7(521639b0-4fb3-17ef-eb43-e083d9910841)
2023-04-12T19:15:09.209Z info vpxd[7F5ED19B3700] [Originator@6876 sub=Default opID=HostSync-host-63038-780d04c9] [StringTable::Expand]: nBkts = 256
-->
2023-04-12T19:15:09.241Z info vpxd[7F5ED19B3700] [Originator@6876 sub=MoHost opID=HostSync-host-63038-780d04c9] [CheckVersionChange] Version of host h01esx.test.com has changed from 6.5.0 to 7.0.3

2023-04-12T19:25:07.226Z info vpxd[7F72EE6BF700] [Originator@6876 sub=vpxLro opID=406e057c] [VpxLRO] -- FINISH lro-729
2023-04-12T19:25:07.227Z info vpxd[7F72EE6BF700] [Originator@6876 sub=Default opID=406e057c] [VpxLRO] -- ERROR lro-729 -- SessionManager -- vim.SessionManager.login: vim.fault.InvalidLogin:
--> Result:
--> (vim.fault.InvalidLogin) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = ""
--> }
--> Args:
-->
--> Arg userName:
--> "vsphere.local\administrator"
--> Arg password:
--> (not shown)
-->
--> Arg locale:
-->
2023-04-12T19:25:09.631Z info vpxd[7F72EF45A700] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-43cec339] Certificate not available, starting hostsync for host: host-63038
2023-04-12T19:25:09.648Z info vpxd[7F730C0EF700] [Originator@6876 sub=HostAccess opID=HostSync-host-63038-300e51d3] Using vpxapi.version.version11 to communicate with vpxa at host hplesx07.hplntstd.com
2023-04-12T19:25:10.969Z info vpxd[7F72EEFD1700] [Originator@6876 sub=vpxLro opID=4c68c682] [VpxLRO] -- BEGIN lro-733 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- 522a774b-5823-d9ea-b375-20224ef53b68
2023-04-12T19:25:10.969Z info vpxd[7F72EEFD1700] [Originator@6876 sub=vpxLro opID=4c68c682] [VpxLRO] -- FINISH lro-733
2023-04-12T19:25:10.973Z info vpxd[7F72EF6DF700] [Originator@6876 sub=vpxLro opID=72c0a7fe] [VpxLRO] -- BEGIN lro-734 -- ServiceInstance -- vim.ServiceInstance.retrieveInternalContent -- 52adc82e-9e35-2ac4-3609-8f8053aaa78d
2023-04-12T19:25:10.973Z info vpxd[7F72EF6DF700] [Originator@6876 sub=vpxLro opID=72c0a7fe] [VpxLRO] -- FINISH lro-734
2023-04-12T19:25:10.976Z info vpxd[7F72EE6BF700] [Originator@6876 sub=vpxLro opID=46554f8d] [VpxLRO] -- BEGIN lro-735 -- SessionManager -- vim.SessionManager.login -- 5230bd7a-0edb-de7b-2949-5e4dd28c6b58
2023-04-12T19:25:11.018Z info vpxd[7F72EE6BF700] [Originator@6876 sub=vpxLro opID=46554f8d] [VpxLRO] -- FINISH lro-735
2023-04-12T19:25:11.025Z info vpxd[7F72EE9C5700] [Originator@6876 sub=vpxLro opID=ad9d60e] [VpxLRO] -- BEGIN lro-736 -- SessionManager -- vim.SessionManager.logout -- 5230bd7a-0edb-de7b-2949-5e4dd28c6b58(52b81548-2aae-8895-4320-4206a0ba84ca)
2023-04-12T19:25:11.028Z info vpxd[7F72EE9C5700] [Originator@6876 sub=vpxLro opID=ad9d60e] [VpxLRO] -- FINISH lro-736
2023-04-12T19:25:11.126Z info vpxd[7F730C0EF700] [Originator@6876 sub=MoHost opID=HostSync-host-63038-300e51d3] [CheckVersionChange] Version of host h01esx.test.com has changed from 6.5.0 to 7.0.3

Meantime I was checking the hostd.log and vpxa.log both from the upgraded host. 

2023-04-12T17:00:16.716Z info hostd[2101299] [Originator@6876 sub=Hostsvc.ServiceSystem] TSM running status is false
2023-04-12T17:00:16.737Z info hostd[2101299] [Originator@6876 sub=Hostsvc.ServiceSystem] TSM-SSH running status is false
2023-04-12T17:00:16.739Z info hostd[2101299] [Originator@6876 sub=Vsansvc] [HostSystemListener] Refresh vSAN vmknics right after hostd is statrted
2023-04-12T17:00:16.741Z warning hostd[2101302] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00000050f7fe4ff0, TCP:localhost.localdomain:80>, (null
), duration: 2msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
--> [context]zKq7AVICAgAAANQIPgEKaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAK5ZIQCf0BoA8yEzAE9WLgDMxy0AVAMuAAIRPwE7fQBsaWJwdGhyZWFkLnNvLjAAAm3RDmxpYmMuc28uNgA=[/context]
2023-04-12T17:00:16.742Z warning hostd[2102084] [Originator@6876 sub=HttpConnectionPool-000001] Failed to get pooled connection; <cs p:00000050f7fe4ff0, TCP:localhost.localdomain:80>, (null
), duration: 2msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
--> [context]zKq7AVICAgAAANQIPgEKaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAK5ZIQCf0BoA8yEzAE9WLgDMxy0AVAMuAAIRPwE7fQBsaWJwdGhyZWFkLnNvLjAAAm3RDmxpYmMuc28uNgA=[/context]
2023-04-12T17:00:16.743Z info hostd[2101302] [Originator@6876 sub=IO.Http] Set user agent error; state: 1, (null), N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)

Finally I decided to stop the vpxa service on the 7.0 host to stop the communication with vCenter. After vpxd service was started successfully.

[root@hplesx07:/var/log] vmware -vl
VMware ESXi 7.0.3 build-20842708
VMware ESXi 7.0 Update 3

[root@hplesx07:/var/log] /etc/init.d/vpxa status
vpxa is running

[root@hplesx07:/var/log] /etc/init.d/vpxa stop
watchdog-vpxa[2104316]: Terminating watchdog process with PID 2104150
vpxa stopped.
root@vcenter [ /var/log/vmware/vpxd ]# service-control --start vmware-vpxd
Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False
2023-04-12T19:55:33.095Z   Service vpxd state STOPPED
Successfully started service vpxd

Thank you for reading. I wish I could help you. Appreciate your comments below.

I am Pubudu Wijerathna who is the author of SystemsMedic blog.
Website https://systemsmedic.net
Posts created 16

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts

Begin typing your search term above and press enter to search. Press ESC to cancel.

Back To Top