{"id":201,"date":"2021-06-18T17:43:02","date_gmt":"2021-06-18T17:43:02","guid":{"rendered":"https:\/\/systemsmedic.net\/?p=201"},"modified":"2023-02-17T18:59:39","modified_gmt":"2023-02-17T18:59:39","slug":"vmotion-issue-stucks-at-14-operation-timed-out","status":"publish","type":"post","link":"https:\/\/systemsmedic.net\/index.php\/2021\/06\/18\/vmotion-issue-stucks-at-14-operation-timed-out\/","title":{"rendered":"vMotion issue stucks at 14% operation timed out"},"content":{"rendered":"\n<p>ESXi 5.5 \/ vCenter 5.5<\/p>\n\n\n\n<p>Today I was asked to put one host in maintained mode and shutdown the ESXi server. Prior to this I had to relocate the VMs in to different ESXi servers in same cluster. After the checking the DRS setting, I just put server in maintained mode. Most of the VMs were migrated successfully except one VM. What I did is cancel the maintained task from the vCenter task list and try to manually migrate this VM. After the manual migration initiated below issues and error messages were appeared.<\/p>\n\n\n\n<p><strong>Experienced symptoms<\/strong><\/p>\n\n\n\n<p><strong><em>vMotion task stucks at 14%<\/em><\/strong><\/p>\n\n\n\n<p>After few min later task was canceled saying <strong><em>Operation timed out<\/em><\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_36-1024x120.png\" alt=\"\" class=\"wp-image-202\" width=\"1021\" height=\"119\" srcset=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_36-1024x120.png 1024w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_36-300x35.png 300w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_36-768x90.png 768w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_36-600x70.png 600w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_36.png 1272w\" sizes=\"auto, (max-width: 1021px) 100vw, 1021px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"339\" src=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2018\/08\/2021-06-18_23h03_55-1024x339.png\" alt=\"\" class=\"wp-image-200\" srcset=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2018\/08\/2021-06-18_23h03_55-1024x339.png 1024w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2018\/08\/2021-06-18_23h03_55-300x99.png 300w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2018\/08\/2021-06-18_23h03_55-768x254.png 768w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2018\/08\/2021-06-18_23h03_55-600x199.png 600w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2018\/08\/2021-06-18_23h03_55.png 1051w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h09_40-1024x122.png\" alt=\"\" class=\"wp-image-203\" width=\"1024\" height=\"122\" srcset=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h09_40-1024x122.png 1024w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h09_40-300x36.png 300w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h09_40-768x92.png 768w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h09_40-600x72.png 600w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h09_40.png 1132w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>After I checked the vMotion network and everything related but nothing wrong with the settings because all other VMs were migrated successfully. After I decide to perform a storage vMotion to verify is this only related with host vMotion but unfortunately, it\u2019s given different error. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_57-1024x202.png\" alt=\"\" class=\"wp-image-204\" width=\"1020\" height=\"201\" srcset=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_57-1024x202.png 1024w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_57-300x59.png 300w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_57-768x151.png 768w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_57-600x118.png 600w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_22h59_57.png 1131w\" sizes=\"auto, (max-width: 1020px) 100vw, 1020px\" \/><\/figure>\n\n\n\n<p>Finally I searched through google and found <a href=\"https:\/\/kb.vmware.com\/s\/article\/2054100\">this VMware KB<\/a>&nbsp; which has been describe the same issue as mine one. Thanks for the VMware.<\/p>\n\n\n\n<p>As per the KB article there should be two .vswp files in VM directory. Normally there are only one vswp file must be there. As VMware says I could see two files as below screen shot.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"348\" src=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_04-1024x348.png\" alt=\"\" class=\"wp-image-205\" srcset=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_04-1024x348.png 1024w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_04-300x102.png 300w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_04-768x261.png 768w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_04-600x204.png 600w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_04.png 1051w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>VMware recommend powering down the VM to fix this issue, but I tried to minimize the down time of the production VM. So, I tried to find the lock outsource of the duplicate <strong>vswp<\/strong> file by using <strong><em>VMKSFSTOOL<\/em><\/strong> command in ESXi shell.<\/p>\n\n\n\n<p>Eg: <strong>vmkfstool -D \u201cpath to vswp file\u201d<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"77\" src=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_10-1024x77.png\" alt=\"\" class=\"wp-image-206\" srcset=\"https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_10-1024x77.png 1024w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_10-300x23.png 300w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_10-768x58.png 768w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_10-600x45.png 600w, https:\/\/systemsmedic.net\/wp-content\/uploads\/2021\/06\/2021-06-18_23h00_10.png 1130w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>As per the above screen shot it was locked by one of the ESXi servers in the cluster. I tried restarting that host but unfortunately lock has been moved to different ESXi server. So finally, I had to coordinate the downtime with application team and shut the VM. Once power down I was able to vMotion the VM and finally performed a storage vMotion and duplicate <strong>vswp<\/strong> file disappeared.<\/p>\n\n\n\n<p><strong>Thanks for reading. Stay safe!!!<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>ESXi 5.5 \/ vCenter 5.5 Today I was asked to put one host in maintained mode and shutdown the ESXi server. Prior to this I had to relocate the VMs in to different ESXi servers in same cluster. After the checking the DRS setting, I just put server in maintained mode. Most of the VMs [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":388,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[],"class_list":["post-201","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-centos"],"_links":{"self":[{"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/posts\/201","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/comments?post=201"}],"version-history":[{"count":1,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/posts\/201\/revisions"}],"predecessor-version":[{"id":207,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/posts\/201\/revisions\/207"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/media\/388"}],"wp:attachment":[{"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/media?parent=201"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/categories?post=201"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/systemsmedic.net\/index.php\/wp-json\/wp\/v2\/tags?post=201"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}