Hi all,

I have a Windows 2008 R2 VM running on VMWare ESXi 4.0 that is connected to a Dell PV MD3000i SAN.  After installing 30+ Windows Updates last week, I am now seeing the following errors in the Event Log:

• Source       = Ntfs,  Event ID: 55
• The       file system structure on the disk is corrupt and unusable.  Please       run the chkdsk utility on the volume StudentsHomeDrives.

• Source       = Ntfs, Event ID: 55
• The       file system structure on the disk is corrupt and unusable.  Please       run the chkdsk utility on the volume Shared

• Source       = Application Popup, Event ID: 26
• Application       Popup: Windows - Delayed Write Failed:  Exception Processing Message       0xc000a082 Parameters ...

• Source       Ntfs, Event ID: 141
• The       description for the Event ID 141 from source Ntfs cannot be found. ....
• The       following information was included with the event:        \Device\HarddiskVolumeShadowCopy225\$Mft

These errors started immediately after installing the updates. (May be a co-incidence.)  These errors only occur during the nightly backup via the Windows Server Backup utility.  The backups report that they are successful and a quick glance seems that they are indeed finishing fine.  The server seems to be running fine and the volumes in question are accessible and no-one has reported any issues accessing files.

I have ran chkdsk via the GUI (No repair) and both volumes report they are fine.  However, If I select the repair option and force a boot time chkdsk, chkdsk fails with the following error during the verfying indexes stage:  An unspecified error occurred (696e647863686b2e dee)  This error occurs even with volumes that are not reporting any errors in the Event Log.  So this may be an issue that has existed from day 1?

I have updated the ESXi servers to the latest patches and the corresponding VMWare tools on the VM in question.  I have tried migrating the VM to another ESXi server in the cluster.  I am not seeing these errors on any of the other VMs in the ESXi cluster.  I have only installed the same updates on 1 other VM in the ESXi cluster.  But, that VM is a print server and I am not backing up that server.

There is a firmware update released in October for the MD3000i.  I will try to install this over Christmas break.  However, this setup was working fine for over 6 months until the recent Windows Updates.  I've narrowed down to the following updates that seem like they might be related to causing this (if anyone knows the KB numbers by heart):

982214, 981852, 978886, 2436673, 2345886, 2305420, 2249857

I've already uninstalled 2249857 as it is the only one that MS recommended may be related to volumes.  Regarding Microsoft... I entered a support ticket through the Partner Portal.  But, as soon as they hear non-microsoft virtualization they don't help much.  They did however suggest I call and open an official support ticket ($$)

Just wondering what I should be looking for...

Thanks all,

Jeff

Hi there.

We have used this:

http://communities.vmware.com/thread/394669?start=0&tstart=0

http://en.blog.skydriver.org/tag/bios-440-rom/

as a source.

But still after applying bios 440 rom file to VM, the VM just want start up. It throws an error: 0x0000005D.

We have copied bios into folder where VM is located.

We had add this parameters:

bios440.filename = ./bios.440.rom

mce.enable = TRUE

cpuid.hypervisor.v0 = FALSE

vmGenCounter.enable = FALSE

But an error message still appears and we are unable to boot 2012 server.

Any other hints please?

With best regards,

I've been struggling with a new Openfiler 2.99/ESXi 4.1 installation for several days, and  am in need of some further help or suggestions. My write performance  with an iSCSI volume to an ESXi 4.1 guest is about 2 MB/s.  Read  performance is fine at about 75 MB/s.  With write performance like that,  the machines are just crawling along.  I had a number of guest VMs  installed locally on the internal (non-RAID) HD of the host ESXi  machine, and was getting good performance - read and write in the 70  MB/s range.  When I migrated one over to the new OF setup, write  performance is essentially non-existent!

The server is a new HP  N40L, with (4) 2TB Seagate Barracude ST2000DL003-9VT1 drives.  6 GB of  RAM, and a 2-port BCM5709 gigabit NIC.  OF 2.99 is installed to a USB  drive, and as soon as it was installed I used the system update function  to update all packages.  OF is connected to the ESXi host through a  Trend S80 switch that supports jumbo frames, although I'm not using  them.  There's nothing else on the switch.

All 4 drives are  configured in a single volume group, RAID 5 w/ 5.73 TB of usable space.   There is currently only 1 iSCSI volume, 1 TB in size, and a single  LUN.  I checked hdparm locally on the OF box, and am getting buffered  disk reads of the individual disks of 145 MB/s.  Buffered reads from  /md0 are 320 MB/s. 

I initially configured the vSwitch and  vmnic for jumbo frames, 9000 MTU.  During troubleshooting though I set  them back to 1500, and changed OF to 1500 to eliminate a jumbo frames  issue or a switch issue.  No improvement.

I turned off flow  control for tx, rx, and autonegotiate.  No change.  I verified that all  of the iSCSI parameters in the software initiator match those in OF.  No  change.

I'm really out of ideas at this point.  Any suggestions for things to check/change would be greatly appreciated.

Every time I reboot a host, it gets stuck on "iscsi_vmk loaded successfully" for about 10-15 minutes. The first couple of times, I've gotten impatient and powered off the host and rebooted thinking it's locked up. One time I let it go and it successfully completed the boot process. Now I'm thinking this is normal and the host's vmware iSCSI software initiator must be rescanning all my iSCSI datastores. Can anyone confirm this is what's taking place or do I have a potential problem here?

Hello again everybody,

I'm trying to copy a couple of 50GB Virtual machines from the primary hard drives datastore to the newly mounted Netgear ReadyNAS device via NFS.  I first tried it via the VSphere Client interface, then from command line via NFS.  Both times thus far, I get to around 15 or 20GB of copying when the ReadyNAS device disconnects and then becomes inactive - unmounted.  I can not remount it unless I reboot the ReadyNAS device, even though I can still ssh into the ReadyNAS or web console into it.  All fingers are pointing to the ReadyNAS device as the culprit, maybe too much for it to handle over NFS.  Any ideas why I cannot complete this copy to a ReadyNAS device?  I'm ultimately trying to reconfigure the RAID on this server so I need to copy the VM's off and eventually back to the reconfigured server.

I checked for any and all firmware updates on the NAS device so it's on the latest version.

Thanks,

Jim

I have a Server 2008 vm that won't boot up. I installed some Windows Updates and then restarted. I see the Windows splash screen and then it just goes black. I did a hard shutdown and powered it back on and tried booting into both Safe Mode and Safe Mode with Networking. Both times, it freezes after crcdisk.sys. Is the only way to fix this a repair using the install disk? I do have a backup of the server in Veeam but it's a month old and I just did a backup of the databases to the local C drive before I did the Windows Updates and restart. If I do a restore, it will wipe out those backups. Is there a way to access the C drive of the vm without logging in, just so I can copy those database backups?

Hi all,

I'de like to know which OID I've to use to monitor my datastores.

I've add ESXi MIBs in my monitoring appliance (PRTG and Zenoss).

I'm particularly intreresting in the following stats :

     - IOPS

     - Disk space

     - Disk use

     - Disk free

I've googled to find this info without sucess.

Can anybody help me on this topic ?

Thanks'

Thierry

Today I used the standalone vmware converter to convert a Hyper V machine (Windows 2008R2 SP1) that was running on a physical server (Windows 2008R2) to an server running Esxi 4.0 The conversion went fine, after a few reboots I installed vmware tools, uninstalled the converter, a few more reboots to let the hardware sort itself out. This machine that I converted was a terminal server, for some reason now, I am unable to RDP into it. I can ping it on the local network, I can acces the internet from the machine and the local network. RDP is enabled, the windows event logs don't show any errors, all services that should be running are running. I am stumped. My next move will be to shut down the machine, edit the hardware, remove and add the network card.

Thoughts?

I'm getting these errors when trying to kill a VM's process in the SSH interface connected to my ESX host.

~ # vim-cmd vmsvc/getallvms | grep -i MV4680
Skipping invalid VM '11888'
Skipping invalid VM '12032'
Skipping invalid VM '12304'
~ # vim-cmd vmsvc/power.off 11888
Powering off VM:
Power off failed
~ #

The VM isn't appearing int the Vcenter inventory and the Opertional System is stuck someway but still responding...

Pinging mv4680.X.X.X.X [X.X.X.X] with 32 bytes of data:

Reply from X.X.X.X: bytes=32 time=1ms TTL=127
Reply from X.X.X.X: bytes=32 time<1ms TTL=127
Reply from X.X.X.X: bytes=32 time<1ms TTL=127
Reply from X.X.X.X: bytes=32 time<1ms TTL=127

Ping statistics for X.X.X.X:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms

I believe that the VM pid is one of the three showed before, but i can't kill it.

Help pls?

Hi,

Recently we've installed ESXi 4.1 Update 1 embedded version on the internal SD card (SanDisk 2GB) for a few Dell PowerEdge M610 blade servers using the ESXi recovery CD from Dell, also available for download from the VMware site:

http://downloads.vmware.com/d/details/dell_esxi_recoverycd/ZHcqYip3ZWJkKmV3

Yesterday we rebooted the servers to make sure everything would come back up.

To our surprise when ESXi come back up on each server, the whole configuration was gone: IP address, server name, password, everything was lost, the configuration was back as if we just installed ESXi.

Any ideas why that happened?

We did a proper reboot for ESXi: one of the servers was rebooted using vCenter, the others were rebooted using the ESXi DCUI using the F11 key.

Before the reboot, one of the ESXi server was up with its configuration for a few days and was managed through vCenter, the others were installed an hour or so before the reboot and weren't connected to vCenter yet: no matter how long they were up, all of them lost the configuration after the reboot.

And we've been unable to reproduce the issue so far, we've reconfigured ESXi on a couple of those servers today and rebooted them from the ESX DCUI: each of them came back online just fine this time, the configuration is still there.

We didn't reconfigure all of them, let me know if there is anything we can check on those still untouched after the reboot to tell how come they lost the configuration.

Thanks.

Hello,

i  know this problem seems to be common, but either there are no solutions available, or solutions are not applicable.

i will start with the hardware and software we are running esxi4 (latest patchlevel). The Problem is occuring on more than one blade,

but i will describe the scenario with one blade. (other blades do have about the same vm setup. hardware is the same)

we do not have any special resource / share configuration. all share,... settings are as on a fresh esxi installation.

Hardware:

   - BLC460 G1, 2 x xeon E5430, 16 gb ram 2 local sas disc, 4 nics (2 for network, 2 for nas)

   - NetApp Filer

The Datastores are mounted through NFS

VMs on the Blade:

- linux (4 gb ram, 2 cpus)

- linux (2 gb ram, 2cpus)

- linux (2 gb ram, 2cpus)

- Windows Server 2008R2 (4 gb ram, 2 cpus)

- all vms do have 4mb video ram (maybe this the problem?)

all vms do have 1 or more e1000 interfaces configured. some vms are migrated from physical machines. the win 2008 r2 server is a fresh installed 2 weeks ago.

we have no blade where we exceed our 8 core / 16 gb ram limit. it doesnt matter if windows 2003, 2008, 2008 R2,.. is running. all linux machines have no gui started. all running in run level 3. windows servers do have vmware tools installed.

now.. from time to time (sometimes after 10 days, sometimes after 2 months) we cant connect to the windows machines anymore. neither over console, nor over rdp. when trying with the vcenter console we get the message  "unable to connect to the mks: error connecting to /bin/vmx process". if this happens, all windows machines on the blade are affected. no pingable anymore, they are just dead. we cant shutdown the vms (stops at 95%). all we can do is reboot the whole esxi server.

sometimes this problem occurs:

any ideas?

I use Veeam Backup & Replication to backup our VMs and I've been having an issue backing up 1 VM for the past week or more. It has always backed up with no issues in the past and all of my other VMs are backing up just fine. However, after less than a minute, this VM always fails and I get an error that says, "Creating VM snapshot | Error: A snapshot operation cannot be performed." Trying to rule out Veeam as the issue, I just tried creating a snapshot of the server in vSphere and I got the error, "The virtual disk is either corrupted or not a supported format." After looking around, it appears that error usually has something to do with a correct snapshot or an issue after trying to delete a snapshot but this is the first one I've ever done, so I don't think that's the issue. Anywhere else I can look?

Thank you.

Hi,

   I need to enable "Traffic management" on a vmkernel port group from the commandline.  I'm trying to use vim-cmd, but i can't seem to find the proper command for this.  Is vim-cmd what i'm suppose to be using or is there a different way of doing this?

Thanks in advance.

Please anyone can confirm or using any of the ff. Intel PCI LAN card on ESXi 4.1?

Intel® PRO/1000 GT Quad Port Server Adapter or
Intel® PRO/1000 MT Dual Port Server Adapter or
Intel® PRO/1000 MT Quad Port Server Adapter or
Intel® PRO/1000 PT Dual Port Server Adapter or
Intel® PRO/1000 PT Quad Port Server Adapter or

Thanks in advance!

Hi,

We are running a VMWare Esxi 5.1 with 3 VMs for around 4 month with no issues. Suddenly from today we are seeing datastore1 is inactive. Due to this we are unable to access our VMs through console. Getting "unable to connect to the mks virtual machine config file does not exist"  while trying to access VM. Now, how can we re-active the datastore. Please help!!!

BR//Kamrul

Hi All,

I'm having problem with my Windows Server 2003 x86 VM, in the middle of installing IIS server, What I did is that i insert the CD into the ESXi host and then from

my VSphere client I point to use the host CD-ROM drive and it hangs,

What I've done:

1. Force restart the VM and it stuck in 95% of restart process.

2. in the SSL console of ESXi: services.sh restart

3. re-register the VM from datastore view into the VCenter (with the same name VMWS05) and it hangs with Status "In Progress", after 20+ minutes it came with the following error:

     An error occurred while communicating with the remote host.

   and the VM is not registered in the VCenter.

4. tried to kill the process like the following but still stuck ?

*     ~ # ps -ef | grep vmws05*

*     3854821 14064 Worker#0:Win2003x86_vmws05 /bin/vmx*

*     14071      vmm0:Win2003x86_vmws05*

*     ~ # kill -9 14071*

*     ash: cannot kill pid 14071: No such process*

*     ~ # ps -ef | grep vmws05*

*     3854821 14064 Worker#0:Win2003x86_vmws05 /bin/vmx*

*     14071      vmm0:Win2003x86_vmws05*

*     ~ # kill -9 14064*

*     ash: cannot kill pid 14064: No such process*

*     ~ # kill -9 3854821*

*     ash: cannot kill pid 3854821: No such process*

*     ~ # ps -ef | grep vmws05*

*     3854821 14064 Worker#0:Win2003x86_vmws05 /bin/vmx*

*     14071      vmm0:Win2003x86_vmws05*

5. I also unable to delete the swap file of this particular VM:

     Cannot delete file Win2003x86_vmws05/Win2003x86_vmws05-301abb4a.vswp

any kind of help would be greatly appreciated.

Note:

VMware ESXi 4.0.0 build 244038

VMwre VCenter Server 4.0.0 build 208111

Kind Regards,

AWT

In ESXi 4, Windows 2008 VMs seem to want to default to the E1000 model NIC when creating a new one. However, after reading the KB on this (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001805), is seems the new VMXNET 3 is supported for Windows 2008 (and Windows 2003 for that matter).

So which is the best to use? It seems the VMXNET it the best choice because it has all the latest bells and whistles - but sometimes that is a bad thing too. The KB really does not say WHAT to use, but rather what options are available...unless I overlooked it.

Any advice?

SB

Today I used the standalone vmware converter to convert a Hyper V machine (Windows 2008R2 SP1) that was running on a physical server (Windows 2008R2) to an server running Esxi 4.0 The conversion went fine, after a few reboots I installed vmware tools, uninstalled the converter, a few more reboots to let the hardware sort itself out. This machine that I converted was a terminal server, for some reason now, I am unable to RDP into it. I can ping it on the local network, I can acces the internet from the machine and the local network. RDP is enabled, the windows event logs don't show any errors, all services that should be running are running. I am stumped. My next move will be to shut down the machine, edit the hardware, remove and add the network card.

Thoughts?

Windows 2008 R2 , x64 VM. I have a SAN LUN (with a single VMFS dedicated to this VM for OS partition) connected inside ESXi. I have installed 2 vNICs for access to an iSCSI network. The  host's vmk ports, vswitches, etc have been setup according to vmware/eqluallogics iSCSI configuration guidelines. Jumbo frames, flow control, etc. It all works well. Performance is good. To use equallogics snapshot software, I installed the MS iSCSI initiator and various Equallogic tools for MPIO. I created a LUN for  data files.

So, it works, but the issue I am having is that approximately every 30 minutes or so, the initiator seems to lose connection with the SAN, its shows iSCSIPrt events 7 and 20, indicating it lost a connection to the data LUN and a vss-control LUN (used for snapshotting). Once this happens, both of those LUNs are re-connected exactly 1 minute later. During this downtime, the VM completely locks up. No network access, even the VM's console inside of the vSphere client locks up.

I am wondering if anyone has heard of this sort of thing or can point me to where the logs are kept in ESXi (/var/log?, I have to still setup a vMA appliance).

Also, is this not a good idea to do iscsi initiator inside a VM? This is to be a production box, but if this is not a supported/good idea, then I will investigate other configurations.

Anyone else's thoughts/experiences would be welcome. I am wondering if it would be better to add in 2 more NICs and dedicate them to iSCSI initiator traffic, instead of using the current 2 SAN nics (HP NC382T cards) for all SAN traffic (esxi sw initiator to SAN, and MS initiator to SAN). I am wondering if having the VM's C drive connected to SAN in ESXi's software iSCSI initiator AND having the same VM's second drive connected via MS iSCSI initiator (both using the same 2 NICs on the ESXi host) might be causing issues.

I need to troubleshoot the freezing issues, not sure where to start. I have this all working on a test physical server without issue, so its got to be something on the vmware side of things. I think.

Hi there.

We have used this:

http://communities.vmware.com/thread/394669?start=0&tstart=0

http://en.blog.skydriver.org/tag/bios-440-rom/

as a source.

But still after applying bios 440 rom file to VM, the VM just want start up. It throws an error: 0x0000005D.

We have copied bios into folder where VM is located.

We had add this parameters:

bios440.filename = ./bios.440.rom

mce.enable = TRUE

cpuid.hypervisor.v0 = FALSE

vmGenCounter.enable = FALSE

But an error message still appears and we are unable to boot 2012 server.

Any other hints please?

With best regards,