ESXi ramdisk full or file table full
We are having an issue with one of the VM’s, we had to shut down 2 couple of VM’s on an ESXi server and later when we tried to power them up they were not powering up.
From the logs, we could see that there was an existing issue on the server, which was stopping it from creating files in the ramdisk either because there was no space or file table full.
As of now, we are planning to go through the directories and check for probable files which might be causing the issue (using df, vdf, ls commands ), VMware KB suggests this is known issue, either SMP traps filling the file table or HP helper log files filling the space in ramdisk which we are looking into.
Wanted inputs from someone who might have faced similar issues and if they are aware of safe temporary workaround which can help us in getting the VM’s started until we identify and close the actual problem.
Idea is to avoid or minimize the downtime of the host.
During the start-up of virtual machines on the host, ESXi services were unable to create necessary files required as the file table of ram disk root was full.
o Datastore checks confirmed that there is enough space available.
o hostd.log file had logs related to ramdisk root is full.
o syslog.log file had logs related to sfcb-CIMXML-Processor and sfcb-vmware_base. It’s suspected that this process was generating a lot of small log files which filled up the allowed number of files that can be created on the ESXi host or the file table.
o Vmkernl.log and vmkwarning.log files had logs indicating attempts to create files during Vm-startup that failed as the file table was full, also had logs associated with failure of file creation by several services.
o Vobd.log had logs associated with failure of file creation by services.
o SSH checks revealed that SSH was not running and was unable to start.
o Disk and Inode usage checks were unable to progress as the connection attempt failed using SSH or ESXi Shell.