How to migrate VMware virtual machines (Linux and Windows) from VMware ESXi to OpenStack. With the steps and commands, it should be easy to create scripts that do the migration automatically.
Just to make it clear, these steps do not convert traditional (non-cloud) applications to cloud ready applications. In this case we started to use OpenStack as a traditional hypervisor infrastructure.
Disclaimer: This information is provided as-is. I will decline any responsibility caused by or with these steps and/or commands. I suggest you don’t try and/or test these commands in a production environment. Some commands are very powerful and can destroy configurations and data in Ceph and OpenStack. So always use this information with care and great responsibility.
Global steps
- Convert VMDK files to single-file VMDK files (VMware only)
- Convert the single-file virtual hard drive files to RAW disk files
- Expand partition sizes (optional)
- Inject VirtIO drivers
- Inject software and scripts
- Create Cinder volumes of all the virtual disks
- Ceph import
- Create Neutron port with predefined IP-address and/or MAC address
- Create and boot the instance in OpenStack
Specifications
Here are the specifications of the infrastructure I used for the migration:
- Cloud platform: OpenStack Icehouse
- Cloud storage: Ceph
- Windows instances: Windows Server 2003 to 2012R2 (all versions, except Itanium)
- Linux instances: RHEL5/6/7, SLES, Debian and Ubuntu
- Only VMDK files from ESXi can be converted, I was not able to convert VMDK files from VMware Player with qemu-img
- I have no migration experience with encrypted source disks
- OpenStack provides VirtIO paravirtual hardware to instances
Requirements
A Linux ‘migration node’ (tested with Ubuntu 14.04/15.04, RHEL6, Fedora 19-21) with:
- Operating system (successfully tested with the following):
- RHEL6 (RHEL7 did not have the “libguestfs-winsupport” -necessary for NTFS formatted disks- package available at the time of writing)
- Fedora 19, 20 and 21
- Ubuntu 14.04 and 15.04
- Network connections to a running OpenStack environment (duh). Preferable not over the internet, as we need ‘super admin’ permissions. Local network connections are usually faster than connections over the internet.
- Enough hardware power to convert disks and run instances in KVM (sizing depends on the instances you want to migrate in a certain amount of time).
We used a server with 8x Intel Xeon E3-1230 @ 3.3GHz, 32GB RAM, 8x 1TB SSD and we managed to migrate >300GB per hour. However, it really depends on the usage of the disk space of the instances. But also my old company laptop (Core i5 and 4GB of RAM and an old 4500rmp HDD) worked, but obviously the performance was very poor.
- Local sudo (root) permissions on the Linux migration node
- QEMU/KVM host
- Permissions to OpenStack (via Keystone)
- Permissions to Ceph
- Unlimited network access to the OpenStack API and Ceph (I have not figured out the network ports that are necessary)
- VirtIO drivers (downloadable from Red Hat, Fedora, and more)
- Packages (all packages should be in the default distributions repository):
“python-cinderclient” (to control volumes)
“python-keystoneclient” (for authentication to OpenStack)
“python-novaclient” (to control instances)
“python-neutronclient” (to control networks)
“python-httplib2” (to be able to communicate with webservice)
“libguestfs-tools” (to access the disk files)
“libguestfs-winsupport” (should be separately installed on RHEL based systems only)
“libvirt-client” (to control KVM)
“qemu-img” (to convert disk files)
“ceph” (to import virtual disk into Ceph)
“vmware-vdiskmanager” (to convert disks, downloadable from VMware)
Steps
1. Convert VMDK to VMDK
In the next step (convert VMDK to RAW) I was unable to directly convert non-single VMDK files (from ESXi 5.5) to RAW files. So, in the first place I converted all VMDK files to single-file VMDK files.
I used the vmware-vdiskmanager tool (use argument -r to define the source disk, use argument -t 0 to specify output format ‘single growable virtal disk’) to complete this action:
vmware-vdiskmanager -r <source.vmdk> -t 0 <target.vmdk>
Do this for all disks of the virtual machine to be migrated.
2. Convert VMDK to RAW
The second step should convert the VMDK file to RAW format. With the libguestfs-tools it is much easier to inject stuff (files and registry) in RAW disks than in VMDK files. And we need the RAW format anyway, to import the virtual disks in Ceph.
Use this command to convert the VMDK file to RAW format (use argument -p to see the progress, use -O raw to define the target format):
qemu-img convert -p <source.vmdk> -O raw <target.raw>
For virtual machines with VirtIO support (newer Linux kernels) you could also convert the VMDK files directly into Ceph by using the rbd output. In that case, go to chapter 6 and follow from there.
3. Expand partitions (optional)
Some Windows servers I migrated had limited free disk space on the Windows partition. There was not enough space to install new management applications. So, I mounted the RAW file on a loop device to check the free disk space.
losetup -f
(to get the first available loop device, like /dev/loop2)
losetup </dev/loop2> <disk.raw>
kpartx -av </dev/loop2>
(do for each partition):
mount /dev/mapper/loop2p* /mnt/loop2p*
du -h (to check free disk space)
umount /mnt/loop2p*
kpartx -d </dev/loop2>
losetup -d </dev/loop2>
If the disk space is insufficient, then you may want to increase the disk size:
Check which partition you want to expand (you can see the partitions, filesystem and size. This should be enough to determine the partition you want to expand):
virt-filesystems --long -h --all -a <diskfile.raw>
Create a new disk (of 50G in this example):
truncate -s 50G <newdiskfile.raw>
Copy the original disk to the newly created disk while expanding the partition (in this example /dev/sda2):
virt-resize --expand /dev/sda2 <originaldisk.raw> <biggerdisk.raw>
Test the new and bigger disk before you remove the original disk!
4. Inject drivers
4.1 Windows Server 2012
Since Windows Server 2012 and Windows 8.0, the driver store is protected by Windows. It is very hard to inject drivers in an offline Windows disk. Windows Server 2012 does not boot from VirtIO hardware by default. So, I took these next steps to install the VirtIO drivers into Windows. Note that these steps should work for all tested Windows versions (2003/2008/2012).
- Create a new KVM instance. Make sure the Windows disk is created as IDE disk! The network card shoud be a VirtIO device.
- Add an extra VirtIO disk, so Windows can install the VirtIO drivers.
- Off course you should add a VirtIO ISO or floppy drive which contains the drivers. You could also inject the driver files with virt-copy-in and inject the necessary registry settings (see paragraph 4.4) for automatic installation of the drivers.
- Start the virtual machine and give Windows about two minutes to find the new VirtIO hardware. Install the drivers for all newly found hardware. Verify that there are no devices that have no driver installed.
- Shutdown the system and remove the extra VirtIO disk.
- Redefine the Windows disk as VirtIO disk (this was IDE) and start the instance. It should boot without problems. Shut down the virtual machine.
4.2 Linux (kernel 2.6.25 and above)
Linux kernels 2.6.25 and above have already built-in support for VirtIO hardware. So there is no need to inject VirtIO drivers. Create and start a new KVM virtual machine with VirtIO hardware. When LVM partitions do not mount automatically, run this to fix:
(log in)
mount -o remount,rw /
pvscan
vgscan
reboot
(after the reboot all LVM partitions should be mounted and Linux should boot fine)
Shut down the virtual machine when done.
4.3 Linux (kernel older than 2.6.25)
Some Linux distributions provide VirtIO modules for older kernel versions. Some examples:
- Red Hat provides VirtIO support for RHEL 3.9 and up
- SuSe provides VirtIO support for SLES 10 SP3 and up
The steps for older kernels are:
- Create KVM instance:
- Linux (prior to kernel 2.6.25): Create and boot KVM instance with IDE hardware (this is limited to 4 disks in KVM, as only one IDE controller can be configured which results in 4 disks!). I have not tried SCSI or SATA as I only had old Linux machines with no more than 4 disks. Linux should start without issues.
- Load the virtio modules (this is distribution specific): RHEL (older versions):https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/ch10s04.html) and for SLES 10 SP3 systems:https://www.suse.com/documentation/opensuse121/book_kvm/data/app_kvm_virtio_install.htm
- Shutdown the instance.
- Change all disks to VirtIO disks and boot the instance. It should now boot without problems.
- Shut down the virtual machine when done.
4.4 Windows Server 2008 (and older versions); depricated
For Windows versions prior to 2012 you could also use these steps to insert the drivers (the steps in 4.1 should also work for Windows 2003/2008).
- Copy all VirtIO driver files (from the downloaded VirtIO drivers) of the corresponding Windows version and architecture to C:\Drivers\. You can use the tool virt-copy-in to copy files and folders into the virtual disk.
- Copy *.sys files to %WINDIR%\system32\drivers\ (you may want to use virt-ls to look for the correct directory. Note that Windows is not very consistent with lower and uppercase characters). You can use the tool virt-copy-in to copy files and folders into the virtual disk.
- The Windows registry should combine the hardware ID’s and drivers, but there are no VirtIO drivers installed in Windows by default. So we need to do this by ourselves. You could inject the registry file with virt-win-reg. If you choose to copy all VirtIO drivers to an other location than C:\Drivers, you must change the “DevicePath” variable in the last line (the most easy way is to change it in some Windows machine and then export the registry file, and use that line).
Registry file (I called the file mergeviostor.reg, as it holds the VirtIO storage information only):
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00000000]
"ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
"Service"="viostor"
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00020000]
"ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
"Service"="viostor"
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00021AF4]
"ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
"Service"="viostor"
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00021AF4&rev_00]
"ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
"Service"="viostor"
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1004&subsys_00081af&rev_00]
"ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
"Service"="viostor"
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\viostor]
"ErrorControl"=dword:00000001
"Group"="SCSI miniport"
"Start"=dword:00000000
"Tag"=dword:00000021
"Type"=dword:00000001
"ImagePath"="system32\\drivers\\viostor.sys"
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion]
"DevicePath"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,69,00,6e,00,66,00,3b,00,63,00,3a,00,5c,00,44,00,72,00,69,00,76,00,65,00,72,00,73,00,00,00
When these steps have been executed, Windows should boot from VirtIO disks without BSOD. Also all other drivers (network, balloon etc.) should install automatically when Windows boots.
5. Customize the virtual machine (optional)
To prepare the operating system to run in OpenStack, you probably would like to uninstall some software (like VMware Tools and drivers), change passwords and install new management tooling etc.. You can automate this by writing a script that does this for you (those scripts are beyond the scope of this article). You should be able to inject the script and files with the virt-copy-in command into the virtual disk.
5.1 Automatically start scripts in Linux
I started the scripts within Linux manually as I only had a few Linux servers to migrate. I guess Linux engineers should be able to completely automate this.
5.2 Automatically start scripts in Windows
I choose the RunOnce method to start scripts at Windows boot as it works on all versions of Windows that I had to migrate. You can put a script in the RunOnce by injecting a registry file. RunOnce scripts are only run when a user has logged in. So, you should also inject a Windows administrator UserName, Password and set AutoAdminLogon to ‘1’. When Windows starts, it will automatically log in as the defined user. Make sure to shut down the virtual machine when done.
Example registry file to auto login into Windows (with user ‘Administrator’ and password ‘Password’) and start the C:\StartupWinScript.vbs.:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce]
"Script"="cscript C:\\StartupWinScript.vbs"
"Parameters"=""
[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon]
"AutoAdminLogon"="1"
"UserName"="Administrator"
"Password"="Password"
6. Create Cinder volumes
For every disk you want to import, you need to create a Cinder volume. The volume size in the Cinder command does not really matter, as we remove (and recreate with the import) the Ceph device in the next step. We create the cinder volume only to create the link between Cinder and Ceph.
Nevertheless, you should keep the volume size the same as the disk you are planning to import. This is useful for the overview in the OpenStack dashboard (Horizon).
You create a cinder volume with the following command (the size is in GB, you can check the available volume types by cinder type-list):
cinder create --display-name <name_of_disk> <size> --volume-type <volumetype>
Note the volume id (you can also find the volume id with the following command) as we need the ids in the next step.
cinder list | grep <name_of_disk>
7. Ceph import
As soon as the Cinder volumes are created, we can import the RAW image files. But first we need to remove the actual Ceph disk. Make sure you remove the correct Ceph block device!
In the first place you should know in which Ceph pool the disk resides. Then remove the volume from Ceph (the volume-id is the volume id that you noted in the previous step ‘Create Cinder volumes’):
rbd -p <ceph_pool> rm volume-<volume-id>
Next step is to import the RAW file into the volume on Ceph (all ceph* arguments will result in better performance. The raw_disk_file variable is the complete path to the raw file. The volume-id is the ID that you noted before).
rbd -p <ceph_pool> --rbd_cache=true --rbd_cache_size=134217728 --rbd_cache_max_dirty=100663296 --rbd_default_format=2 import <raw_disk_file> volume-<volume-id>
Do this for all virtual disks of the virtual machine.
Note that you could also convert VMDK files directly into Ceph. In that case replace the rbd import command by the qemu-img convert command and use the rbd output.
Be careful! The rbd command is VERY powerful (you could destroy more data on Ceph than intended)!
8. Create Neutron port (optional)
In some cases you might want to set a fixed IP-address or a MAC-address. You can do that by create a port with neutron and use that port in the next step (create and boot instance in OpenStack).
You should first know what the network_name is (nova net-list), you need the ‘Label’. Only the network_name is mandatory. You could also add security groups by adding
--security-group <security_group_name>
Add this parameter for each security group, so if you want to add i.e. 6 security-groups, you should add this parameter 6 times.
neutron port-create --fixed-ip ip_address=<ip_address> --mac-address <mac_address> <network_name> --name <port_name>
Note the id of the neutron port, you will need it in the next step.
9. Create and boot instance in OpenStack
Now we have everything prepared to create an instance from the Cinder volumes and an optional neutron port.
Note the volume-id of the boot disk.
Now you only need to know the id of the flavor you want to choose. Run nova flavor-list to get the flavor-id of the desired flavor.
Now you can create and boot the new instance:
nova boot <instance_name> --flavor <flavor_id> --boot-volume <boot_volume_id> --nic port-id=<neutron_port_id>
Note the Instance ID. Now, add each other disk of the instance by executing this command (if there are other volumes you want to add):
nova volume-attach <instance_ID> <volume_id>
