Storing your data in the cloud

Migrate from VMware to OpenStack

Introduction

NOTE: Content in this article was copied from:
http://www.npit.nl/blog/2015/08/13/migrate-to-openstack/

How to migrate VMware virtual machines (Linux and Windows) from VMware ESXi to OpenStack. With the steps and commands, it should be easy to create scripts that do the migration automatically.

Just to make it clear, these steps do not convert traditional (non-cloud) applications to cloud ready applications. In this case we started to use OpenStack as a traditional hypervisor infrastructure.

Disclaimer: This information is provided as-is. I will decline any responsibility caused by or with these steps and/or commands. I suggest you don’t try and/or test these commands in a production environment. Some commands are very powerful and can destroy configurations and data in Ceph and OpenStack. So always use this information with care and great responsibility.

Global steps 

1. Convert VMDK files to single-file VMDK files (VMware only)
2. Convert the single-file virtual hard drive files to RAW disk files
3. Expand partition sizes (optional)
4. Inject VirtIO drivers
5. Inject software and scripts
6. Create Cinder volumes of all the virtual disks
7. Ceph import
8. Create Neutron port with predefined IP-address and/or MAC address
9. Create and boot the instance in OpenStack

Specifications

Here are the specifications of the infrastructure I used for the migration:

• Cloud platform: OpenStack Icehouse
• Cloud storage: Ceph
• Windows instances: Windows Server 2003 to 2012R2 (all versions, except Itanium)
• Linux instances: RHEL5/6/7, SLES, Debian and Ubuntu
• Only VMDK files from ESXi can be converted, I was not able to convert VMDK files from VMware Player with qemu-img
• I have no migration experience with encrypted source disks
• OpenStack provides VirtIO paravirtual hardware to instances

Requirements

A Linux ‘migration node’ (tested with Ubuntu 14.04/15.04, RHEL6, Fedora 19-21) with:

• Operating system (successfully tested with the following):

• RHEL6 (RHEL7 did not have the “libguestfs-winsupport” -necessary for NTFS formatted disks- package available at the time of writing)
• Fedora 19, 20 and 21
• Ubuntu 14.04 and 15.04

• Network connections to a running OpenStack environment (duh). Preferable not over the internet, as we need ‘super admin’ permissions. Local network connections are usually faster than connections over the internet.
• Enough hardware power to convert disks and run instances in KVM (sizing depends on the instances you want to migrate in a certain amount of time).

We used a server with 8x Intel Xeon E3-1230 @ 3.3GHz, 32GB RAM, 8x 1TB SSD and we managed to migrate >300GB per hour. However, it really depends on the usage of the disk space of the instances. But also my old company laptop (Core i5 and 4GB of RAM and an old 4500rmp HDD) worked, but obviously the performance was very poor.

• Local sudo (root) permissions on the Linux migration node
• QEMU/KVM host
• Permissions to OpenStack (via Keystone)
• Permissions to Ceph

• Unlimited network access to the OpenStack API and Ceph (I have not figured out the network ports that are necessary)
• VirtIO drivers (downloadable from Red Hat, Fedora, and more)
• Packages (all packages should be in the default distributions repository):

“python-cinderclient” (to control volumes)

“python-keystoneclient” (for authentication to OpenStack)

“python-novaclient” (to control instances)

“python-neutronclient” (to control networks)

“python-httplib2” (to be able to communicate with webservice)

“libguestfs-tools” (to access the disk files)

“libguestfs-winsupport” (should be separately installed on RHEL based systems only)

“libvirt-client” (to control KVM)

“qemu-img” (to convert disk files)

“ceph” (to import virtual disk into Ceph)

“vmware-vdiskmanager” (to convert disks, downloadable from VMware)

Steps

1. Convert VMDK to VMDK 

In the next step (convert VMDK to RAW) I was unable to directly convert non-single VMDK files (from ESXi 5.5) to RAW files. So, in the first place I converted all VMDK files to single-file VMDK files.

I used the vmware-vdiskmanager tool (use argument -r to define the source disk, use argument -t 0 to specify output format ‘single growable virtal disk’) to complete this action:

vmware-vdiskmanager -r <source.vmdk> -t 0 <target.vmdk>

Do this for all disks of the virtual machine to be migrated.

See also: https://www.vmware.com/support/ws45/doc/disks_vdiskmanager_eg_ws.html

2. Convert VMDK to RAW 

The second step should convert the VMDK file to RAW format. With the libguestfs-tools it is much easier to inject stuff (files and registry) in RAW disks than in VMDK files. And we need the RAW format anyway, to import the virtual disks in Ceph.

Use this command to convert the VMDK file to RAW format (use argument -p to see the progress, use -O raw to define the target format):

qemu-img convert -p <source.vmdk> -O raw <target.raw>

For virtual machines with VirtIO support (newer Linux kernels) you could also convert the VMDK files directly into Ceph by using the rbd output. In that case, go to chapter 6 and follow from there.

Also check: http://linux.die.net/man/1/qemu-img

3. Expand partitions (optional) 

Some Windows servers I migrated had limited free disk space on the Windows partition. There was not enough space to install new management applications. So, I mounted the RAW file on a loop device to check the free disk space.

losetup -f

(to get the first available loop device, like /dev/loop2)

losetup </dev/loop2> <disk.raw>

kpartx -av </dev/loop2> 

(do for each partition):

mount /dev/mapper/loop2p* /mnt/loop2p*

du -h (to check free disk space)

umount /mnt/loop2p*

kpartx -d </dev/loop2> 

losetup -d </dev/loop2>

If the disk space is insufficient, then you may want to increase the disk size:

Check which partition you want to expand (you can see the partitions, filesystem and size. This should be enough to determine the partition you want to expand):

virt-filesystems --long -h --all -a <diskfile.raw>

Create a new disk (of 50G in this example):

truncate -s 50G <newdiskfile.raw>

Copy the original disk to the newly created disk while expanding the partition (in this example /dev/sda2):

virt-resize --expand /dev/sda2 <originaldisk.raw> <biggerdisk.raw>

Test the new and bigger disk before you remove the original disk!

Also check: http://libguestfs.org/virt-resize.1.html  and http://libguestfs.org/virt-filesystems.1.html.

4. Inject drivers

4.1 Windows Server 2012

Since Windows Server 2012 and Windows 8.0, the driver store is protected by Windows. It is very hard to inject drivers in an offline Windows disk. Windows Server 2012 does not boot from VirtIO hardware by default. So, I took these next steps to install the VirtIO drivers into Windows. Note that these steps should work for all tested Windows versions (2003/2008/2012).

1. Create a new KVM instance. Make sure the Windows disk is created as IDE disk! The network card shoud be a VirtIO device.
2. Add an extra VirtIO disk, so Windows can install the VirtIO drivers.
3. Off course you should add a VirtIO ISO or floppy drive which contains the drivers. You could also inject the driver files with virt-copy-in and inject the necessary registry settings (see  paragraph 4.4) for automatic installation of the drivers.
4. Start the virtual machine and give Windows about two minutes to find the new VirtIO hardware. Install the drivers for all newly found hardware. Verify that there are no devices that have no driver installed.
5. Shutdown the system and remove the extra VirtIO disk.
6. Redefine the Windows disk as VirtIO disk (this was IDE) and start the instance. It should boot without problems. Shut down the virtual machine.

4.2 Linux (kernel 2.6.25 and above)

Linux kernels 2.6.25 and above have already built-in support for VirtIO hardware. So there is no need to inject VirtIO drivers. Create and start a new KVM virtual machine with VirtIO hardware. When LVM partitions do not mount automatically, run this to fix:

(log in)

mount -o remount,rw /

pvscan

vgscan

reboot

(after the reboot all LVM partitions should be mounted and Linux should boot fine)

Shut down the virtual machine when done.

4.3 Linux (kernel older than 2.6.25)

Some Linux distributions provide VirtIO modules for older kernel versions. Some examples:

• Red Hat provides VirtIO support for RHEL 3.9 and up
• SuSe provides VirtIO support for SLES 10 SP3 and up

The steps for older kernels are:

1. Create KVM instance:
2. Linux (prior to kernel 2.6.25): Create and boot KVM instance with IDE hardware (this is limited to 4 disks in KVM, as only one IDE controller can be configured which results in 4 disks!). I have not tried SCSI or SATA as I only had old Linux machines with no more than 4 disks. Linux should start without issues.
3. Load the virtio modules (this is distribution specific): RHEL (older versions):https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/ch10s04.html) and for SLES 10 SP3 systems:https://www.suse.com/documentation/opensuse121/book_kvm/data/app_kvm_virtio_install.htm
4. Shutdown the instance.
5. Change all disks to VirtIO disks and boot the instance. It should now boot without problems.
6. Shut down the virtual machine when done.

For Red Hat, see: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/ch10s04.html

For SuSe, see: https://www.suse.com/documentation/opensuse121/book_kvm/data/app_kvm_virtio_install.htm

4.4 Windows Server 2008 (and older versions); depricated

For Windows versions prior to 2012 you could also use these steps to insert the drivers (the steps in 4.1 should also work for Windows 2003/2008).

1. Copy all VirtIO driver files (from the downloaded VirtIO drivers) of the corresponding Windows version and architecture to C:\Drivers\. You can use the tool virt-copy-in to copy files and folders into the virtual disk.
2. Copy *.sys files to %WINDIR%\system32\drivers\ (you may want to use virt-ls to look for the correct directory. Note that Windows is not very consistent with lower and uppercase characters). You can use the tool virt-copy-in to copy files and folders into the virtual disk.
3. The Windows registry should combine the hardware ID’s and drivers, but there are no VirtIO drivers installed in Windows by default. So we need to do this by ourselves. You could inject the registry file with virt-win-reg. If you choose to copy all VirtIO drivers to an other location than C:\Drivers, you must change the “DevicePath” variable in the last line (the most easy way is to change it in some Windows machine and then export the registry file, and use that line).

Registry file (I called the file mergeviostor.reg, as it holds the VirtIO storage information only):

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00000000]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00020000]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00021AF4]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00021AF4&rev_00]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1004&subsys_00081af&rev_00]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\viostor]
 "ErrorControl"=dword:00000001
 "Group"="SCSI miniport"
 "Start"=dword:00000000
 "Tag"=dword:00000021
 "Type"=dword:00000001
 "ImagePath"="system32\\drivers\\viostor.sys"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion]
 "DevicePath"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,69,00,6e,00,66,00,3b,00,63,00,3a,00,5c,00,44,00,72,00,69,00,76,00,65,00,72,00,73,00,00,00

When these steps have been executed, Windows should boot from VirtIO disks without BSOD. Also all other drivers (network, balloon etc.) should install automatically when Windows boots.

See: https://support.microsoft.com/en-us/kb/314082  (written for Windows XP, but it is still usable for Windows 2003 and 2008).

See also: http://libguestfs.org/virt-copy-in.1.html and http://libguestfs.org/virt-win-reg.1.html

5. Customize the virtual machine (optional) 

To prepare the operating system to run in OpenStack, you probably would like to uninstall some software (like VMware Tools and drivers), change passwords and install new management tooling etc.. You can automate this by writing a script that does this for you (those scripts are beyond the scope of this article). You should be able to inject the script and files with the virt-copy-in command into the virtual disk.

5.1 Automatically start scripts in Linux

I started the scripts within Linux manually as I only had a few Linux servers to migrate. I guess Linux engineers should be able to completely automate this.

5.2 Automatically start scripts in Windows

I choose the RunOnce method to start scripts at Windows boot as it works on all versions of Windows that I had to migrate. You can put a script in the RunOnce by injecting a registry file. RunOnce scripts are only run when a user has logged in. So, you should also inject a Windows administrator UserName, Password and set AutoAdminLogon to ‘1’. When Windows starts, it will automatically log in as the defined user. Make sure to shut down the virtual machine when done.

Example registry file to auto login into Windows (with user ‘Administrator’ and password ‘Password’) and start the C:\StartupWinScript.vbs.:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce]
 "Script"="cscript C:\\StartupWinScript.vbs"
 "Parameters"=""

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon]
 "AutoAdminLogon"="1"
 "UserName"="Administrator"
 "Password"="Password"

6. Create Cinder volumes 

For every disk you want to import, you need to create a Cinder volume. The volume size in the Cinder command does not really matter, as we remove (and recreate with the import) the Ceph device in the next step. We create the cinder volume only to create the link between Cinder and Ceph.

Nevertheless, you should keep the volume size the same as the disk you are planning to import. This is useful for the overview in the OpenStack dashboard (Horizon).

You create a cinder volume with the following command (the size is in GB, you can check the available volume types by cinder type-list):

cinder create --display-name <name_of_disk> <size> --volume-type <volumetype>

Note the volume id (you can also find the volume id with the following command) as we need the ids in the next step.

cinder list | grep <name_of_disk>

Cinder command information: http://docs.openstack.org/cli-reference/content/cinderclient_commands.html

7. Ceph import 

As soon as the Cinder volumes are created, we can import the RAW image files. But first we need to remove the actual Ceph disk. Make sure you remove the correct Ceph block device!

In the first place you should know in which Ceph pool the disk resides. Then remove the volume from Ceph (the volume-id is the volume id that you noted in the previous step ‘Create Cinder volumes’):

rbd -p <ceph_pool> rm volume-<volume-id>

Next step is to import the RAW file into the volume on Ceph (all ceph* arguments will result in better performance. The raw_disk_file variable is the complete path to the raw file. The volume-id is the ID that you noted before).

rbd -p <ceph_pool> --rbd_cache=true --rbd_cache_size=134217728 --rbd_cache_max_dirty=100663296 --rbd_default_format=2 import <raw_disk_file> volume-<volume-id>

Do this for all virtual disks of the virtual machine.

Note that you could also convert VMDK files directly into Ceph. In that case replace the rbd import command by the qemu-img convert command and use the rbd output.

Be careful! The rbd command is VERY powerful (you could destroy more data on Ceph than intended)!

8. Create Neutron port (optional) 

In some cases you might want to set a fixed IP-address or a MAC-address. You can do that by create a port with neutron and use that port in the next step (create and boot instance in OpenStack).

You should first know what the network_name is (nova net-list), you need the ‘Label’. Only the network_name is mandatory. You could also add security groups by adding

 --security-group <security_group_name>

Add this parameter for each security group, so if you want to add i.e. 6 security-groups, you should add this parameter 6 times.

neutron port-create --fixed-ip ip_address=<ip_address> --mac-address <mac_address> <network_name> --name <port_name>

Note the id of the neutron port, you will need it in the next step.

9. Create and boot instance in OpenStack 

Now we have everything prepared to create an instance from the Cinder volumes and an optional neutron port.

Note the volume-id of the boot disk.

Now you only need to know the id of the flavor you want to choose. Run nova flavor-list to get the flavor-id of the desired flavor.

Now you can create and boot the new instance:

nova boot <instance_name> --flavor <flavor_id> --boot-volume <boot_volume_id> --nic port-id=<neutron_port_id>

Note the Instance ID. Now, add each other disk of the instance by executing this command (if there are other volumes you want to add):

nova volume-attach <instance_ID> <volume_id>

OpenStack use cases and tips

OpenStack is probably one of the most fast-growing project in the history of Open Source and as such it is prone to be obfuscated by its shininess. Understanding when and even IF to employ a solution or another is key to successful business.

Why OpenStack?

If by any case you stumbled here without knowing what OpenStack is I highly suggest you to get a glance of OpenStack before proceeding in reading this article. That said, there are two ways of using OpenStack: through a provider or hosting it. In this article I’ll discuss on the use of OpenStack to manage infrastructure and avoid application management purposely (it is a way too broad topic to merge it with this one). Before we dive in the argument let’s summarize the advantages and the disadvantages of OpenStack:

Pros:

• It can easily scale across multiple nodes.
• It is easily manageable and provides useful statistics.
• It can include different hypervisors.
• It is vendor-free hence no vendor lock-in.
• Has multiple interfaces: Web, CLI and REST.
• It is modular: you can add or remove components you don’t need.
• Components like Sahara, Ironic and Magnum can literally revolutionize your workflow.
• It can be employed over an existing virtualized environment.

Cons:

• Installation and upgrades are literally barriers. (If you’re not using tools like RDO.)
• It can break easily if not managed correctly.
• Lacks of skills in the staff is a major issue.
• Can become overkill for small projects.

Hypothetical use cases

Case 1: University

The reality universities face is ever-changing and providing a stable solution to host services can sometimes become a difficult task. In a small university OpenStack is almost not needed, and employing it is overkill, but in a large university the use of OpenStack can be justified by the enormous amount of hardware resources employed in order to provide services to its students. In this scenario the use of OpenStack simplifies the management of internal services and also improve security by providing isolation (achieved by virtualization). In case the university has multiple locations over a geographical region, centralizing hardware resources in the principal location reduces the number of machines needed and the cost of maintenance allowing a more flexible approach. OpenStack helps in this case only if the university is large enough and/or has multiple locations.

Case 2: IT School

This case is probably a must if the IT team is skilled enough. Employing OpenStack in a IT School gives enormous flexibility both to its students and to its IT team. An OpenStack deployment could be easily configured to use LDAP and provide students with their own lab. Using a large central cluster could also replace the need of desktop machines (to be replaced with thin-clients), this would achieve an easier maintenance and could potentially improve the security (many schools use outdated software because of the absence of update plans). If thin-clients are not a choice OpenStack Ironic could provide an easy way to provision bare metal machines. Of course the use of OS is only indicated if the school is large enough and has enough machines, but usually IT schools tend to have lots of machines and older hardware.

Case 3: IT Small-Medium Business / Enterprise

Depending on the true nature of the SMB/Enterprise OpenStack can become a key to a successful business. Enterprises are the primary OpenStack field but when should you deploy it in SMBs? The first answer is when you expect growth. If you know the workload in a year will double you will surely need to address this problem. OpenStack innate ability to scale out will enable the SMB to scale at will and avoid overbuying. A big data company could benefit from Sahara which combines the power of Hadoop/Spark with the flexibility of OpenStack. For the most adventurous there’s also containerization with the Magnum component enabling the use of Docker and Kubernetes (which is not included in the actual release but will be in the next one). OpenStack is surely a great way to start hosting your SMB/Enterprise cloud, but it can also prove difficult to manage it; skill availability can become a major issue. Also, migrating from an existing virtualized environment might be difficult depending on the case.

Conclusions

As you can see OpenStack can become your best friend if you know how to use it. However theinstallation is the hardest barrier in the first place, and in enterprises migrating from an existing environment to OpenStack can be a challenge. The second problem is the lack of skills needed in order to install and manage the system. But if you’ve got both problems solved you can experience a boost in flexibility and a reduction in costs that you could never expect. If the project is big enough and you haven’t already started, OpenStack can look difficult from the outside but will pay better in the long run.