VR : CloudStack Virtual Router
CS: CloudStack
UserData: Any data that has to be provided to userVm during deploy
ConfigDrive: Drive created by mounting an ISO containing userdata and password info specific to userVm.

Use cases

Cloud Admin create offering for the network without any service.
The new way of accessing the user data is by Cloudstack provided script. This script needs to be placed inside the vm.
Password setting script for the user vm got changed. So password enabled template needs to be updated with the given password script.

Functional requirements

Support deploying a VM in a network whose network offering does not provide any service
In this case network doesn't have VR.
Support deploying VM in network with only userdata service.
Retrieving IP assigned by external DHCP server to userVM. Store it in CS DB.
In the absence of VR userdata/password/ssh key access should be provided with ConfigDrive.

Assumptions

This feature is for only CS Advanced Zone shared network.
DHCP Admin has to configure the DHCP server with IP range in the shared network broadcast domain.
The IP lease time may not be infinite like VR.

Work Flow

Pre-setup

DHCP admin configures the DHCP server with DHCP IP range for the shared network.
DHCP server should be running in the shared network broadcast domain to assign IP address to VMs.
DNS server is up and running outside the cloudstack.

CS Admin

Admin has to create a network offering without any services for the shared network.
Admin also has to create Shared network using the network offering created in step # 1.
Create required templates for the user Vm (with required tools, password and userdata scripts).

- Xen - install xentools
- VMware - install vmware tools
- KVM

Network Creation by CS admin

1. Create network offering without DHCP, DNS service.

a. UserData service is NOT selected:

Create empty network offering with no service selected. Only DHCP, DNS services are provided by external servers.

Metadata - information is included in the config drive

Userdata, vm password, ssh key - If these are passed then included in the config drive with user data service.

2. Create Advanced Zone shared network with above offering. While creating network, provide VLAN ID and IP range .

The IP range given here will NOT be used by CS for assigning IPs to nics. This IP range should match with the

DHCP configured range.

CS User/Admin

Template creation

Create templates for the UserVMs.
Windows template needs to be installed with PV drivers.
For Vmware templates, VMware tools needs to be installed.
If user data and password services are needed, then the templates needs to be configured with UserData and password scripts.
TODO: Scripts locations will be updated later similar to the existing password scripts publishing mechanism.
TODO: Update with the script paths inside the template.

VM deploy

Deploy VM in the shared network.
For this shared network, router won't be created as there are no services offered by the router.
UserVm is deployed and running.
UserVM ip configuration is not done by the cloudstack.

Workflow for the VM DHCP ip address.

Once the VM boot up successfully, VM template is configured to send the DHCP request.
User VM got the IP successfully from the external DHCP server.
CS retrieves the ip from the VM and update in the DB nics table.
CS logs about the VM ip updation in DB.
listVms API response shows the IP address which is retrieved.
VM not got the IP from the DHCP server because of some reason.
In this case CS is not going to update the nics table 'ip4_address' column.
ip4_address column shows null.
An event is generated for the VM IP address update.
VM IP info can be obtained from the listVirtualMachines/listNics API response.

Note: VM ip address information updating into DB will start after the VM enteres into running state.

Config drive

Config drive is an ISO attached to VM with files containing data includes userdata, metadata, password and ssh key.

On vm starting cloudstack creates an ISO with the vm data files and attach this iso to the VM in read only.

Once the VM boot up the iso can be mounted inside the VM. Using scripts the data inside the VM will be accessed.

For each VM an ISO will be created. One user VM can't have access to other user VM config drive ISO.

Config drive ISO device is created by-label. So the device and device number can be attached as below.

The default label is 'config' and it can be updated from the global settings "vm.configdrive.label".

Note: If "vm.configdrive.label" updated, make sure the same name is used in the user vm config drive scripts.

Device:

/dev/disk/by-label/config

DeviceId: Device can be found using blkid command

#blkid -t LABEL='config'
/dev/xvdb: LABEL="config" TYPE="iso9660"

Note:

If your guest operating system does not use udev, the /dev/disk/by-label directory is not present.

You can use the blkid command to identify the block device that corresponds to the configuration drive.

#mount /dev/disk/by-label/config /mnt/
mount: block device /dev/disk/by-label/config is write-protected, mounting read-only
[root@localhost ~]# cd /mnt/
[root@localhost mnt]# ls
cloudstack

Config drive contents:

[root@localhost mnt]# ls -lR
.:
total 1474
dr-xr-xr-x 1 root root 2048 Mar 10 07:13 cloudstack
./cloudstack:

total 4
dr-xr-xr-x 1 root root 2048 Mar 10 07:13 metadata
dr-xr-xr-x 1 root root 2048 Mar 10 07:13 password

./cloudstack/metadata:
total 3
-r-xr-xr-x 1 root root 5 Mar 10 10:23 availability_zone.txt
-r-xr-xr-x 1 root root 49 Mar 10 10:23 cloud_identifier.txt
-r-xr-xr-x 1 root root 9 Mar 10 10:23 instance_id.txt
-r-xr-xr-x 1 root root 9 Mar 10 10:23 local_hostname.txt
-r-xr-xr-x 1 root root 14 Mar 10 10:23 service_offering.txt
-r-xr-xr-x 1 root root 2 Mar 10 10:23 vm_id.txt

./cloudstack/password:
total 1
-r-xr-xr-x 1 root root 14 Mar 10 10:23 vm_password.txt
[root@localhost mnt]#

Reading UserData Inside the VM

CS bundles UserData, metadata,ssh key and password into an ISO. This ISO is attached to the user VM during VM deployment process as a ConfigDrive.

Linux VM

CS provides the script which needs to be placed in the Vm template.

This script mounts the ISO inside the user Vm. Script can read UserData file contents from the mount path.

Windows VM

Once the windows VM is up and running ISO is mounted automatically. User can browse attached ISO inside the VM.

From this drive, user can access the UserData.

Setting VM password

Linux/windows VM

The password enabled template with the script inside the template, will automatically sets the VM password.

Note: The public location of the scripts for userdata/password are provided for download.

Troubleshooting

VM ip address is not shown in the vm nic page. Below can be the reasons
.
a. VM got the ip but CS failed to retrive/update the CS DB.
    Look at the support matrix for wether the hypervisor/VM support the CS reading vm ip address.
    Look into CS logs for why CS failed to retrive or update the DB.
b. Make sure required tools are installed in the VM.
c. VM did not get the ip address from the DHCP server. So CS failed to retrieved it.
    Look at the VM console and trouble shoot DHCP server/client
VM password not set for password enabled template.
a. Check the support matrix for the hypervisor/vm support for password
a. Make sure the required tools for config drive got installed in the VM template.
b. Check the CS logs wether MS sent the userdata details to VM config drive.
c. There can be issue with password script inside the VM, Check the vm logs and run it in the debug mode to see failures.
Failed to read VM userdata in the VM.
a. Check the support matrix for the hypervisor/vm support for userdata
b. Make sure the required tools for config drive got installed in the VM template.

c. Check the CS logs wether MS sent the password details to VM config drive.

d. There can be issue with userdata script inside the VM if it is linux vm, Check the vm logs and run it in the debug mode to see failures.

Design

Network life cycle

When user launches a VM in admin created shared network, network get setup in the host.

As part of the VM deployment network creation process router won't be created for this network.

All the assumptions of dependency of the VR for the network is loosed.

Restart network basically a dummy operation because there are no rules setup

Network scavenger for this network is inactive to shutdown the router when there are no VMs in the network.

Getting DHCP IP from the VM by CS

1. Retrieve the IP from the Vm and update in the CS DB.

a. VmWare

With the help of vmware tools, Vm IP will be read from the information by the hypervisor

Note: Required vmware tools needs to be installed in Vm.

b. XenServer

Retrieve the VM IP from XenServer using the below command by passing the VM name.

#xe vm-list name-label=<vmname> params=networks

The output is parsed to get the vm ip address.

Note: Windows VM requires PV drivers to be installed to get the VM IP.

c. KVM

Use the guestfish command in KVM host to read the VM dhcp lease and parse it get the VM IP address.

#guestfish --ro -i -d i-2-21-VM download /var/lib/dhclient/dhclient-eth0.leases -

Note: Reading IP using guestfish command works for only Linux guest VMs.

ConfigDrive

Config drive contains the user VM related userdata, metadata, ssh key and password files. These files are packed in an ISO file.

During the VM start, ISO is attached to VM. The attached ISO can be accessed as a ConfigDrive within the VM.

MS creates the ISO and copies it to hypervisor (xen, kvm) with <vmName>.iso.

In case of VMware, MS creates the ISO and copies it to VM folder on primary storage. In case multiple primary storages exist, one with ROOT volume would be chosen to store the config drive ISO.

Xenserver:

The iso get copied into the path /opt/xensource/packages/iso/configdrive_iso/

A new SR with name 'ConfigDriveISOs' get created in the xenserver.

Config drive iso VBD get attached to vm.

Example:

#ls /opt/xensource/packages/iso/configdrive_iso/
i-2-324-QA.iso i-2-327-QA.iso i-2-328-QA.iso i-2-329-QA.iso i-2-330-QA.iso

KVM:

The iso get copied into path /var/libvirt/images

VMware:

The config drive iso is copied to path "[<Name of ROOT volume's datastore>] <UserInstance-internal-name>/"

Config drive is created on management server in path /tmp/<VM-NAME>/configdrive/ and then copied over to ESXi datastore (cloudstack primary storage).

When user instance gets destroyed, config drive iso file would be deleted as well.

Config drive iso would be created on 2nd cd/dvd drive of user instance, such that any other iso image or vmware tools get mounted on 1st cd/dvd drive. This means existing functionality of supporting 1 cd/dvd drive is still supported. Inside user instance the iso image would be available as "/dev/hdb". User can run command "mount /dev/hdb /configdrive" to mount the config drive iso over mount point /configdrive

Example datastore path : [aad39d84d3c63d3988bd3ffe8e2ffc83] i-2-3-VM/i-2-3-VM.iso

ConfigDrive across the VM life cycle

When VM is destroyed, ISO from the hypervisor get deleted.
When user VM password is reset, updated ISO with password file get copied into hypervisor.
When VM is migrated to new new host, the ISO is removed from source host and copied into new host.

UserData, metadata, ssh key

Shared network without DHCP, DNS service
CS writes userdata in key/value pair format to userdata.txt file. This file will be included in the ISO.

Password

If the template is password enabled, then password string for the VM is placed in passowrd.txt file and it is included in the ISO.

Accessing data in the ConfigDrive

Accessing ConfigDrive depends on the guest Vm OS.

UserData, metadata, ssh key

Windows VM
For windows VM the ISO is mounted automatically. So user can browse through the files in the ConfigDrive.
Linux VM
For linux based VM cloudstack provides the script config-drive-read.sh. It needs to be placed in the userVm.
This script mounts the ISO, user can access the files in the drive after it is mounted.

Password

Linux/windows VM:
CloudStack provides the script for setting the password.
cloud-set-vm-password.sh script will read the password file and update the root password of the VM during boot.

Fetching VM ip address and Updating CS DB

A new concurrent hash map with nicId as key and vmId and count as values get created. It get initialised with running vms of shared network on management server start.

On VM start, vm details get added into Map and on stop VM details get deleted from the Map. The default value for the count is 10.

This map contains the list of VMs to which the vm ip address fetch is needed.

A new task VmIpFetchTask get scheduled in an interval of 3 minutes (configurable from global settings) by MS for getting vm ip address from the map.

Polling task VmIpFetchTask loop through the nic ids in map and create a thread from the thread pool for each VM for getting vm ip address from the hypervisor.

The thread pool default maximum size is 10, it can be configurable from the global settings.

Thread will send command 'GetVmIpAddressCommand' to the hypervisor and get the answer.

Command answer result is true:

On command successful answer will contains VM ip address. This ip address get updated into the 'nics' table 'ip4_address' column.

nic id of the vm get deleted from the map and it will not tried for vm ip retrieval.

Command answer result is false:

On failed to get vm ip address the count (number of trials) in the map get decremented.

VmIpFetchTask will remove the nic entries of the vms from the map for which count is zero.

Once the count is zero the vms are no more tried for fetching ip address.

Configuration values

The VMIP read thread tries for reading ip from in interval of 180 seconds until it gets the ip.

externaldhcp.vmip.retrieval.interval = 180 seconds

The VMIP read thread tries max of 10 times incase it see failures.

externaldhcp.vmip.max.retry = 10

Number of threads for fetching vms ip address

externaldhcp.vmipFetch.threadPool.max = 10

polling in cluster of management servers

Selecting MS to poll:

Using global lock MS server is selected to own the threads.

The polling thread will be running in one MS. The thread creates the tasks to fetch the VM ip address.

ISO creation:

ISO will be created in the MS with the userdata and password files.

mkisofs binaries should be available in the MS

Support Matrix:

The following matrix lists the support of user \\data and password services.

Hypervisor/GuestVM OS	Windows VM	Linux VM
Xen	Userdata password ssh key	Userdata Password sshkey
KVM	Userdata password ssh key	Userdata Password ssh key
VmWare	Userdata password ssh key	Userdata Password ssh key

Hypervisor/GuestVM OS

Windows VM

Linux VM

Xen

Userdata

password

ssh key

Userdata

Password

sshkey

KVM

Userdata

password

ssh key

Userdata

Password

ssh key

VmWare

Userdata

password

ssh key

Userdata

Password

ssh key

The following matrix lists support for retrieving IP of the VM

	Windows VM	Linux VM
Xen	Yes	Yes
KVM	Yes	Yes
VmWare	Yes	Yes

Out of Scope

HyperV and Baremetal hypervisor specific changes are out of scope for this release

Known Issues

In KVM, for windows VM, via virt-win-reg which is part of libguestfs
In KVM, reading IP from linux vm using guestfish is very slow.
The VM name, VM IP has to be updated in DNS server by DNS admin outside of cloudstack.
If DHCP IPs are exhausted in DHCP server, VM will not get IP.
VMs created using the ISO will not have the tools installed. So CS fails to retrieve ip from this VM.
If hypervisor (all three hypervisors) is not able read VM IP info from specific guest VM, then the VM IP details are not shown in the listVms API

Usage

No impact.

Security

No impact.

API Changes

No changes

DB Changes

No DB changes.

Other Features affected by this feature

NIC secondary IP address
listVms API may not contain User Vm IP when unavailable

Hypervisors supported

XenServer
KVM
VmWare

Out of scope

Hyperv
Baremetal
LXC

UI Flow

No changes

Upgrade

There are no upgrade changes. This feature works with fresh deployments.

APPENDIX

Command references:

KVM:

Reading VM file from kvm host

#guestfish --ro -i -d i-2-21-VM download /var/lib/dhclient/dhclient-eth0.leases -

lease

interface "eth0";

fixed-address 10.1.1.210;

filename "pxelinux.0";

option subnet-mask 255.255.255.0;

option routers 10.1.1.1;

option dhcp-lease-time 4294967295;

option dhcp-message-type 5;

option domain-name-servers 10.1.1.1,4.2.2.2;

option dhcp-server-identifier 10.1.1.1;

option broadcast-address 10.1.1.255;

option host-name "VM-c5e5b415-4407-4e03-9c40-7210eb5f5e0b";

option domain-name "cs2cloud.internal";

renew 0 2082/11/22 09:51:45;

rebind 0 2133/12/6 18:17:14;

expire 5 2150/12/11 13:05:52;

Linux:

#virt-cat i-2-7-VM /var/lib/dhclient/dhclient-eth0.leases | tail -16 | grep 'fixed-address' | awk 'print $2' | sed -e 's/;//'

Windows:

Get Network GUID:

# virt-win-reg i-11-223-VM "HKLM\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\NetworkCards\\2" ServiceName

Get DHCP IPAddress

# virt-win-reg i-11-223-VM "HKEY_LOCAL_MACHINE\\SYSTEM\\ControlSet001\\Services\\Tcpip\\Parameters\\Interfaces\\A2E805D4-9CCF-4EA9-8832-17611F5D3FA8" DhcpIPAddress

2. Xen:

#xe vm-list name-label=i-2-24-QA params=networks

networks (MRO) : 0/ip: 10.147.47.100

The output is parsed to get the vm ip address.

3. Currently cloudstack is passing boot args for the system vm and read from the cloud-early-config: get_boot_params()

4. Widows server, client - Updating DNS entry (vm name, vmip got from the vm list response) can be done using self-register (Dynamic update)

Linux DNS server, client - DNS admin has to update (nsupdate) the DNS server with VM details.

External DNS

1. For windows VM and windows DNS server, DNS record can be updated using Dynamic update.
  Understanding Dynamic Update | Microsoft Docs
2. For other DNS servers, DNS record can be updated using 'nsupdate'
  http://linuxcommand.org/man_pages/nsupdate8.htm

List of Items to Document

Setting up dhcp and dns server external to CloudPlatform before provisioning the network
For vmware , templates must have vmware tools/pv drivers installed for IP Addres retrieval to work.
We need to document the procedure on how to prepare the templates with required tools
VM ip will not be available immediately after VM entered into running state.

以上是关于DHCP/DNS offload and Config drive support for adv shared network的主要内容，如果未能解决你的问题，请参考以下文章

Mellanox CX4 offload 卸载功能介绍

DHCP+DNS+WWW

DHCP + DNS + WEB综合架构的搭建

dhcp ,DNS 的配置

软考(23)-Linux-DHCP/DNS