DTC-Xen Installation

DTC-Xen / Dom0 Howtos

DTC-Xen / DomU Howtos


DTC Howtos




Devel docs

Wiki - i18n

Wiki - Meta

Howto setup LVM over RAID1 under Debian in order to run Xen

This page is part 1/6 in our A server with DTC-Xen tutorial.
(Make sure you follow the step one after the other)

1. Why?

1.a. What is this document about?

This howto explains how to have a working setup using LVM and RAID1 ready to host Xen VPS.

1.b. Why do you need RAID?

RAID is something you HAVE to use if you care a bit about your customer's data. RAID in software is the cheapest way to do mirroring, and it makes it easy to fit in a 1U server: that will save on costs, and you will be able to give back the saved money to your customers.

1.b. RAID1/RAID5/RAID10? Spare disks?

RAID10 in n2 mode is also a good idea, as it speeds up access too, but it's more than twice faster to fail on one of the HDD. RAID5 has the highest fail rate, uses more CPU, and is slower than RAID10, but saves more usable space for you.

Only RAID1 is described here. If you are using RAID10 or RAID5, keep the root filesystem (eg: the /) on a RAID1 on 2 disks only (even using hardware RAID), as this is the safest way (other types of RAID are using more disks, so will fail faster, and you really don't want to lose your root filesystem). Spare disks are also a good idea if you can afford it.

1.c. What type of emulated disk can a VPS boot from?

When using Xen, all of your VPS will have to use a separate disk. There are 3 ways to give a disk to a Xen VPS. The first one (which is the most commonly (mis)used) is a file loopback. This is a very poor setup, as it's slow, and as the file-loopback descriptors are limited in numbers in the kernel (even if that number can be set to a higher value, it's not something you want to do).

The second way is to use a NFS server. This is a very good idea, as with a NFS server you will be able to do live migration of your VMs from one physical server to another. But this is a very expensive setup, and you will have to use a gigabit LAN network with many servers.

The last way is the one we use: we use LVM to give an access to a physical partition of the server. LVM allows resize of the partition on-the-fly (you will just need to shutdown the server for a very short time to do it), and it's as fast as a normal partition. Moreover, LVM partitions are not limited in number.

1.d. What's the thing we want to do here?

You will need some LVM to give a (fast, preferably on RAID10) physical partition to your VPS. But Linux doesn't know how to boot on LVM, so we need to use a small RAID1 partition on which you will boot.

Our setup uses 1GB / (root filesystem) partition, so it's more easy to understand how it works (i.e: the /boot is still in /boot and not in / at the grub stage). Some might prefer to use a separate /boot partition that will be able to be even smaller (only few megs for this /boot partition would be enough). Both setup work, it's more a mater of tastes. The issue with a separate /boot partition, is that you don't have access to the recovery tools when you do a single user boot, so I don't like it. Doing the early boot process on a partition that includes /bin and /lib is always safer.

Let's start the tutorial now!

2. Booting the install CD

We always use the Debian network installer, but a normal full CD or DVD is ok.

Etch Notes: First, boot the installer media, type expert, then hit enter at the prompt.

Lenny Notes: First, boot the installer media and start the graphical or text setup (it doesn't mater), but make sure to choose the expert mode.

Do the beginning of the setup as usual (language, region, keyboard, homename, domain, mirror), until you get to the partition setup screen.

3. Creating the physical partitions

You should now create the partitions for RAID1. First, create a small partition at the beginning of each disk. They will be used for the / (root partition), and it doesn't have to be bigger than 800 MB, 1GB is ok too and safer. Set the boot flag on each disk for this partition, and select "use as", then "physical device for RAID".

Next select the free space on each disk, and create a partition that should take all the remaining space but 5BG, and then select to use it as a RAID device like you just did for the (small) first partition. You should leave 5GB unused for 2 reasons. First, the end of the disks are often where you find bad sectors. Second, when a HDD needs to be replaced (because it's broken for example), it's hard to find exactly the same type. If you buy a replacement disk that is few MB smaller, and can't create a RAID1 partition of exactly the same size, then you are stuck.

Now you should have 2 disks, each with 1 small partition of 800 MB with the boot flag on, and a big one with all the rest of the space but 5GB. All partitions should be marked to be used for RAID1. It's time to go in "finish and write to disk", and then reboot (yes, without the partition being mounted!). Linux loads the partition tables ONLY at boot time, so you really need to reboot. Note that this seems to be true only for IDE devices, newer hardware using SATA or SCSI don't seem to need a reboot. (Etch Note: I don't think the reboot is necessary with the etch installer.)

4. Creating 2 RAID1 devices

Restart the installer, do like you did before, and reach the partitioning utility. You should now see the partitions you created.

Go in the RAID utility menu of the partition screen (on top of the menu). Select the option to create RAID device, select RAID1, say you want to use 2 disks, and no spare disk (so: create -> raid1 -> 2 -> 1). Select the first and the third entry (the small boot partition on each disks). Do it again with the 2 remaining partitions (the big one) on each disk. Leave the RAID1 menu. It should start to copy the first disk to the 2nd (RAID1 is done at device level, not file system, it never knows if the disks are empty, in fact it doesn't care at all about the content of the data itself).

Now, back at the partman screen, you should see the 2 partitions on each disk, plus 2 RAID1 devices.

5. Creating the LVM devices

5.a. Creating the volume group (VG)

Select the 2nd RAID1 device (the one using the big partition). Say you want to use it as a physical volume for LVM device (a volume group, then go back to the main screen.

It's now time to create the LVM partitions for your xen management operating system (the domO).

Go on the top entry of the partition screen (partman), and go to "Configure the Logical Volume Manager". Pick "Create volume group".

It will prompt you for a volume group name. At GPLHost, we call it "nodeXXXXvg0", with XXXX being the number of the node. You can just "hostnamevg0" or "alphavg0" if your Xen server hostname will be Best is to use a different volume name for each of your data center server, otherwise you are risking to have a name conflict when you pull the HDD from one server and put it in another one in order to repair. As of DTC-Xen 0.5.0, the shell scripts are using whatever is the last entry when calling vgdisplay from the shell (or what's configured in /etc/dtc-xen/dtc-xen.conf) in order to determine what volume group to use for provisioning new VPSes.

Next, select the (only) available device you just created that is in fact the big RAID1 device. That should be the only device that you can choose anyway (if you have only one big volume group).

You have now a volume group where you can create some LVM partitions.

5.b. Creating the logical partitions (LVs)

After the first volume group is created, you can create logical partitions. Now go in the "logical volume" creation. Create 5 logical volumes. It will ask you for a name, just call it "usr" for the /usr partition and so on. A partition scheme like this one should be ok (proven to work in production):

    device                      path    name    size
    /dev/md0                     /       rootfs    1GB <--- at least 1GB
    /dev/mapper/nodeXXXXvg0-usr  /usr    usr      10GB <--- Need space to host /usr/share/
    /dev/mapper/nodeXXXXvg0-var  /var    var      12GB <--- Your actual total RAM plus 4GB
    /dev/mapper/nodeXXXXvg0-tmp  /tmp    tmp     512MB <--- Avoids filling the /
    /dev/mapper/nodeXXXXvg0-root /root   root    512MB <--- Avoids filling the /
    /dev/mapper/nodeXXXXvg0-swap swap    swap    512MB

Notes for the size of /var:

Your /var partition will be used to store the memory of all of your VMs when the dom0 shuts down (with the save option of Xen). So you NEED to have enough space to store all of the physical memory of your server, if you like this option. Let's say you are running with 4GB of RAM, then 6 or 8 GB for your /var seems reasonable. However, in /etc/default/xendomains there is a XENDOMAINS_SAVE= that tells where to save the VMs. Note as well that the Debian Lenny save feature is currently broken, and you have to put an empty string there (eg: XENDOMAINS_SAVE=""), otherwise you will have some issue with the clock system of the VPS when they are restored.

Notes for the size of /usr:

This will be used to host your VPS images. Currently all the images have a total of a bit less than 6GB, you need at least that plus more the rest of /usr (5GB for /usr is enough so you can compile Xen in /usr/src). Using a separate LVM for /usr/share/dtc-xen-app is also an option.

This should leave most of the LVM space free to be used by your all your VPS.

6. Selecting type (ext3) and mount points

Exit the LVM utility, and go back to the main partition screen. You should now see the 2 raid1 device, the 5 logical volumes you just made, and the 2 physical partitions on each disks. It's time for you to mount the partitions.

First, select the small RAID1 device. Say you want to use it as ext3, and that the mount point is / (the root partition), using ext3 fs.

Then select the "usr" logical volume, say you want to use it as ext3, and select the mount point. Do the same for /var, /tmp and /root (for /root you will need to type the full path of the mount point). Select the "swap" logical volume and say you want to use it as swap space.

7. You are good to go!

All your partitions are now created and the mount point is selected for all of them. Just go in "Finish and write change to disk", it will format them all and do the rest of the install as usual.

One last thing: the DI will warn you again that Linux doesn't load partition tables on the fly, but at boot time only. Just ignore the message this time: it's true it can't load the partition tables, but it knows about the changes you did with LVM and RAID1 arrays.

8. Important thing

When doing this, you HAVE to follow the order:

  • Create the RAID partitions on both disks
  • Reboot
  • Create the LVM physical volume
  • Create the LVM logical volumes
  • Mount the / on the first RAID1 device
  • Select the moint point for the LVM volumes

If you don't, you will screw everything.

You can now proceed to Upgrading to Xen kernel.

Page last modified on July 20, 2012, at 03:28 PM EST