1. Why?
This howto explains how to have a working setup using LVM and RAID1 ready to host Xen VPS.
RAID is something you HAVE to use if you care a bit about your customer's data. RAID1 in software is the cheapest way to do mirroring, and it makes it easy to fit in a 1U server: that will save on costs, and you will be able to give back the saved money to your customers.
When using Xen, all of your VPS will have to use a separate disk. There are 3 ways to give a disk to a Xen VPS. The first one (which is the most commonly (miss)used) is a file loopback. This is a very poor setup, as it's slow, and as the file-loopback descriptors are limited in numbers in the kernel (even if that number can be set to a higher value, it's not something you want to do). The second way is to use a NFS server. This is a very good idea, as with a NFS server you will be able to do live migration of your VMs from one physical server to another. But this is a very expensive setup, and you will have to use a gigabit LAN network with many servers. The last way is the one we use: we use LVM to give an access to a physical partition of the server. LVM allow resize of the partition on-the-fly (you will just need to shutdown the server for a very short time to do it), and it's as fast as a normal partition. Moreover, LVM partitions are not limited in number.
You will need LVM to give a (fast) physical partition to your VPS, but Linux don't know how to boot on LVM, so you will need to use a small RAID1 partition on which you will boot. Our setup uses a 500 MB to 1GB / (root) partition, so it's more easy to understand how it works (i.e: the /boot is still in /boot and not in / at the grub stage), but some might prefer to use a separate /boot partition that will be able to be even smaller (only few megs for this /boot partition would be enough). Both setup work, it's more a mater of tastes.
Let's start the tutorial now!
2. Booting
We always use the Debian network installer, but a normal full CD or DVD is ok.
Sarge Notes:
If you use amd64 arch and newer hardware, we advice you to use the daily built debian installer (with the latest kernel).
First, boot the Debian Installer (DI). On the syslinux prompt, you need to type "expert26" in order to boot with kernel 2.6 and have the installer ask more questions.
Do the beginning of the setup as usual (keyboard, region, timezone, etc.), and don't forget to ask to load the RAID and the LVM modules. Then you should arrive at the partition setup screen.
Etch Notes:
First, boot the installer media and hit enter at the prompt. Do the beginning of the setup as usual (language, region, keyboard, homename, domain, mirror), until you get to the partition setup screen.
3. Creating the physical partitions
You should now create the partitions for RAID1. First, create a small partition at the beginning of each disks. They will be used for the / (root partition), and it doesn't have to be bigger than 800 MB. Set the boot flag on each disks for this partition, and select "use as", then "physical device for RAID".
Then select the free space on each disk, and create a partition that should take all the remaining space but 5BG, and then select to use them as RAID device like you just did for the (small) first partition. You should leave 5GB unused for 2 reasons. First, the end of the disks are often where you find bad sectors. Second, when a HDD needs to be replaced (because it's broken for example), it's hard to find exactly the same type. If you buy a replacement disk that is few MB smaller, and can't create a RAID1 partition of exactly the same size, then you are stucked.
Now you should have 2 disks, with each 1 small partition of 800 MB with the boot flag on, and a big one with all the rest of the space but 5GB. All partition marked to be used for RAID1. It's time to go in "finish and write to disk", and then reboot (yes, without the partition being mounted!). Linux loads the partition tables ONLY at boot time, so you really need to reboot. Note that this seems to be true only for IDE devices, newer hardware using SATA or SCSI don't seem to need a reboot. (Etch Note: I don't think the reboot is necessary with the etch installer.)
4. Creating 2 RAID1 devices
Restart the installer, do like you did before, and reach the partitioning utility. You should now see the partitions you created.
Go in the RAID utility menu of the partition screen (on top of the menu). Select the option to create RAID device, select RAID1, say you want to use 2 disks, and no spare disk (so: create -> raid1 -> 2 -> 1). Select the first and the third entry (the small boot partition on each disks). Do it again with the 2 remaining partition (that big one) on each disk. Leave the RAID1 menu. It should start to do the copy of the first disk to the 2nd (RAID1 is done at device level, not file system, it never knows if the disks are empty, in fact it doesn't care at all of the content of the datas itself).
Now, back at the partman screen, you should see the 2 partitions on each disks, plus 2 RAID1 devices.
5. Creating the LVM devices
Select the 2nd RAID1 device (the one using the big partition). Say you want to use it as a LVM device, then go back to the main screen.
It's now time to create the LVM partitions for your xen management operating system (the domO).
Sarge Notes:
Go on the top entry of the partition screen (partman), and go in the LVM manager. Go in the "volume group" management, and create the LVM array. It will normaly detect the LVM partition that you have set on the big raid device. It will prompt you for a name, call it "lvm1" as this is the name that will use DTC-Xen (otherwise you will have to do some more configuration later), and select the (only) available device you just created that is in fact the big RAID1 device.
Etch Notes:
Go on the top entry of the partition screen (partman), and go to "Configure the Logical Volume Manager". Pick "Create volume group". It will prompt you for a name, call it "lvm1", as this is the name that will use DTC-Xen (otherwise you will have to do some more configuration later), and select the (only) available device you just created that is in fact the big RAID1 device.
Exit the volume group creation menu, and now go in the "logical volume" creation. Create 5 logical volumes. It will ask you for a name, just call it "usr" for the /usr partition and so on. A partition scheme like this one should be ok:
name size
usr 3GB
var 1GB
tmp 400MB
root 400MB
swap 512MB
Notes for the size of /var:
Your /var partition will be used to store the memory of all of your VMs when the dom0 shuts down. So you NEED to have enough space to store all of the physical memory of your server. Let's say you are running with 4GB of RAM, then 6 or 8 GB for your /var seems reasonable.
This should leave most of the LVM space free to be used by your all your VPS. 5GB for /usr is enough so you can compile Xen in /usr/src.
6. Selecting type (ext3) and mount points
Exit the LVM utility, and go back to the main partition screen. You should now see the 2 raid1 device, the 5 logical volumes you did, and the 2 physical partitions on each disks. It's time for you to mount the partitions.
First, select the small RAID1 device. Say you want to use it as ext3, and that the mount point is / (the root partition), using ext3 fs.
Then select the "usr" logical volume say you want to use it as ext3, and select the mount point. Do the same for /var, /tmp and /root (for /root you will need to type the full path of the mount point). Select the "swap" logical volume and say you want to use it as swap space.
7. You are good to go!
All your partitions are now created and the mount point is selected for all of them. Just go in "Finish and write change to disk", it will format them all and do the rest of the install as usual.
One last thing: the DI will warn you again that Linux doesn't load partition tables on the fly, but at boot time only. Just ignore the message this time: it's true it can't load the partition tables, but it knows about the changes you did with LVM and RAID1 arrays.
8. Important thing
When doing this, you HAVE to follow the order:
- Create the RAID partitions on both disks
- Reboot
- Create the LVM physical volume
- Create the LVM logical volumes
- Mount the / on the first RAID1 device
- Select the moint point for the LVM volumes
If you don't, you will screw everything.
