TR Forums

Hi all,

Like the subject says I'm interested in using Ubuntu as storage for my Xen infrastructure. The server that is currently hosting the VM's is running a supported version of Nexenta. My experience with it has been less than stellar. I'm sure it's a great product, but I don't have the time to invest in getting up to speed on the nuances of administering a Nexenta appliance. I will be re purposing a Dell R510 for hosting the virtual machines.

Anyone here have any experience using Linux as the back end storage for XenServer? Any recommendations or best practices that anyone knows of that would help me get this project started? The R510 has 2 seperate sets of disks. 11x 1TB and the other 8x 1TB. It's also got buckets of RAM that was being used for the ZFS arc cache that's likely going to go unused in this new setup. Does anyone have any experience in utilizing that in some useful way for this type of build?

I am also open to other solutions. I am going to look at FreeNas and other versions of Linux, but a big factor in chosing Ubuntu is it allows for the OpenManage tools to be installed. If I were to forgo those tools again (like the previous admin did to use Nexenta) it'd have to be a pretty compelling argument.

I don't have experience with Xen, but Ubuntu works great as a host for VirtualBox VMs. Use one of the LTS releases (either 12.04 or the upcoming 14.04); stay away from the non-LTS releases for any sort of production server.

Debian or CentOS would also be good choices, IMO.

With the announcement that Red Hat made today I'm switching gears to Cent. Digging further into it Ubuntu was kind of unofficially officially supported. Dell has a community that makes sure the tools work on it, but they also don't take any ownership if they don't. Red Hat, however, is officially officially supported...so I'm using CentOS =P

I've also settled on using NFS for the datastores. I'm digging into properly bonding NIC's for this job (which was one of the nightmares of Nexenta and Xen with iSCSI) right now. I'm waiting on a refurb UCS-51 right now to get my test box up and running. Once I do (and if nobody has a better suggestion) I'll post back with some results of how my Xen environment performs with some various solutions.

LaChupacabra wrote:

With the announcement that Red Hat made today...

What announcement?

Cent right now does a far better job at supporting the Xen host kernel than Ubuntu. However, unless you are going to use Cent as your Dom0 there will be little to gain from it. NFS servers done on just about any Linux platform are pretty universal. The packages / config files might install in different locations. The defaults might be slightly different but there's not much that's going to change your experience. Now the file system you choose and the bandwidth you have between your host and the datastore, those are some of the things that can make massive differences.

If you are going to homebrew it then MDADM+LVM2 all of the way. I would have said Solaris but without the support packages (available only if you buy a support license..although it's not expensive) you are pretty much on your own. There is ZFSonLinux which i use at home that you could try as well. It would be a good way to use all of that memory you have.

Buub wrote:

LaChupacabra wrote:

With the announcement that Red Hat made today...

What announcement?

Red Hat is collaborating with CentOS now. Not quite sure what to make of this myself.

It could be an attempt to counter to what Red Hat perceives as a threat from other "free" server OSes (Ubuntu Server and Debian). By bringing the free server distro that is most closely related to RHEL under the official Red Hat umbrella they may attract some of the people who would otherwise run a Debian derivative to the Red Hat camp.

I've also seen some comments to the effect that it is an attempt to shut down Oracle Linux. In effect, sending a message of "Why use Oracle's RHEL clone when you can use ours?"

There's also a potential downside for CentOS over the long term if Red Hat takes over critical responsibilities then loses interest.

kc77 wrote:

The defaults might be slightly different but there's not much that's going to change your experience. Now the file system you choose and the bandwidth you have between your host and the datastore, those are some of the things that can make massive differences.

Agreed. The reason behind settling on CentOS was about using the Dell Open Manage tools. Using a product like Nexenta or FreeNAS means running the build as the vendor sees fit to implement it. It's a forest for the trees situation. They will tell you if a drive is down, but they do not monitor things like the temperature sensors in the server. To me that is a big deal. Reliability is more important than performance. The tools also allow you to do things like live firmware updates. I haven't been able to check on the Nexenta appliance, but my feeling is the previous admin here didn't believe in firmware updates. In my previous life as a consultant I had 2 clients that lost whole servers due to disk corruption caused by a bug in Dell's backplane firmware. It was my job to clean that up. If their previous provider had simply applied the firmware updates the corruption never would have happened.

It sounds like you have some experience with Xen. I'm setting this up as shared storage for 2 hypervisors. Do you have any recommendations for file system and formatting of the drives? Again, reliability is paramount. But if swapping EXT3 for 4 or NFS for iSCSI will create speed gains it'd be silly to not explore them.

LaChupacabra wrote:

Agreed. The reason behind settling on CentOS was about using the Dell Open Manage tools. Using a product like Nexenta or FreeNAS means running the build as the vendor sees fit to implement it. It's a forest for the trees situation. They will tell you if a drive is down, but they do not monitor things like the temperature sensors in the server. To me that is a big deal. Reliability is more important than performance. The tools also allow you to do things like live firmware updates. I haven't been able to check on the Nexenta appliance, but my feeling is the previous admin here didn't believe in firmware updates. In my previous life as a consultant I had 2 clients that lost whole servers due to disk corruption caused by a bug in Dell's backplane firmware. It was my job to clean that up. If their previous provider had simply applied the firmware updates the corruption never would have happened.

It sounds like you have some experience with Xen. I'm setting this up as shared storage for 2 hypervisors. Do you have any recommendations for file system and formatting of the drives? Again, reliability is paramount. But if swapping EXT3 for 4 or NFS for iSCSI will create speed gains it'd be silly to not explore them.

Do you have extra money to spend?

Is the datastore / storage appliance meant to service other things than just Xen within your data center?

I would seriously suggest either using RHEL so that you have support for when **** goes south or use CentOS/ScientificLinux which are effectively the same thing without the support.

dltd.

kc77 wrote:

Do you have extra money to spend?

Is the datastore / storage appliance meant to service other things than just Xen within your data center?

Not really, part of the problem is I work at a non profit. I'm dealing with the sunk costs of the systems that were put in a year or two ago. That's why re purposing existing hardware is fine, but investing in new software is not.

It will be used primarily as storage for Xen. One of the above volumes is going to be dedicated to that. The other volume will be used as a backup target for our virtaul backups and half a dozen things that just need bulk storage accessible via nfs or samba. I have a second rack in an ISP's datacenter close to our offices with a similar setup. The servers are not replicated directly (didn't have the budget) but the virtual backups do replicate, as well as the datastores that contain those half dozen archives. There are 2 Xen hypervisors there that run various things like XenApp and our websites. We've got a tunnel running between our main offices and that second location.

I think that's everything relevant to give a good idea of what this needs to do. I've got CentOS 6.5 up and running on a test box and am starting to carve that up to do what this needs to do. I'm happy to answer any other questions people may have! I'm also super grateful for the help people have already offered =)

clone wrote:

apparently Ubuntu is bloated with spyware.

http://www.youtube.com/watch?v=CP8CNp-vksc

https://www.eff.org/deeplinks/2012/10/p ... data-leaks

We've had this discussion already when the original story broke. Yes, Canonical royally f**ked up by enabling this feature by default. But it is easy to disable, and isn't installed at all if you use one of the alternate desktops (Kubuntu, Xubuntu, or Lubuntu). It also isn't installed if you use the Server version (which the OP would presumably be doing), so it is a non-issue unless he decides to install the Unity desktop on his server for some bizarre reason. (Xubuntu or Lubuntu would be more sensible choices for a server box if he insists on having a GUI available.)

LaChupacabra wrote:

I haven't been able to check on the Nexenta appliance, but my feeling is the previous admin here didn't believe in firmware updates. In my previous life as a consultant I had 2 clients that lost whole servers due to disk corruption caused by a bug in Dell's backplane firmware. It was my job to clean that up. If their previous provider had simply applied the firmware updates the corruption never would have happened.

Though sometimes applying firmware makes things worse:

http://community.spiceworks.com/topic/2 ... ned-drives
http://h30499.www3.hp.com/t5/ProLiant-S ... -p/4654411

I'm in the pro-firmware camp but man, you can't even rely on vendors to test this stuff, what hope do you have on your own? I'm glad I didn't get bitten by that on a production server.

Scrotos wrote:

Though sometimes applying firmware makes things worse:

http://community.spiceworks.com/topic/2 ... ned-drives
http://h30499.www3.hp.com/t5/ProLiant-S ... -p/4654411

I'm in the pro-firmware camp but man, you can't even rely on vendors to test this stuff, what hope do you have on your own? I'm glad I didn't get bitten by that on a production server.

Reading through stories like that always makes me cringe. There was an issue with the HP firwmare updater that if you attempted to pull updates from the HP ftp site (not even apply them, but just query the ftp site through the supplied tools) that it would reboot certain servers. Fortunately for me the first time I was affected by this was on another client's vCenter. After reading through the spiceworks case I'm amazed that guy applied the firmware update before seeing if the replacement backplane (running the older firmware) would fix the issue. Theoretically you should be able to swap those without affecting your RAID. When doing firmware updates I stick to the quarterlies, and usually wait for a month or two after they've been released.

Lots of people run VM servers using NFS storage. Just try to get at least a dedicated 10Gb storage network with switches and NICs that support jumbo frames.

As for firmware, don't make change for the sake of change. Be very conservative. My rule is: bugs affecting me and security holes only.

just brew it! wrote:

Buub wrote:

LaChupacabra wrote:

With the announcement that Red Hat made today...

What announcement?

Red Hat is collaborating with CentOS now. Not quite sure what to make of this myself.

It could be an attempt to counter to what Red Hat perceives as a threat from other "free" server OSes (Ubuntu Server and Debian). By bringing the free server distro that is most closely related to RHEL under the official Red Hat umbrella they may attract some of the people who would otherwise run a Debian derivative to the Red Hat camp.

I've also seen some comments to the effect that it is an attempt to shut down Oracle Linux. In effect, sending a message of "Why use Oracle's RHEL clone when you can use ours?"

There's also a potential downside for CentOS over the long term if Red Hat takes over critical responsibilities then loses interest.

Red Hat hired some of the CentOS devs, and while they have effectively "acquired" CentOS, it's not any different then how Fedora works.

RH does this quite often. They'll hire devs that work on a particular project as a way to ensure the viability of the project and as a way to support FOSS software. The devs get to keep their repos, and they get to keep their code. They just get paid to work on their software. It's really very benign.

A lot of devs creating newer technologies have been using Ubuntu as the base, and RH would like to change that. My understanding is CentOS is going to be to RHEL what Ubuntu Server is to Debian, except being essentially interchangeable with RHEL. It's going to split the difference between Fedora and RHEL. RHEL will still be the upstream project for CentOS, but CentOS will have access to software with higher version numbers and newer technologies that would be excluded from RHEL because they haven't been fully vetted by RH's engineering teams.

LaChupacabra wrote:

kc77 wrote:

Do you have extra money to spend?

Is the datastore / storage appliance meant to service other things than just Xen within your data center?

Not really, part of the problem is I work at a non profit. I'm dealing with the sunk costs of the systems that were put in a year or two ago. That's why re purposing existing hardware is fine, but investing in new software is not.

It will be used primarily as storage for Xen. One of the above volumes is going to be dedicated to that. The other volume will be used as a backup target for our virtaul backups and half a dozen things that just need bulk storage accessible via nfs or samba. I have a second rack in an ISP's datacenter close to our offices with a similar setup. The servers are not replicated directly (didn't have the budget) but the virtual backups do replicate, as well as the datastores that contain those half dozen archives. There are 2 Xen hypervisors there that run various things like XenApp and our websites. We've got a tunnel running between our main offices and that second location.

I think that's everything relevant to give a good idea of what this needs to do. I've got CentOS 6.5 up and running on a test box and am starting to carve that up to do what this needs to do. I'm happy to answer any other questions people may have! I'm also super grateful for the help people have already offered =)

OK well that makes it simple then. You want to serve out the VM's using NFS. For the file system Ext4 + LVM2+MDADM with RAID 10 and RAID 1 for the OS. For a very long time Ext3 was pretty much the go to. But Ext4 has a ton of features and it's just as stable now. Although I would update that switch when you can to move it to 10Gb. It's worth it and will make your VMs run extremely well. Make sure you adjust NFS properly and use aggressive caching on MDADM. That will help you use up the memory you have. It won't perform like ZFS but you'll be close enough, especially on a 1Gb network.

kc77 wrote:

For the file system Ext4 + LVM2+MDADM with RAID 10 and RAID 1 for the OS.

That's pretty close to my standard setup for servers these days. OS is 2 SSDs connected to the onboard SATA in software RAID 1, and the storage array is SAS hard drives drives in RAID 10 via a RAID card.

XFS is another well tested filesystem that can handle large amounts of data.

It's been a while since I posted an update. I had a birthday (hooray me!), some time off, and have been dealing with a lot of user issues the past week. Here's where I'm at:

I've got a test system setup. Xenserver 6.2 as the host with CentOS 6.5 providing the storage. Just a simple NFS store on a hardware backed Raid 1. Nothing fancy. A Catalyst 2950 is providing the switching. I'm using LACP (configured on the Storage server, Catalyst and Xen correctly, the Catalyst is showing all the connections bonded). All peachy, right? Nope.

Disk IO is still running at the speed of a single network connection. Checking things out on the Storage server shows that all incoming data is being received on one interface and all outgoing data is being sent out on the other. I can pull a network cable while the test servers are running and everything fails over correctly. So my question is why both interfaces aren't being used for sending and receiving?

The storage system for the XenServer is currently on it's own gigabit connection, with a dedicated network seperate from management. The Storage server has 2 bonded interfaces on a 100m/b connection. Fastest transfers I've seen is 15 MB/s. Xen shows the connections running a 200m/b. I've also configured the XenServer to use the management network on a bonded (200m/b) interface with the Storage server on the management network. Same results.

After an hour on phone support with Citrix they don't know why it's happening either. I have a couple of things left to try, but it's looking like I'll be going back to iSCSI for the storage system. I'll give another update when there is something to say.

Send over your NFS settings. Basically I'm thinking bonding two 100Mb links + old crusty 2950 + poorly configured NFS + Raid 1 isn't going to set the world on fire.

Have not been terribly impressed with Citrix. My previous company used Citrix CloudStack, and it was quirky. Worked for the most part, except occasionally when it didn't.

I've been FAR happier with VMware vCloud Director at my current company. And we have a HUGE VM cloud.