Virtualization Server advice

Building a new system? Need help choosing between parts? Then step in and let our trained gerbils assist you.

Moderator: JustAnEngineer

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 12:19 pm

Scrotos: yep, $5000 is for hardware.

Coldfirex: Yeah, I'm probably going with Datacenter 2012. Still haven't decided on the hypervisor of choice yet. While I don't wish Microsoft any ill will per se, it's nice not to feel utterly dependent upon them. Plus, if I go with ESXi or XenServer, I can convert my existing VM host machine over to the same platform without needing to purchase an additional Windows Server license.

Shodanshok:

CPU: 1P will likely be sufficient, I won't complain if 2P is possible. As you said, I'm not much worried about clockspeed, but cores/threads are more=better. AMD or Intel I don't much care.

RAM: I'm aiming for 64 GB, but haven't gotten to looking at registered vs unregistered. My guess is that I'll opt for unregistered for $ savings.

I/O performance: hasn't been an issue for 5 VMs on a RAID 1 array (two 2TB disks).

Linux + KVM: I can give it a try. One of my requirements is ease of administration. VMware has proven itself easy enough. Xenserver looks about as simple. Hyper-V is anecdotally simpler than either of those. Realistically speaking, professional support needs to be an option so that if serious trouble is encountered I can call a tech even if it's a $200 phone call.

RAID: No, fake is not an option. Soft RAID only seems to be an option with Linux + KVM (or else VMware Workstation on Linux or Windows). So it seems like hardware RAID wins by default.

Build or buy: I've decided a custom build is not an option. It will be a system from HP, Dell, or Supermicro.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 12:22 pm

RAID:

I think it's going to be hardware RAID; I'm just going to assume that for the simplicity's sake for the moment, but regardless I'm 98% sure it'll be hardware. I've heard this from you guys:

RAID 6 or RAID 10.

Anyone have anything against RAID 1?

I know it becomes space-inefficient pretty quickly, but in terms of reliability and rebuild time, any red flags?
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 12:29 pm

flip-mode wrote:CPU: 1P will likely be sufficient, I won't complain if 2P is possible. As you said, I'm not much worried about clockspeed, but cores/threads are more=better. AMD or Intel I don't much care.

RAM: I'm aiming for 64 GB, but haven't gotten to looking at registered vs unregistered. My guess is that I'll opt for unregistered for $ savings.

I don't think you can run 64GB of unregistered in a 1P system. AFAIK unregistered RAM maxes out at 8 GB/DIMM and 2 DIMMs/channel.

Edit:

Looks like SuperMicro does make a 1P Opteron board with quad channel support. However, the specs still indicate that you need registered DIMMs if you want to go past 32GB. (As a guess, this is probably due to the PCB trace lengths required to reach the further DIMM sockets... or possibly a technical limitation of the Opteron IMC.)

FWIW our 2P Opteron build uses 64GB of registered RAM.

Edit 2:

The only reason to prefer RAID 10 over RAID 1 is performance.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37673
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 12:51 pm

Cool. Registered it is. I'll have to give the RAID setup some thought. I imagine that two 2TB drives in RAID 1 will give me all the storage space I need, more than enough, actually. The question then is all about performance. With less than 10 VMs doing no database and no file serving, just domain authentication and software license services for less that 20 desktops, I'm hoping RAID 1 will be sufficient. The EXSi machine I have that is running email, license server, an ubuntu FTP server, and a Revit server on Windows 2008 (Revit Server is by far the heaviest burden, uses half of the 16 GB of physical RAM) is doing quite with I/O and it's just two 2TB in RAID 1.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 1:27 pm

So, the unlicensed version of XenServer does not let me apply updates through the XenCenter. That's a bummer. It can only be done at the physical console.

Edit,
The XenServer update process isn't so bad. Download the patch, use WinSCP to transfer to the XenServer machine, run a couple commands. Worst part of it is that - according to the guide I followed - the update commands require the patch files to be registered with the system, each file is then given a UUID that is some ridiculous length, and then the actual update command requires the admin to type in that UUID. :-? It seems silly to me that I can't just type in the file name - and maybe I can... I didn't actually try to do that, I opted to follow the instruction instead. But it wasn't all that bad and the updates applied with lightning speed. The system needed to be rebooted after applying the updates, which is to be expected.

I haven't gotten round to creating an actual VM yet :lol: Also, I need to look into the process of migrating my VMware VMs to XenServer and see what is involved there.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 2:19 pm

Check out the Dell R515

PE R515 with up to 12 Hot Swap Hard Drives, LED
64GB Memory,(4x16GB) 1600MT/s, Dual Rank LVRDIMMs at Std Volt, for 2 Processors
2x AMD Opteron™ 4386, 3.1GHz, 8C, Turbo CORE, 8M L2/8M L3, 1600Mhz Max Mem
PERC H700 Integrated RAID Controller 512MB Cache,12HD
RAID 10 for PERC H200 and H700 Controllers, x8 and x12 Chassis
750 Watt Redundant Power Supply
(6) 600GB 15K RPM SAS 6Gbps 3.5in Hot-plug Hard Drive (1.8TB HDD Space)
3Yr Basic Hardware Warranty Repair: 5x10 HW-Only, 5x10 NBD Onsite

Sub-total $4,890.54

Make sure to call them, I'm pretty sure you could get that price out the door and maybe add on some more warranty.
You have to remember you don't want to purchase for what is good enough currently. Any business expects to grow and this will allow you to add more hard drive space and more memory easily, this might be good enough for quite some time as it is.

I first compared the R520 and R720 but the AMDs get you a lot more for your money it seems.
paco
Minister of Gerbil Affairs
 
Posts: 2081
Joined: Wed Jul 21, 2004 7:14 pm
Location: So Cal

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 2:30 pm

That is pretty impressive.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 2:33 pm

That's one of the carrots Citrix holds out for people who buy support. That's off set by getting VM migration though. Here is a link to the features of a XenServer subscription. (http://xenserver.org/get-support.html) Of course, the free version of vSphere isn't any better in this regard.

Citrix is running a $199 annual per socket subscription special right now. It's normally $500.

You can get support for KVM from Red Hat. You would need to buy a RHEL license with the correct number of guests.
RHEL 2-sockets with unlimited virtual guests $1,999

shodanshok wrote:VMWare also have a dinamically-controlled ballon driver to reclaim unused guest memory.


Xen 4.0+ has memory balloning, and memory sharing and memory paging are in the tech preview stage with the notes stating, "Preview, due to limited tools support. Hypervisor side in good shape."

Xen Release Features
http://wiki.xenproject.org/wiki/Xen_Release_Features
Flatland_Spider
Gerbil Elite
 
Posts: 832
Joined: Mon Sep 13, 2004 8:33 pm
Location: The 918/539

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 3:05 pm

flip-mode wrote:RAID:

I think it's going to be hardware RAID; I'm just going to assume that for the simplicity's sake for the moment, but regardless I'm 98% sure it'll be hardware. I've heard this from you guys:

RAID 6 or RAID 10.

Anyone have anything against RAID 1?

I know it becomes space-inefficient pretty quickly, but in terms of reliability and rebuild time, any red flags?


RAID 10/1+0 is just RAID 1 with more than 2 disks. A 4-disk RAID 1 is the same thing as a 4-disk RAID 10 in practice. They both lose 50% of their capacity but at least you get some striping action to give a speed boost.
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 3:30 pm

flip-mode wrote:RAID:

I think it's going to be hardware RAID; I'm just going to assume that for the simplicity's sake for the moment, but regardless I'm 98% sure it'll be hardware. I've heard this from you guys:

RAID 6 or RAID 10.

Anyone have anything against RAID 1?

I know it becomes space-inefficient pretty quickly, but in terms of reliability and rebuild time, any red flags?


Pure RAID1 does not scale write performance, and tend to sub-optimally scale read performance also. Moreover, when used with more than two disks, you will lose too much space (but you gain in redundancy).

RAID6 is mostly OK, but it has its drawbacks:
- low write performance when used with low level RAID cards
- configured with 4 disks, it does not provide much more space than a RAID10 setup
- rebuilds take a lot of time
- if the array become degraded, I/OI speed degrade a LOT.

With RAID10 you lose more space but:
- you gain in redundancy (sometime you can lose two disks and the array will continue to work)
- you gain in performance (no read/modify/write)
- in case of disk failures, you don't have to do a true rebuild, but a disk-wide copy only (which is faster).

Regarding KVM, if you don't very well understood Linux and you don't have spare time to do some test, I suggest you to remain with what you know better. KVM administration is a little clunky compared to other virtualizer.

Regards.
Last edited by shodanshok on Fri Sep 20, 2013 3:43 pm, edited 1 time in total.
www.ilsistemista.net - test & bench :)
shodanshok
Gerbil
 
Posts: 25
Joined: Thu May 31, 2012 3:39 am

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 3:35 pm

Flatland_Spider wrote:
Xen 4.0+ has memory balloning, and memory sharing and memory paging are in the tech preview stage with the notes stating, "Preview, due to limited tools support. Hypervisor side in good shape."

Xen Release Features
http://wiki.xenproject.org/wiki/Xen_Release_Features


Good to know, but I rarely use tech-preview level features in production environment. Moreover, the real important thing is to dynamically manage the ballon driver, responding to guest memory pressure in the right manner/time. For example, KVM has balloning support since quite a bit, but the dynamic component (MOM - memory overcommit manager) is, in CentOS/RHEL 6.x, somewhat incomplete (it don't provide a clean method to interfacing with Windows guests, while Linux guests are very well supported).

On the other side, VMWare has excellent balloning support since VMWare server 1.x

Regards.
www.ilsistemista.net - test & bench :)
shodanshok
Gerbil
 
Posts: 25
Joined: Thu May 31, 2012 3:39 am

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 3:40 pm

Scrotos wrote:
flip-mode wrote:RAID:

I think it's going to be hardware RAID; I'm just going to assume that for the simplicity's sake for the moment, but regardless I'm 98% sure it'll be hardware. I've heard this from you guys:

RAID 6 or RAID 10.

Anyone have anything against RAID 1?

I know it becomes space-inefficient pretty quickly, but in terms of reliability and rebuild time, any red flags?


RAID 10/1+0 is just RAID 1 with more than 2 disks. A 4-disk RAID 1 is the same thing as a 4-disk RAID 10 in practice. They both lose 50% of their capacity but at least you get some striping action to give a speed boost.


It depends. If it is a pure RAID1 array of N disks, you have a redundancy level of N-1, but the space of one disk only.
www.ilsistemista.net - test & bench :)
shodanshok
Gerbil
 
Posts: 25
Joined: Thu May 31, 2012 3:39 am

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 3:43 pm

After goofing around with XenServer for a little bit today, some things seem much less intuitive than with ESXi. Seems like some extra steps need to be taken to install OS from an .iso file - I need to create an .iso library or something like that.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 3:49 pm

paco wrote:Check out the Dell R515

PE R515 with up to 12 Hot Swap Hard Drives, LED
64GB Memory,(4x16GB) 1600MT/s, Dual Rank LVRDIMMs at Std Volt, for 2 Processors
2x AMD Opteron™ 4386, 3.1GHz, 8C, Turbo CORE, 8M L2/8M L3, 1600Mhz Max Mem
PERC H700 Integrated RAID Controller 512MB Cache,12HD
RAID 10 for PERC H200 and H700 Controllers, x8 and x12 Chassis
750 Watt Redundant Power Supply
(6) 600GB 15K RPM SAS 6Gbps 3.5in Hot-plug Hard Drive (1.8TB HDD Space)
3Yr Basic Hardware Warranty Repair: 5x10 HW-Only, 5x10 NBD Onsite

Sub-total $4,890.54

Make sure to call them, I'm pretty sure you could get that price out the door and maybe add on some more warranty.
You have to remember you don't want to purchase for what is good enough currently. Any business expects to grow and this will allow you to add more hard drive space and more memory easily, this might be good enough for quite some time as it is.

I first compared the R520 and R720 but the AMDs get you a lot more for your money it seems.


I second this. The only possible change is to use 8x 2TB WD SE or RE driver, which are slower (7200RPM) but with much higher space/cost ratio. However, DELL is very angry to let you use 3dr part disk, so:
- you had to run the very latest RAID firmware version (previous version blocked 3dr part driver)
- you had to buy the disk tray/slot from amazon, ebay or other 3dr part resellesr (DELL will not sell you the enclosure without a drive).

In comparison, HP is more open minded. However the last time I checked they did not have a server with the same performance/cost ratio than DELL R515.
www.ilsistemista.net - test & bench :)
shodanshok
Gerbil
 
Posts: 25
Joined: Thu May 31, 2012 3:39 am

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 4:07 pm

paco wrote:Check out the Dell R515

Sub-total $4,890.54


I got around $5K as I included the rails, that's another $130 or so with the cable management arm.

Here's how the 12-drive config (the case, I mean) looks if anyone is interested: http://www.ebay.com/itm/DELL-POWEREDGE- ... 4abd5ee8f6

...though I'd almost get that one and then sell off the 2 TB drives and piecemeal the rest. :D

I found a "build a server" for HP with the same type of chassis:

http://h71016.www7.hp.com/dstore/Middle ... LFF+Server

HP is waaay more expensive. But then again, I never buy vendor stuff directly from the vendor. I hate to say it, but these cheesy places on ebay and internet stores are often HP or Dell authorized resellers and give way better prices. I haven't gotten a "new" HP server in ages because they don't offer any killer features and I get the support later on anyway.

shodanshok wrote:In comparison, HP is more open minded. However the last time I checked they did not have a server with the same performance/cost ratio than DELL R515.


Not new, they really don't. I like their ecosystem, though.
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 4:16 pm

shodanshok wrote:I second this. The only possible change is to use 8x 2TB WD SE or RE driver, which are slower (7200RPM) but with much higher space/cost ratio. However, DELL is very angry to let you use 3dr part disk, so:
- you had to run the very latest RAID firmware version (previous version blocked 3dr part driver)
- you had to buy the disk tray/slot from amazon, ebay or other 3dr part resellesr (DELL will not sell you the enclosure without a drive).

In comparison, HP is more open minded. However the last time I checked they did not have a server with the same performance/cost ratio than DELL R515.


I just wanted to revisit this. Often when you buy a name-brand server with a support agreement, they cover EVERYTHING. However, that is contingent on you using supported hardware. If you have a bunch of WD drives and one dies, they ain't gonna support you. You have an HP drive in a supported HP server, they'll send a replacement out ASAP (depending on your support plan).

If your RAID controller goes belly-up, they'll ship you a new one. If you had WD drives attached, maybe it's no longer covered. I got some 3rd-party RAID memory and it didn't work but because it was an HP part in an HP server, they just sent me new memory. If I hadn't had gone home that day earlier (Saturday), the courier would have had it to me an hour after my call.

So yeah, don't mix and match with your components if you're getting a name-brand and expect support. If you're going to do that route, might as well go 100% custom with random pieces to save a few bucks.

Edit: I forgot one nice thing about HP RAID controllers is that if one dies, you can pop another in and it'll pick up the RAID just fine. Or if the server dies, move the drives to another server and with an HP RAID controller it will pick up the RAID just fine.
Last edited by Scrotos on Fri Sep 20, 2013 4:33 pm, edited 1 time in total.
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 4:23 pm

mmmmmdonuts21 wrote:I have had great success with both supermicro and tyan in the pass. Can you use old equipment from ebay? If so look for the Dell C6100. It is a steal for the price what these badboy servers are. http://www.ebay.com/itm/Dell-Poweredge-C6100-2U-8x-XEON-QC-L5520-2-26GHz-4xNODES-NO-HDD-96GB-Ram-Tested-/251283578250?pt=COMP_EN_Servers&hash=item3a81ab1d8a


Holy crap I just checked this out. NICE.
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Virtualization Server advice

Postposted on Fri Sep 20, 2013 7:10 pm

Scrotos wrote:
mmmmmdonuts21 wrote:I have had great success with both supermicro and tyan in the pass. Can you use old equipment from ebay? If so look for the Dell C6100. It is a steal for the price what these badboy servers are. http://www.ebay.com/itm/Dell-Poweredge-C6100-2U-8x-XEON-QC-L5520-2-26GHz-4xNODES-NO-HDD-96GB-Ram-Tested-/251283578250?pt=COMP_EN_Servers&hash=item3a81ab1d8a

Holy crap I just checked this out. NICE.

Looks like someone must have recently decommissioned a sizable data center! :wink:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37673
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Virtualization Server advice

Postposted on Sat Sep 21, 2013 2:48 am

Blackberry...
Calm seas never made a skilled mariner.
drsauced
Graphmaster Gerbil
 
Posts: 1464
Joined: Mon Apr 21, 2003 1:38 pm
Location: Here!

Re: Virtualization Server advice

Postposted on Sun Sep 22, 2013 2:48 pm

kc77 wrote:Having run KVM for four years now in production it is has been rock solid. I'm talking Tonka truck tough. Had we not moved to a new file system layout they would still be running. I'm not saying it is THE way to go but it should be considered especially if Hyper-V is anywhere in the discussion. If uptime is a factor then Windows being the hypervisor for your VM's isn't exactly the epitome of uptime on Tuesdays.


Oh I know KVM is damn solid - most massive cloud IaaS providers run off of it. It's just not the most user friendly option for someone's first production virtualization project which is why I said:

Beelzebubba9 wrote:*Unless you're awesome with KVM or Xen, then you should be working for a IaaS provider making $250K a year and not on baby VM hosts.


Maybe I'm being a little harsh regarding KVM's complexity, but I see it as being a bit more complex and harder for a novice to implement than ESXi and Hyper-V. Also, I haven't heard anything about Hyper-V server being unreliable in production environments and most of the hardcore Virtualization Engineers I know don't have issues with it. We're a VMware shop, so I have only token experience with Hyper-V, so that that for what you will. :)
Beelzebubba9
Gerbil
Silver subscriber
 
 
Posts: 64
Joined: Tue Oct 19, 2004 10:23 am
Location: New York, NY

Re: Virtualization Server advice

Postposted on Mon Sep 23, 2013 10:10 am

Good Monday morning, gerbils. XenServer won't let me create an "ISO Library". This is slightly annoying. I'll have to burn an OS to disk and then put the disk in the physical host to create a VM, I guess.

On the topic of Vsphere Hypervisor, I just read this:
Unlimited number of cores per physical CPU
Unlimited number of physical CPUs per host
Maximum vCPUs per virtual machine: eight
Limitation of 32GB RAM limit per server/host has been removed from the free Hypervisor.
Operating system support: Microsoft OS (18 versions), Linux (54 versions), Mac OS X 10, Solaris, FreeBSD, etc. (See a complete list of supported versions.)
- See more at: http://www.vmware.com/products/vsphere- ... I5gBz.dpuf


So, no CPU limitations, no core count limitations, and no RAM limitations. Heh, cool.

It's going to be a challenge for me to thoroughly evaluate the differences between XenServer and ESXi. My real job is building design and detailing. :-?
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Virtualization Server advice

Postposted on Mon Sep 23, 2013 10:35 am

flip-mode wrote:It's going to be a challenge for me to thoroughly evaluate the differences between XenServer and ESXi. My real job is building design and detailing. :-?

Welcome to the wonderful world of being your organization's designated "guy who gets to wear the part-time IT hat". :lol:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37673
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Virtualization Server advice

Postposted on Tue Sep 24, 2013 11:53 am

flip-mode wrote:Good Monday morning, gerbils. XenServer won't let me create an "ISO Library". This is slightly annoying. I'll have to burn an OS to disk and then put the disk in the physical host to create a VM, I guess.

On the topic of Vsphere Hypervisor, I just read this:
Unlimited number of cores per physical CPU
Unlimited number of physical CPUs per host
Maximum vCPUs per virtual machine: eight
Limitation of 32GB RAM limit per server/host has been removed from the free Hypervisor.
Operating system support: Microsoft OS (18 versions), Linux (54 versions), Mac OS X 10, Solaris, FreeBSD, etc. (See a complete list of supported versions.)
- See more at: http://www.vmware.com/products/vsphere- ... I5gBz.dpuf


So, no CPU limitations, no core count limitations, and no RAM limitations. Heh, cool.

It's going to be a challenge for me to thoroughly evaluate the differences between XenServer and ESXi. My real job is building design and detailing. :-?


As I did something similar in the past, let me give you one suggestion: first, identify yours typical workloads and the must-have features (eg: live migration, thin provvision, ecc).

Then benchmark them on the two hypervisor with guest agent/driver in place. Sometime you can find some very significant, unexpected differences.

Regads.
www.ilsistemista.net - test & bench :)
shodanshok
Gerbil
 
Posts: 25
Joined: Thu May 31, 2012 3:39 am

Previous

Return to System Builders Anonymous

Who is online

Users browsing this forum: No registered users and 7 guests