Personal computing discussed

Moderators: renee, JustAnEngineer

 
frumper15
Gerbil XP
Topic Author
Posts: 380
Joined: Mon Jan 18, 2010 3:25 pm

New Server for Small Business (part deux)

Fri Nov 04, 2016 3:47 pm

It doesn't seem that long ago and yet forever ago that I posted this thread https://techreport.com/forums/viewtopic.php?f=33&t=87234 about my efforts to put a new server into service at work, replacing an aging Small Business Server 2003 machine running Core2 era hardware with an Ivy Bridge Xeon machine and some SSDs running in RAID1.  I did a bunch of testing and messing around to get comfortable with it.  It's been running pretty much problem free for the last 3+ years.  I could let this thing run for another 3 years and I think it would still meet our needs but I've always been nervous that I will be out of town and something will happen - hardware failure, virus, power surge, etc that will leave the business dead in the water and while it hasn't happened in three years I've always been on the lookout for something better.  I think I found it.

TLDR: Using 2 cheap'n'cheerful servers to enable Redundancy and additional features for our business. 

I wanted to post this to act as a sounding board for some of you guys that have way more experience with this than I do and also as a resource to point folks to if they have questions on how we do it in our business.  My current solution actually came about as a bit of a happy coincidence as I decided to take advantage of a screaming deal on some Lenovo TS140 servers about a year ago.  I think I paid $285 each for a box with Xeon E3-1225V3 / 3.2 GHz, 4GB DDR3 ECC memory, 500GB WD RE4 drive, slim DVD and a bunch of surprisingly quiet fans (seriously - I've had two running next to my desk for the better part of a year and they're nearly silent).  I was originally intending to make them client machines but I was also curious about virtualization with Hyper-V as I have been running client Hyper-V on my W8.1 pro install at home with WHS for a year or two without problem.  I went ahead and got a couple server NICs (Intel i350-T4) that supported SR-IOV, 4 sticks of 8GB DDR3 ECC memory and 3 500GB Crucial MX200 SSDs (for about the same price as I paid for 2x 250GB samsung 840s 3 years ago).  I downloaded the Tech Previews of Server 2016 standard and Hyper-V and started to mess around.  With the commercial release of Server 2016 I decided it was time to put what I had been playing with into practice

Here's the setup that I'm going with; 
2x Lenovo TS140 servers
-One acts as the Hyper-V server and will actually run the production environment.  It has 24GB of memory and 2x 500GB SSDs in RAID1 using the Intel RAID on the motherboard
-The other acts as the management  server.  It is running Server 2016 essentials, but it isn't running the essentials features - I stopped the wizard as soon as it booted into Windows, removed the "Essentials Experience" feature, etc.  I followed the steps listed here: https://community.spiceworks.com/topic/1127829-how-to-configure-windows-server-2012-r2-essentials-hyper-v-role-as-host
Using the 4-port NICs allowed me to configure redundant network connections (using two different Gig switches) for both the Host/Management machines as well as within the VM guest OS.  I also have a patch cable between the machines in case both switches die - I had an extra port and cable so why not? 

In order to save cost I needed to be able to use Server 2016 Essentials as my server OS which avoids the more expensive Server Standard product as well as the need for client CALs which would have been more than double what I paid for Essentials from newegg.  I also really like the client computer backup feature (as a long-time WHS user) as well as the dead simple Remote Web workplace and VPN features to allow folks to get to their work PCs or server info remotely.  The sticking  point for me was whether or not I could do what I wanted to - namely run Server 2016 Essentials as a virtual machine on a Hyper-V 2016 host and manage it from my other Essentials (but not really) physical server running replication (more on that in a minute).  In a word, yes you can.  And now onto how this all makes my problem of uptime and reliability less of a problem.

Replication - specifically the Shared Nothing replication that is available in Hyper-V 2016 (and maybe 2012?) - allows for what I hope to be the answer to my dilemma of a failure occurring in my absence.  Every 5 minutes any changes that are made to the production VM that is running on my Hyper-V server are replicated over to a non-running instance of the same VM on my management server (running Essentials 2016 [but not really]).  It also makes hourly recovery points that I can quickly either roll-back to or what I'll probably use them for is to check something that was accidentally deleted or moved in our dispatching software that doesn't have an "undo" button.  

Now, should something happen and the primary server becomes unavailable I will be able to either restart the server remotely (something not possible previously), failover to the replicated copy, or with a few simple instructions someone else will be able to initiate a failover in a few minutes.  Restarts are also much quicker using Hyper-V of either the Guest OS or the Host OS - currently they can be as long as 10 minutes using our current server.  Hopefully I've improved the security of things as well having Hyper-V running on the bare metal which is essentially a server Core installation there shouldn't be as many attack vectors (I think that's what the kids are calling them these days).  

I haven't made the change to our production server just yet - I only received my release version of Server 2016 earlier this week so I want to make sure any bugs are worked out but I ran the Tech Previews for close to a year without any stability issues.  I also need to do this after-hours or on a weekend because I once-again plan on starting over on my domain configuration with the clients rather than attempt to add the new servers into my existing environment.  This gives me an easy "Plan B" if things go south (plug the old server back in, convert back the few clients I have tried moving to the new environment by restoring backups) and with only 10 client machines it will take less time than figuring out another way.  

It may seem like total overkill for a small business but the reality is this didn't cost me any more than what I spent 3 years ago for a single server and I have gained twice the storage, a lot of redundancy (dual servers, redundant LAN connections using built-in NIC Teaming, separate Management server) and the ability to upgrade my server in the future without even having to shutdown the Guest OS - I forgot the mention share-nothing VM moving is also possible which is pretty neat. 

A couple of things I'm still working on - I'm considering using either Veeam or Altaro backup for the actual VM-level in addition to the built-in server backup that I have running.  I'm a bit obsessed with backups and unfortunately my go-to, Crashplan, doesn't play well with Server Essentials so it might be abandoned this time around in favor of something else - possible Microsoft's Azure or another cloud solution.  Suggestions here would be appreciated.

I'm also toying with the idea of getting a Xeon E3-1246 v3 to add the 4 hyperthreaded cores for future-proofing and they're available now but probably won't be in an year or so.  If you made it this far, thanks for reading.  Would love to have some feedback from those more experienced in case I'm heading into a world of pain before I get into production.
i7-8086K | Z370 AORUS GAMING WIFI | 32GB DDR4-2400 | EVGA GTX 1080 Ti | 512GB 960 Pro | 27" Dell 2560x1440 Gsync | Fractal R6 | Seasonic Focus Plus 850W | Win10 Pro x64.
 
frumper15
Gerbil XP
Topic Author
Posts: 380
Joined: Mon Jan 18, 2010 3:25 pm

Re: New Server for Small Business (part deux)

Tue Nov 15, 2016 10:55 pm

So far the server has been rock solid but I've run into a snag on the performance side of things and hope you guys can help.  The host machine running Hyper-V server 2016 uses 2 MX200 SSDs in RAID1 to host the Operating system and guest machine.  When I run crystaldiskmark on the guest OS, I get great read performance - 700 mb/s but pretty unimpressive write performance - around 70 mb/s.  I ran a test failover machine on my management server which has similar specs to the host server with less RAM and only a single MX300 drive.  There I get 300 mb/s read but also almost 300 mb/s write.  So the single drive has lower read performance (expected) but way better write performance (good but confusing).

So what gives?  It seems like the server is taking advantage of the dual drives for reading at over 700 mb/s but write speeds are pretty bad.  Could this be a driver issue?  It doesn't seem to be a configuration problem with the guest OS as the write performance is good when running on the other host machine.  Does Hyper-V not play well with RAID arrays?  Is there a huge write penalty for RAID1 arrays I'm not aware of?  I stumbled through some other forum posts about getting 4k sectors aligned between host and guest and vhdx, etc. but everything seemed to indicate that was fixed in Server 2012 so I would hope 2016 should still be sorted out there too.

Any ideas of what I can try?  I would like to keep a RAID1 array for redundancy if possible but I don't know that I want to live with less than 100MB/s write speeds either.
i7-8086K | Z370 AORUS GAMING WIFI | 32GB DDR4-2400 | EVGA GTX 1080 Ti | 512GB 960 Pro | 27" Dell 2560x1440 Gsync | Fractal R6 | Seasonic Focus Plus 850W | Win10 Pro x64.
 
frumper15
Gerbil XP
Topic Author
Posts: 380
Joined: Mon Jan 18, 2010 3:25 pm

Re: New Server for Small Business (part deux)

Fri Nov 18, 2016 5:51 pm

If anyone cares to follow along I figured out what the performance issue was over here: https://techreport.com/forums/viewtopic.php?f=5&t=118797
Turns out it was what seems to be a combination of a wonky RAID1 alignment (inconclusive) as well what might be a re-write penalty associated with the SSDs being filled with information from the constantly being written Hyper-V Replica log files which themselves also affect performance.  I'm glad to have found explanations for the performance I am seeing and the nice thing is I can always look into higher performing storage as it becomes available in the future if it becomes a bottleneck.
i7-8086K | Z370 AORUS GAMING WIFI | 32GB DDR4-2400 | EVGA GTX 1080 Ti | 512GB 960 Pro | 27" Dell 2560x1440 Gsync | Fractal R6 | Seasonic Focus Plus 850W | Win10 Pro x64.
 
frumper15
Gerbil XP
Topic Author
Posts: 380
Joined: Mon Jan 18, 2010 3:25 pm

Re: New Server for Small Business (part deux)

Thu Jan 12, 2017 1:05 pm

I did end up getting the Xeon E3-1246 v3 to get hyperthreading and a slightly higher clockspeed before I put the servers into production over the long Thanksgiving weekend a few months back.  So far everything has run pretty well.  I had to work through some networking issues that were causing some stability issues early on but those seem to be fixed now.  When I start to add up all the additional costs I spent to get where we are now it's not as "budget" a solution as I was maybe convincing myself earlier, but I think it's very reliable and high performing for what we did spend. 
i7-8086K | Z370 AORUS GAMING WIFI | 32GB DDR4-2400 | EVGA GTX 1080 Ti | 512GB 960 Pro | 27" Dell 2560x1440 Gsync | Fractal R6 | Seasonic Focus Plus 850W | Win10 Pro x64.

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On