Personal computing discussed

Moderators: renee, morphine, Steel

 
morphine
TR Staff
Posts: 11600
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 4:23 pm

LukeCWM wrote:
Yes, it is a small business server. Hardware for the replacement will be ordered next week. But maybe everyone's right that storage isn't the current bottleneck. I just want to be forward thinking enough to buy the best product for the money now since we rarely purchase.

It's reasonably easy to image your HDDs and move them to SSDs eventually. Doing the same with your board, CPU, ECC RAM, not so much.
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
 
LukeCWM
Gerbil
Topic Author
Posts: 52
Joined: Thu Nov 01, 2012 12:49 pm

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 5:57 pm

I got to learn about Perfmon. Hooray!

Here is an eight minute snapshot I just took (late in the day, users not super active). I ran a report in the middle of the graph, shown by when the processor (blue line) jumps up. RAM consistently has about 240 MB unused.

Image

Am I tracking the right things for the hard drives? Is the scale useful? Does this tell us anything besides that my co-workers get chatty right before closing time?
 
morphine
TR Staff
Posts: 11600
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 6:01 pm

1) You're out of RAM. 240MB is just the bare minimum that the OS keeps free to play switcharoo with the open apps.
2) The HDD queue length is massive, and you have an I/O bottleneck. This is probably related to #1, you're probably seeing tons of paging.
---- 2.1) Don't go thinking that SSDs alone would help you. You need RAM first.
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
 
LukeCWM
Gerbil
Topic Author
Posts: 52
Joined: Thu Nov 01, 2012 12:49 pm

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 6:10 pm

morphine wrote:
1) You're out of RAM. 240MB is just the bare minimum that the OS keeps free to play switcharoo with the open apps.
2) The HDD queue length is massive, and you have an I/O bottleneck. This is probably related to #1, you're probably seeing tons of paging.
---- 2.1) Don't go thinking that SSDs alone would help you. You need RAM first.


I completely agree about the RAM. The replacement server will likely have 24-32 GB. RAM is cheap right now.

Are you sure about the HDD queue depth? The scale is massive: it was the only way I could get it to show anything other than a flat line. If I am reading it correctly (my first time using this program), the box right beneath the graph says the maximum queue depth was 4, with an average of 0.

Any counters I should add?
 
morphine
TR Staff
Posts: 11600
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 6:15 pm

Aaaand, of course I am an idiot for not looking at the scale.

And that's a good thing, because my prescrition for now is: add RAM.
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
 
LukeCWM
Gerbil
Topic Author
Posts: 52
Joined: Thu Nov 01, 2012 12:49 pm

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 6:58 pm

morphine wrote:
Aaaand, of course I am an idiot for not looking at the scale.

And that's a good thing, because my prescrition for now is: add RAM.


We can't add RAM without buying a 64-bit OS.

1. You think our storage is sufficiently fast?
2. How should I measure this tomorrow? Set up a log? Over what counters and how often should it take samples?

Thanks for all the help, I really appreciate it. :D
 
morphine
TR Staff
Posts: 11600
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 7:04 pm

LukeCWM wrote:
1. You think our storage is sufficiently fast?

I think so, but we don't know the exact details of your DBMS' workload. However, I reiterate the point that you can image the disk and move to SSDs later.

LukeCWM wrote:
2. How should I measure this tomorrow? Set up a log? Over what counters and how often should it take samples?

What you're doing right now, as a baseline analysis, is reasonable, though you should run it over a full day's work. Keep in mind that with so little RAM, whatever measurements you take are tainted - for example, a "lack of RAM" issue ends up often looking like "lack of I/O", since the operating system starts paging like it's going out of style. Ideally, you'd get some measurements at the SQL Server level itself (assuming that's the backend), but I'm not into the mechanics necessary for that (I'm doing the MySQL thing).
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
 
Wirko
Gerbil Team Leader
Posts: 296
Joined: Fri Jun 15, 2007 4:38 am
Location: Central Europe

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 7:41 pm

In case you choose SSDs, I think that you need not worry about the durability.

The database engine does not burn the disk like crazy unless it has to. It does that, of course, if the users often store large chunks of data in it and then update and delete that data over and over many times. This is very unlikely in a CRM application though. There may be a lot of text data entry there but the amount of it, in GB, is small. The app may also use the database to store large scanned documents, multimedia and other things but these are stored once and then kept for a very long time. A 10 MB file, when stored in the DB, only causes 20 MB to be written to disk (10 MB to the data file plus 10 MB to the transaction log file). It's a rough estimate because the DBMS also writes various other logs and can do data relocation, index rebuilding, etc., but still. Reporting? Should use RAM, not disk, for temporary data, but that may or may not be true.

You would be able to plan better if you knew the amount of data that is written to the DB on an average day, with data entry, reporting and other processing that puts any load on the database. Perfmon looks like the right tool if it can be made to show cumulative data, not just current bytes/s. I have no idea how to do that, though.
 
morphine
TR Staff
Posts: 11600
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 7:46 pm

Wirko wrote:
A 10 MB file, when stored in the DB, only causes 20 MB to be written to disk (10 MB to the data file plus 10 MB to the transaction log file).

Storing files in the database, if done without a really good reason, is reason enough for me to pick up an axe and go all American Psycho.
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 8:31 pm

 
absurdity
Gerbil Elite
Posts: 890
Joined: Sat Mar 02, 2002 7:00 pm
Location: VT

Re: Consumer Storage for a Server?

Thu Nov 08, 2012 10:32 pm

For what it's worth, that sort of memory usage is typical of what I see in any system running SBS 2003. They're all very starved for memory, and running terribly because of it.

You won't get anywhere with PAE. SBS 2003 only supports up to 4GB, even with PAE enabled:
http://msdn.microsoft.com/en-us/library/aa366778.aspx

My general feeling about consumer gear is that it's not built, tested or warrantied for high up-time. You can buy spares and keep them in stock yourself to cover failures, but what happens three years from now when you're out of spares and the product was discontinued two years prior?
 
Wirko
Gerbil Team Leader
Posts: 296
Joined: Fri Jun 15, 2007 4:38 am
Location: Central Europe

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 3:17 am

Morphine: agreed. Anyone who does incremental backups would probably go find his axe (there should always be an axe in every server room) and help you out with that.
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 3:29 am

Wirko wrote:
Morphine: agreed. Anyone who does incremental backups would probably go find his axe (there should always be an axe in every server room) and help you out with that.

Don't get me started on the train wreck that is trying to do incrementals on a server that has lots of large (multi-GB) Outlook .pst files on it. Outlook "touches" all attached .pst files even if the user doesn't *look* at (let alone modify) them. WTF? :roll:
Nostalgia isn't what it used to be.
 
Wirko
Gerbil Team Leader
Posts: 296
Joined: Fri Jun 15, 2007 4:38 am
Location: Central Europe

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 5:34 am

Luke, I understand that you're not much into database administration. See if you find anything useful in this article, anyway. There's a mention of "cumulative physical I/O statistics" in it:

http://www.simple-talk.com/sql/performa ... ql-server/
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 9:40 am

absurdity wrote:
You won't get anywhere with PAE. SBS 2003 only supports up to 4GB, even with PAE enabled:
http://msdn.microsoft.com/en-us/library/aa366778.aspx

My general feeling about consumer gear is that it's not built, tested or warrantied for high up-time.


Ah, there is the data I was wondering about. I was kind of rushed when I wrote that.

Consumer gear is fine as long as your servers are disposable. This assumes they are clustered, boot off the network, and there is 100+ of them.

just brew it! wrote:
Don't get me started on the train wreck that is trying to do incrementals on a server that has lots of large (multi-GB) Outlook .pst files on it. Outlook "touches" all attached .pst files even if the user doesn't *look* at (let alone modify) them.


.pst files are a separate rant. I have no idea why they thought that was a good idea.
 
LukeCWM
Gerbil
Topic Author
Posts: 52
Joined: Thu Nov 01, 2012 12:49 pm

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 10:12 am

Thanks for the valuable discussion points. I'll be reading up on them and also taking some measurements today.
 
dextrous
Gerbil Elite
Posts: 570
Joined: Mon Nov 22, 2004 1:49 pm
Location: Ooooooooooklahoma

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 10:55 am

You are definitely memory constrained. If you fix that bottleneck, you might find your current storage is sufficient. To measure your current I/O, perfmon is the correct tool. To see if SSDs are going to be beneficial, you really need to fix the memory issue first so your storage isn't being hammered by paging. Then you could do some analysis on your disks to see if they are the bottleneck. Something to keep in mind is that the more memory you have for SQL, the more data is in memory and the less important the speed of your storage becomes. To see how your storage is performing, some important stats to watch in perfmon are:

1. Disk Reads/sec & Disk Writes/sec - These are your read & write IOPs.
2. Avg. Disk sec/Read & Avg. Disk sec/Write - These are how long your disks take to complete read & write operations.
3. Current DIsk Queue Length - This is how many operations are queued up (should be < 10 on average or you have problems, < 5 is good)
4. % Disk Read Time & % Disk Write Time - These should give you a good idea how much of your workload is writes vs reads and might give you better understanding about SSD durability for your application.
5. Avg. Disk Bytes/Read & Avg. Disk Bytes/Write - These will tell you how big your reads and writes are. Lot's of small reads and writes are really where SSDs shine vs HDDs.

For comparison, I have 50+ servers running on a SAN with 24 15k RPM SAS drives. These servers host a wide range of applications - Exchange, SQL databases, Oracle databases, domain controllers, terminal servers, SharePoint servers, web servers, file servers, etc etc. Right now, here are the averages for the last 10 mintues:

1. Disk Reads/sec & Disk Writes/sec - 1325, 550
2. Avg. Disk sec/Read & Avg. Disk sec/Write - 1.4, 1.6
3. Current DIsk Queue Length - 2.2
4. % Disk Read Time & % Disk Write Time - N/A
5. Avg. Disk Bytes/Read & Avg. Disk Bytes/Write - 28k, 12k
"I take sibling rivalry to the whole next level, if it doesn't require minor sugery or atleast a trip to the ER, you don't love her." - pete_roth
"Yeah, I see why you'd want a good gas whacker then." - VRock
 
Forge
Lord High Gerbil
Posts: 8253
Joined: Wed Dec 26, 2001 7:00 pm
Location: Gone

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 2:20 pm

I haven't been following the whole thread closely, but I just wanted to suggest RAID6 over RAID5, and RAID 01/10 over either. The only reason to go RAID5/6 is if storage space is the main concern. Even with a capable hardware controller, 10/01 tends to be faster, recovers from degraded states more reliably, and simpler to implement.

It's not a major concern here, since you're looking towards SSDs, but with conventional HDDs, RAID5 is becoming almost unusable. The error rates on consumer HDDs coupled with the ever-increasing size of the disks is putting the odds of error-during-resilver dangerously close to 75%. Do you really want a RAID array where a resilver is likely to be impossible? Personally, the data I handle is worth enough to merit a few additional disks, to get the same size and likely better speeds than a RAID5/6 array.
Please don't edit my signature for me. Thanks.
 
LukeCWM
Gerbil
Topic Author
Posts: 52
Joined: Thu Nov 01, 2012 12:49 pm

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 4:28 pm

Thanks for the thoughts on RAID, I appreciate it.

At this point (with the assumption that storage isn't a bottleneck for a database for 15-25 users when using fast mechanical drives), I think I'm leaning towards RAID 1 for the OS, and RAID 1 for the database storage, probably 15k RPM drives. I don't think we need the size of RAID 5 for a while still, and RAID 5 and 6 do have the performance hit. Do you think RAID 1 is secure enough, Forge?
 
LukeCWM
Gerbil
Topic Author
Posts: 52
Joined: Thu Nov 01, 2012 12:49 pm

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 4:34 pm

I ran a log for several hours today. Here are the charts I thought pertinent. Apart from adding more RAM to test, which we can't do, does this tell us anything more about the bottleneck?

Image

Image

Image

Image

Image
 
dextrous
Gerbil Elite
Posts: 570
Joined: Mon Nov 22, 2004 1:49 pm
Location: Ooooooooooklahoma

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 5:08 pm

I don't know if it tells you anything you don't already know. You may take a look at Pages Input/sec to see how often pages are read from disk instead of memory. I'm not sure what a decent value looks like for a SQL server, but our main MSSQL server averages 10/sec if I exclude the backup window where it spikes to over 12k. It doesn't have tons of memory either, so this may not be such a good number.
"I take sibling rivalry to the whole next level, if it doesn't require minor sugery or atleast a trip to the ER, you don't love her." - pete_roth

"Yeah, I see why you'd want a good gas whacker then." - VRock
 
Scrotos
Graphmaster Gerbil
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 5:35 pm

Forge wrote:
Personally, the data I handle is worth enough to merit a few additional disks, to get the same size and likely better speeds than a RAID5/6 array.


So if I had 8 x 100 GB drives with no hot spares, that's...

700 GB RAID 5
600 GB RAID 6
400 GB RAID 10/01

Is my math correct on that? I'd have to spring for a 12 or 14 bay storage solution to equal the size of one of the RAIDs using parity? I'm not familiar enough with RAID to know what resilvering is... ah, I see: http://lonesysadmin.net/2012/03/23/why- ... silvering/

So you're saying that people trying to rebuild RAIDs will get an error almost 75% of the time in their attempt? Seriously? How do storage systems work at all with that high of an error rate?!?
 
Forge
Lord High Gerbil
Posts: 8253
Joined: Wed Dec 26, 2001 7:00 pm
Location: Gone

Re: Consumer Storage for a Server?

Fri Nov 09, 2012 9:09 pm

The failure rate I quoted would be for a large RAID5 with large disks (multi-TB). With the hypothetical 100GB disks, the failure rate would be far lower, perhaps single digit.

Yes, your projected sizes look correct. RAID1 is very disk-number-intensive. On the other hand, it degrades pretty much perfectly (a degraded RAID1 is a perfectly dandy single disk), and good RAID controllers can rearrange reads to interleave them among the member disks, improving access time. For really critical data, where backups are only that, and the data cannot ever be offline, no exceptions, you can even run RAID1 with >2 disks in the mirror, for really ridiculous uptime numbers.

For your purposes, it looks like I/O is not bad, more RAM is always better for SQL, and things aren't being hit too hard in any one place. I'd buy more RAM. RAM is still cheap. I just put 32GB more in each of our Dell R310s at work, to fill them at 48GB each. Two 16GB kits each, 100$ per kit. Performance doesn't come a lot cheaper than that.
Please don't edit my signature for me. Thanks.
 
LukeCWM
Gerbil
Topic Author
Posts: 52
Joined: Thu Nov 01, 2012 12:49 pm

Re: Consumer Storage for a Server?

Mon Nov 12, 2012 11:20 am

dextrous wrote:
I don't know if it tells you anything you don't already know. You may take a look at Pages Input/sec to see how often pages are read from disk instead of memory. I'm not sure what a decent value looks like for a SQL server, but our main MSSQL server averages 10/sec if I exclude the backup window where it spikes to over 12k. It doesn't have tons of memory either, so this may not be such a good number.


Is the Pages Input/sec in the Paging object, the Memory object, the Physical Disk object, or the Logical Disk object?
 
LukeCWM
Gerbil
Topic Author
Posts: 52
Joined: Thu Nov 01, 2012 12:49 pm

Re: Consumer Storage for a Server?

Mon Nov 12, 2012 11:27 am

Forge wrote:
The failure rate I quoted would be for a large RAID5 with large disks (multi-TB). With the hypothetical 100GB disks, the failure rate would be far lower, perhaps single digit.

Yes, your projected sizes look correct. RAID1 is very disk-number-intensive. On the other hand, it degrades pretty much perfectly (a degraded RAID1 is a perfectly dandy single disk), and good RAID controllers can rearrange reads to interleave them among the member disks, improving access time. For really critical data, where backups are only that, and the data cannot ever be offline, no exceptions, you can even run RAID1 with >2 disks in the mirror, for really ridiculous uptime numbers.

For your purposes, it looks like I/O is not bad, more RAM is always better for SQL, and things aren't being hit too hard in any one place. I'd buy more RAM. RAM is still cheap. I just put 32GB more in each of our Dell R310s at work, to fill them at 48GB each. Two 16GB kits each, 100$ per kit. Performance doesn't come a lot cheaper than that.


I'd like to put more RAM in this server, but with the cost of buying a new OS and new SQL, and that the RAM would still be DDR2 at 333 MHz, I think it is just time to replace this server. And based on your advice, I definitely think I'll aim high for the RAM on the next machine.

How much do servers care about memory channels? I know for desktops, dual-channel RAM is obvious and strongly recommended (twice the speed, right?). Triple-channel RAM had a brief run, but now desktops are back to dual. Unless I'm mistaken, most all servers memory is quad-channel. Is it less important for servers to have appropriate numbers of slots filled? Or should I be aiming to have 16 GB or 32 GB or 64 GB of identical RAM sticks in increments of four?
 
Forge
Lord High Gerbil
Posts: 8253
Joined: Wed Dec 26, 2001 7:00 pm
Location: Gone

Re: Consumer Storage for a Server?

Mon Nov 12, 2012 11:32 am

LukeCWM wrote:
How much do servers care about memory channels? I know for desktops, dual-channel RAM is obvious and strongly recommended (twice the speed, right?). Triple-channel RAM had a brief run, but now desktops are back to dual. Unless I'm mistaken, most all servers memory is quad-channel. Is it less important for servers to have appropriate numbers of slots filled? Or should I be aiming to have 16 GB or 32 GB or 64 GB of identical RAM sticks in increments of four?


Number of channels is less important than having at least one stick per channel. Also, I haven't seen a lot of quad-channel servers, the ones I've got are Xeon E5s with three memory channels. They're recent purchases of a recent model, too.
Please don't edit my signature for me. Thanks.
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: Consumer Storage for a Server?

Mon Nov 12, 2012 10:53 pm

The recent AMD G34s are quad channel parts, and that's probably what he's been looking at.
 
LukeCWM
Gerbil
Topic Author
Posts: 52
Joined: Thu Nov 01, 2012 12:49 pm

Re: Consumer Storage for a Server?

Tue Nov 13, 2012 11:23 am

So is it better, for example, to aim for 48 GB of RAM than 64 GB of RAM if the server is triple-channel?

Actually, I was looking here: http://ark.intel.com/products/64591/Int ... -Intel-QPI
 
Scrotos
Graphmaster Gerbil
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Consumer Storage for a Server?

Tue Apr 16, 2013 1:47 pm

I'm curious. How did this turn out?

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On