Personal computing discussed

Moderators: renee, morphine, Steel

 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

more server fun

Wed Jun 06, 2018 7:39 am

One of this week's projects at work is to get my dev group's test assets straightened out. A lot of our testing is done in VMs, but sometimes you just need to run on real hardware, so we have a rack in the on-site datacenter where we host our physical test systems.

Budgetary/procurement red tape can make it difficult to get new gear in a timely manner, so sometimes we need to get a little creative with repurposing older stuff (just like I do at home). Yesterday I was finally able to rack a "new to us" 4U server and populate all of the drive bays. The chassis was cobbled together out of spare/discarded parts scrounged from other departments, and IT found us a couple of cases of "new old stock" HDDs in a storeroom which provided most of the disks.

Here's a pic of the drive bays (box is pulled halfway out of the rack) right after all of the drives were installed:
Image

The front half of the enclosure has 48 hot swap bays; the rear half (not visible) contains the motherboard, disk controllers, and PSUs. Yes, fully loaded, these things are HEAVY!

A few years ago when I first saw this style of chassis, my initial reaction was "WTF, how do those drives get enough airflow? The ones in the middle will cook themselves!" Turns out if you don't care about noise level, you just keep throwing fans at the problem... 6 of them, in this case (3 pushing air in from the front and 3 pulling at mid-chassis). :lol: As long as your datacenter cooling is adequate, the drives are fine...

After some issues getting our imaging server configured correctly (we don't add physical servers to our setup that often so there were a couple of false starts on my part), the box booted right up, imaged itself to the latest version of our test branch, and detected and formatted all 48 drives (fully automatic, other than I had to manually tell it a couple of the repurposed ones were OK to overwrite, since they weren't blank). 288TB of spinning rust ready for duty! :D
Nostalgia isn't what it used to be.
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: more server fun

Wed Jun 06, 2018 7:50 am

One of today's projects is moving another (similar) server into the same rack. For the past few months the other box has been living (and running) on the floor of a test lab... need to get it out of there. When I get in we'll probably debate whether to try moving it full or pull the drives first. I've racked chassis like this when fully loaded before, and it's a royal PITA... I'm leaning towards pulling at least some of the drives.
Nostalgia isn't what it used to be.
 
SuperSpy
Minister of Gerbil Affairs
Posts: 2403
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: more server fun

Wed Jun 06, 2018 9:03 am

The rule I've always went with is as long as there is some kind of air movement across the drives, they are fine. Which actually makes sense if you consider their surface area to watts ratio. High performance 7200 RPM disk drives draw like, 5 watts tops, apart from the initial spin-up. Which is nothing compared to the ~24 square inches of surface area (assuming only the top and bottom of the drive are getting air).

As long as they aren't relying on convection cooling it's fine.
Desktop: i7-4790K @4.8 GHz | 32 GB | EVGA Gefore 1060 | Windows 10 x64
Laptop: MacBook Pro 2017 2.9GHz | 16 GB | Radeon Pro 560
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: more server fun

Wed Jun 06, 2018 10:24 am

2nd server moved and racked without incident. Pulling half of the drives starting from the front lightened it and improved the center of gravity enough to make it more manageable.
Nostalgia isn't what it used to be.
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: more server fun

Wed Jun 06, 2018 10:25 am

What's the protection method / filesystem on those guys? That's a lot of spinning rust to protect. :)

Throwing fans at JBODs is actually the reason that consumer-style SATA drives are getting better in terms of error correction in "noisy" environments. Fans, in general, do bad things to dense drive enclosures as the harmonics tend to upset the head positioning pretty dramatically. It's impressive how loud a 4U96/108 drive enclosure can be at full tilt.
Victory requires no explanation. Defeat allows none.
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: more server fun

Wed Jun 06, 2018 10:56 am

Waco wrote:
What's the protection method / filesystem on those guys? That's a lot of spinning rust to protect. :)

Proprietary erasure coding scheme. In practice it is typically striped across multiple (3 or more) servers so it can tolerate random drive and server failures with no downtime. For ultra-high availability applications it can even be striped across multiple datacenters, preventing downtime even in the event of a full datacenter outage.

Waco wrote:
Throwing fans at JBODs is actually the reason that consumer-style SATA drives are getting better in terms of error correction in "noisy" environments.

With occasional exceptions. We recently got an entire batch of drives (from a vendor who shall remain nameless to protect the guilty) that clearly have some sort of issue in this area. With certain access patterns*, even a single drive in an otherwise vibration-free environment will log multiple G-sense errors just from the vibrations of its own seeking. SMH at how stuff like this makes it through product testing.

*Initializing the i-node tables during a ext4 format operation is apparently one of these problematic access patterns. :roll:

Waco wrote:
Fans, in general, do bad things to dense drive enclosures as the harmonics tend to upset the head positioning pretty dramatically. It's impressive how loud a 4U96/108 drive enclosure can be at full tilt.

Yup. If I'm going to be in the datacenter for more than a few minutes I use some form of hearing protection!
Nostalgia isn't what it used to be.
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: more server fun

Wed Jun 06, 2018 11:37 am

just brew it! wrote:
Proprietary erasure coding scheme. In practice it is typically striped across multiple (3 or more) servers so it can tolerate random drive and server failures with no downtime. For ultra-high availability applications it can even be striped across multiple datacenters, preventing downtime even in the event of a full datacenter outage.

Ah, nice! I don't typically deal with datacenter-crossing storage since bandwidth is key for us, but that's cool. I'm always happy to hear about sites that are doing erasure across wide spans instead of the more typical "just make more copies" mantra that so many companies fall to because protection in other ways is "hard". :)
Victory requires no explanation. Defeat allows none.
 
CampinCarl
Graphmaster Gerbil
Posts: 1363
Joined: Mon Jul 04, 2005 9:53 pm

Re: more server fun

Wed Jun 06, 2018 11:44 am

Re: Data drives in a noisy environment, a fun, old video from Sun pre-nazi invasion:
https://www.youtube.com/watch?v=tDacjrSCeq4
Gigabyte AB350M Gaming-3 | R7 1700X | 2x8 GB Corsair Vengeance DDR4-3200 (@DDR4-2933)| Samsung 960 Evo 1TB SSD | Gigabyte GTX1080 | Win 10 Pro x86-64
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: more server fun

Wed Jun 06, 2018 2:37 pm

Waco wrote:
Ah, nice! I don't typically deal with datacenter-crossing storage since bandwidth is key for us, but that's cool.

You and the vast majority of people. It's just one more option when reliability and availability matter more than raw bandwidth.
Nostalgia isn't what it used to be.
 
Ummagumma
Gerbil
Posts: 42
Joined: Fri May 27, 2016 9:18 pm

Re: more server fun

Wed Jun 06, 2018 5:08 pm

Quite an impressive expression in "make do IT".

I don't consider "make do IT" a negative term in this case since the corresponding company acquisition paperwork and process sounds quite depressing.

I would call yourself and the team(s) that helped you "recyclers".
Did you expect to read anything useful here?
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: more server fun

Wed Jun 06, 2018 10:30 pm

Ummagumma wrote:
Quite an impressive expression in "make do IT".

I don't consider "make do IT" a negative term in this case since the corresponding company acquisition paperwork and process sounds quite depressing.

I would call yourself and the team(s) that helped you "recyclers".

We even ended up with a few extra registered ECC DIMMs (from one of the carcasses we cannibalized), I've got some extra SAS HBAs stashed away in a closet from a previous "IT recycling" adventure, and now I know who has a pile of spare 48-drive chassis (with SAS/SATA backplanes). So if we need to stand up another one we've got a head start on the parts. Unfortunately the drives are the most expensive part; I don't think IT is going to discover another 2 extra cases of HDDs in the storeroom, and any other "extra" drives that are lying around unused are probably not being used because they're bad. :lol:
Nostalgia isn't what it used to be.

Who is online

Users browsing this forum: No registered users and 20 guests
GZIP: On