Personal computing discussed

Moderators: renee, morphine, Steel

 
Scrotos
Graphmaster Gerbil
Topic Author
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

How do SAS enclosures and cables work?

Thu Mar 14, 2013 4:57 pm

Ok, actually, my questioning is a bit more specific. Let's take an example of the HP DL320s: http://h18004.www1.hp.com/products/quic ... 71_na.html

It supports 14 SAS/SATA drives. Now the integrated controller is a P400: http://h18000.www1.hp.com/products/quic ... 00_na.html

Now here's where I get a bit lost. The P400 has "Eight (8) SAS physical links distributed across 2 internal x4 wide port connectors." My understanding of SAS is that that would be 8 SAS/SATA drives. However, "The SA-P400 Controller supports up to 18 drives depending on the server implementation."

...what? And even more: "Up to 38TB of total storage with 38 x 1TB 3.5" SATA MDL HDD"

So it can support 8 drives? Or 18? Or 38? But all with 8 physical links/channels for SAS?

Or how about the P800: http://h18004.www1.hp.com/products/serv ... index.html
http://h18000.www1.hp.com/products/quic ... 16_na.HTML

You can put a P800 in that DL320s to handle the internal drives and hook up to an external SAS enclosure. 16 physical links via 2 x 4x internal and 2 x 4x external SAS connectors. Now this sucker can link to an external SAS chassis like thus: "Up to 96 TB of external storage per PCI slot with 8 HP 60 Modular Smart Array enclosures and 96 x 1TB 3.5" SATA MDL hard drives"

The MSA 60 enclosure is this: http://h18004.www1.hp.com/products/quic ... 7_div.HTML

I get that you can daisy chain a bunch of them. But I still have issues with the physical SAS links.

Do the SAS backplanes built into these servers and enclosures multiplex all the drives and just flood the SAS interface going to the controller with the max bandwidth it'll take? Like an ethernet switch, conceptually? 7 ports on your ethernet switch are all dumping max data and it's getting parceled out into the one remaining output port, so to speak?

I see some DIY storage solutions where you need, say, 20 drives so you have 1 controller with 5 SAS 4-channel ports or multiple SAS controllers to handle all the drives. Yet HP and other vendors like Dell have controllers that seem unable to handle the physical load of all the drives they claim to handle. What do they do different?
 
Krogoth
Emperor Gerbilius I
Posts: 6049
Joined: Tue Apr 15, 2003 3:20 pm
Location: somewhere on Core Prime
Contact:

Re: How do SAS enclosures and cables work?

Thu Mar 14, 2013 6:10 pm

I think you answered most of your questions.

HP and Dell probably use have their own SAS daisy chain cabling to do what they described in their documentation.
Gigabyte X670 AORUS-ELITE AX, Raphael 7950X, 2x16GiB of G.Skill TRIDENT DDR5-5600, Sapphire RX 6900XT, Seasonic GX-850 and Fractal Define 7 (W)
Ivy Bridge 3570K, 2x4GiB of G.Skill RIPSAW DDR3-1600, Gigabyte Z77X-UD3H, Corsair CX-750M V2, and PC-7B
 
Arvald
Gerbil Elite
Posts: 761
Joined: Tue Sep 27, 2011 12:14 pm
Location: Gerbil-land, Canada

Re: How do SAS enclosures and cables work?

Thu Mar 14, 2013 8:38 pm

Good coverage here... seems you just need an expander to attach more but the bandwidth looks to get shared
http://en.wikipedia.org/wiki/Serial_attached_SCSI
 
Scrotos
Graphmaster Gerbil
Topic Author
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: How do SAS enclosures and cables work?

Fri Mar 15, 2013 11:23 am

Ya, so here I am bouncing around between vendor sites and I never bother to wiki it up. Duh. Thanks for the link.

Interesting. So I guessed the gist of it. I always had in my mind that a cascaded set of SAS drives would have like trillions of bytes of bandwidth and all that jazz. But no, it's just for capacity in most cases because you're limited by X SAS links unless you have some enclosure that lets you put multiple SAS links into it.

Whew. Makes sense now. Thanks again!
 
TechNut
Gerbil
Posts: 18
Joined: Tue Apr 03, 2007 8:58 pm

Re: How do SAS enclosures and cables work?

Fri Mar 15, 2013 1:40 pm

Yes, the bandwidth is measured on the number of "ports" you have and can put into an expander.

With SAS, it is 3Gb/s/port, and SAS2, 6gb/s/port. Sometimes you will hear this referred too as "lanes" as well. Each port can connect to a drive, or it can connect to another expander.

So, if you have a P400, which sports, 8 SAS ports (2 connectors, 4 ports per connector), you will have 8 ports x (3Gbs x 2 (send/receive)), or 48Gb/s of bandwidth that the card can use.

However, there is a catch... SAS is full duplex, SATA is not.

If you use SATA drives, you only have 24Gb/s of bandwidth (and technically less due to SATA over SAS encoding loss).

Interestingly, I believe your P400 card, is only 4x and PCIe 1.0, so, it's maximum bandwidth with SAS drives is 20Gb/s. You would not get full performance out of it, if you put an expander in. Now, 20Gb/s is about 1GB/s which is not too shabby, but do not expect 20 SSD's to deliver amazing performance, when 8 medium end SATA drives will perform the same with that controller (the exception being random seek latency with the SSD's).

I know this from real life of using a RAID 0 of 4 SSD's and a P400. It's not all that fast.

HTH!
 
Arvald
Gerbil Elite
Posts: 761
Joined: Tue Sep 27, 2011 12:14 pm
Location: Gerbil-land, Canada

Re: How do SAS enclosures and cables work?

Fri Mar 15, 2013 2:14 pm

Scrotos wrote:
Ya, so here I am bouncing around between vendor sites and I never bother to wiki it up. Duh. Thanks for the link.

Interesting. So I guessed the gist of it. I always had in my mind that a cascaded set of SAS drives would have like trillions of bytes of bandwidth and all that jazz. But no, it's just for capacity in most cases because you're limited by X SAS links unless you have some enclosure that lets you put multiple SAS links into it.

Whew. Makes sense now. Thanks again!

You are welcome.
I've taken to checking Wikipedia as one of my first sources for background technical information like this... if it is there it is in nearly textbook form so it should be accurate
 
Bauxite
Gerbil Elite
Posts: 788
Joined: Sat Jan 28, 2006 12:10 pm
Location: electrolytic redox smelting plant

Re: How do SAS enclosures and cables work?

Fri Mar 15, 2013 2:20 pm

You pretty much figured it out for yourself, and the network analogy works well.

SAS expander = switch, typically 24-36 ports (there are "true" SAS switches out there but keeping it simple)

SAS adapter = NIC, usually 4-8 ports.
Anything bigger usually means a built-in expander. You can chain expanders if done properly, this is where some of the monster enclosures come in.
Expanders won't have trouble keeping up with moderate ratios of spinning rust per "true" system ports, ~4:1 should be easy, after that depends.
SSDs on the contrary will notice quickly.

SAS = full duplex, native. Some drives are dual port as well, multipath capable.
SATA = half duplex, tunneling overhead.

SAS cable = usually 4 ports per.
Most enterprise stuff uses backplanes to split for drives instead of fanout cables. Fanout cables have a caveat people often mess up: they are directionally specific.
TR RIP 7/7/2019
 
Scrotos
Graphmaster Gerbil
Topic Author
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: How do SAS enclosures and cables work?

Fri Mar 15, 2013 2:56 pm

Aye, it's actually unrelated to my SSD in a HP project. Thanks fer the help everyone! I've got one more topic that's throwing me for a loop lately and I'm sure it'll cause some arguments. I'll pop that in a different thread.

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On