Page 1 of 1

How do SAS enclosures and cables work?

Posted: Thu Mar 14, 2013 4:57 pm
by Scrotos
Ok, actually, my questioning is a bit more specific. Let's take an example of the HP DL320s: http://h18004.www1.hp.com/products/quic ... 71_na.html

It supports 14 SAS/SATA drives. Now the integrated controller is a P400: http://h18000.www1.hp.com/products/quic ... 00_na.html

Now here's where I get a bit lost. The P400 has "Eight (8) SAS physical links distributed across 2 internal x4 wide port connectors." My understanding of SAS is that that would be 8 SAS/SATA drives. However, "The SA-P400 Controller supports up to 18 drives depending on the server implementation."

...what? And even more: "Up to 38TB of total storage with 38 x 1TB 3.5" SATA MDL HDD"

So it can support 8 drives? Or 18? Or 38? But all with 8 physical links/channels for SAS?

Or how about the P800: http://h18004.www1.hp.com/products/serv ... index.html
http://h18000.www1.hp.com/products/quic ... 16_na.HTML

You can put a P800 in that DL320s to handle the internal drives and hook up to an external SAS enclosure. 16 physical links via 2 x 4x internal and 2 x 4x external SAS connectors. Now this sucker can link to an external SAS chassis like thus: "Up to 96 TB of external storage per PCI slot with 8 HP 60 Modular Smart Array enclosures and 96 x 1TB 3.5" SATA MDL hard drives"

The MSA 60 enclosure is this: http://h18004.www1.hp.com/products/quic ... 7_div.HTML

I get that you can daisy chain a bunch of them. But I still have issues with the physical SAS links.

Do the SAS backplanes built into these servers and enclosures multiplex all the drives and just flood the SAS interface going to the controller with the max bandwidth it'll take? Like an ethernet switch, conceptually? 7 ports on your ethernet switch are all dumping max data and it's getting parceled out into the one remaining output port, so to speak?

I see some DIY storage solutions where you need, say, 20 drives so you have 1 controller with 5 SAS 4-channel ports or multiple SAS controllers to handle all the drives. Yet HP and other vendors like Dell have controllers that seem unable to handle the physical load of all the drives they claim to handle. What do they do different?

Re: How do SAS enclosures and cables work?

Posted: Thu Mar 14, 2013 6:10 pm
by Krogoth
I think you answered most of your questions.

HP and Dell probably use have their own SAS daisy chain cabling to do what they described in their documentation.

Re: How do SAS enclosures and cables work?

Posted: Thu Mar 14, 2013 8:38 pm
by Arvald
Good coverage here... seems you just need an expander to attach more but the bandwidth looks to get shared
http://en.wikipedia.org/wiki/Serial_attached_SCSI

Re: How do SAS enclosures and cables work?

Posted: Fri Mar 15, 2013 11:23 am
by Scrotos
Ya, so here I am bouncing around between vendor sites and I never bother to wiki it up. Duh. Thanks for the link.

Interesting. So I guessed the gist of it. I always had in my mind that a cascaded set of SAS drives would have like trillions of bytes of bandwidth and all that jazz. But no, it's just for capacity in most cases because you're limited by X SAS links unless you have some enclosure that lets you put multiple SAS links into it.

Whew. Makes sense now. Thanks again!

Re: How do SAS enclosures and cables work?

Posted: Fri Mar 15, 2013 1:40 pm
by TechNut
Yes, the bandwidth is measured on the number of "ports" you have and can put into an expander.

With SAS, it is 3Gb/s/port, and SAS2, 6gb/s/port. Sometimes you will hear this referred too as "lanes" as well. Each port can connect to a drive, or it can connect to another expander.

So, if you have a P400, which sports, 8 SAS ports (2 connectors, 4 ports per connector), you will have 8 ports x (3Gbs x 2 (send/receive)), or 48Gb/s of bandwidth that the card can use.

However, there is a catch... SAS is full duplex, SATA is not.

If you use SATA drives, you only have 24Gb/s of bandwidth (and technically less due to SATA over SAS encoding loss).

Interestingly, I believe your P400 card, is only 4x and PCIe 1.0, so, it's maximum bandwidth with SAS drives is 20Gb/s. You would not get full performance out of it, if you put an expander in. Now, 20Gb/s is about 1GB/s which is not too shabby, but do not expect 20 SSD's to deliver amazing performance, when 8 medium end SATA drives will perform the same with that controller (the exception being random seek latency with the SSD's).

I know this from real life of using a RAID 0 of 4 SSD's and a P400. It's not all that fast.

HTH!

Re: How do SAS enclosures and cables work?

Posted: Fri Mar 15, 2013 2:14 pm
by Arvald
Scrotos wrote:
Ya, so here I am bouncing around between vendor sites and I never bother to wiki it up. Duh. Thanks for the link.

Interesting. So I guessed the gist of it. I always had in my mind that a cascaded set of SAS drives would have like trillions of bytes of bandwidth and all that jazz. But no, it's just for capacity in most cases because you're limited by X SAS links unless you have some enclosure that lets you put multiple SAS links into it.

Whew. Makes sense now. Thanks again!

You are welcome.
I've taken to checking Wikipedia as one of my first sources for background technical information like this... if it is there it is in nearly textbook form so it should be accurate

Re: How do SAS enclosures and cables work?

Posted: Fri Mar 15, 2013 2:20 pm
by Bauxite
You pretty much figured it out for yourself, and the network analogy works well.

SAS expander = switch, typically 24-36 ports (there are "true" SAS switches out there but keeping it simple)

SAS adapter = NIC, usually 4-8 ports.
Anything bigger usually means a built-in expander. You can chain expanders if done properly, this is where some of the monster enclosures come in.
Expanders won't have trouble keeping up with moderate ratios of spinning rust per "true" system ports, ~4:1 should be easy, after that depends.
SSDs on the contrary will notice quickly.

SAS = full duplex, native. Some drives are dual port as well, multipath capable.
SATA = half duplex, tunneling overhead.

SAS cable = usually 4 ports per.
Most enterprise stuff uses backplanes to split for drives instead of fanout cables. Fanout cables have a caveat people often mess up: they are directionally specific.

Re: How do SAS enclosures and cables work?

Posted: Fri Mar 15, 2013 2:56 pm
by Scrotos
Aye, it's actually unrelated to my SSD in a HP project. Thanks fer the help everyone! I've got one more topic that's throwing me for a loop lately and I'm sure it'll cause some arguments. I'll pop that in a different thread.