So, I should probably explain my current setup.
My current setup is a Dell Dimension 2100, with a 1.1 GHz Coppermine Celeron, 384 MiB RAM, and a generic PCI SATA card (with a RAID BIOS, but it ain't actually RAID, and I'm using Linux softraid). It's running Ubuntu Server 10.04, running a full LAMP stack for various web-based services (my blog and my Tiny Tiny RSS install being the main things), Transmission for torrents (I think I'd prefer sticking with Transmission, so I'd really want an OS that has that packaged for it), as well as being a Samba and NFS fileserver (I'd like to add AFP to that in the future, but the version of netatalk that's packaged for 10.04 is unable to support Time Machine backups, so I've seen no point in it yet). I run a few other minor things (like an irssi session) on it.
The new hardware is a Dell CS24-SC (a custom system that has flooded the market lately, immediate predecessor to the C1100, vaguely similar to a PowerEdge 1950-III, but with four 3.5" bays, and a different graphics/remote management chip, a LSI 1064E SATA/SAS "RAID" controller (it's really fauxraid)). Specs are dual Xeon L5420 (essentially a 50 watt 2.5 GHz Core 2 Quad), 16 GiB RAM.
Really, I'm not tied to Ubuntu or even Linux, although I'm definitely used to it. I could go for just about any *nix here. And, there's enough resources that if something really does need Linux itself, I could always spin up a VM (which, there are things that I'd like the ability to spin up a VM for, so decent virtualization support for whatever I use is important, but that's available pretty much everywhere, and I don't need the main OS to run in a VM, just some guests (Linux and maybe Windows)).
So, here's what I'm looking for:
Relatively low maintenance - set it and forget it is the ideal here. I've found that my Ubuntu server has pretty much done this well, although I've learned to handle updates of web apps manually, rather than through package management (partially because the versions in package management are ancient and insecure), and I don't like that I have to do that - I'd rather have all updates centralized.
ZFS would be nice, but certainly isn't a must-have. There's plenty of things that I like about it, but I realize that this means switching OSes (because Linux and ZFS ain't exactly low maintenance, and I'd rather not go for btrfs right now). And, Linux softraid+LVM can get some of the benefit (but not with anywhere near ZFS's flexibility).
Excellent documentation. Bad documentation sends me into apoplectic fits, and I find that a lot of Linux distros are guilty of this (I don't know how much outright WRONG documentation I've read for various Linux distros). I don't even mind if it's a bit hard to figure something out, as long as TFM tells me how to do it properly.
Good updating policies. My ideal OS would be one that keeps all packages up to date, in a way that avoids breaking things, in a rolling release model. But, that's not really possible. Ubuntu fails here, because the LTS releases don't get good backports, and the non-LTS releases require frequent major upgrades that are to be treated as somewhat dangerous.
I'm thinking that FreeBSD and Solaris/Illumos (but which side of the fork, and which distro if the Illumos side?) are probably where I need to look here, but I'm certainly open to continuing on Linux if that's what's best for me, too.