Building a Dell PowerEdge T410 NAS with NixOS and ZFS

For some time I've been thinking about getting a NAS for personal usage. However, most of the prebuilt solutions are too expensive here and they don't even come with hard drives... I then decided to research a cheap way to build it my own.

One famous guide for home-built NAS is the NAS KILLER series by u/JDM_WAAAT. I tried to find most of the parts shown there but I always missed one as it was either not available or couldn't be shipped here. This way my only choice was to look at the prebuilt options they mention on the series. I saw a mention of some models of Dell PowerEdges and decided to take a look at the local second-hand market.

There it was, a Dell PowerEdge T410 for R\$400 (equivalent of \$75) including shipping. Such a steal considering that they go fora bout R\$3.5k here! The specs are the following:

You can see the whole cost of this setup below:

Upgrading the PowerEdge RAID Controller (PERC)

Unfortunately my server came with a Dell PERC 6/I which only supports disks as big as 2TB. Doing some research over the internet I found out that I had two options of upgrades here: H200 or H700.

As I'm going to use ZFS as my filesystem, I went with H200 because I can just use it in IT mode (as JBOD) making it possible to pass all the drives directly to my ZFS pool without the hardware interfering much.

Now something that I want to confess here... I was afraid to buy this server and have to pay enterprise prices for hardware or even restrict my ability to expand/replace the system. However, I learned that I can use my new H200 PERC on a regular desktop⁴ with some special cables you can buy for cheap so I might even be able to build a smaller machine with the same amount of disks and a more balanced power/comsuption ratio.

Setting up SSH

The first thing I do on the NixOS installation media is to change the nixos user password to proceed the installation in another computer:

$ passwd

# over the other computer
$ ssh nixos@<ip>

Configuring the disks

Fortunately, using ZFS with NixOS is a breeze. It has a really good support and I can even boot from a ZFS pool. Let's start by listing the disks and getting their IDs:

[nixos@nixos:~]$ ls -al /dev/disk/by-id/
total 0
drwxr-xr-x 2 root root 340 Nov  2 15:12 .
drwxr-xr-x 7 root root 140 Nov  2 14:54 ..
lrwxrwxrwx 1 root root   9 Nov  2 15:12 scsi-35000c500571d23bf -> ../../sdb
lrwxrwxrwx 1 root root   9 Nov  2 15:12 scsi-35000c500964ac36f -> ../../sdd
lrwxrwxrwx 1 root root   9 Nov  2 15:12 scsi-35000c500964b5e7b -> ../../sdc
lrwxrwxrwx 1 root root   9 Nov  2 15:12 scsi-35000c500964b723b -> ../../sdf
lrwxrwxrwx 1 root root   9 Nov  2 15:12 scsi-35000c500964bbbd3 -> ../../sde

We should use the disks IDs on our ZFS pool, this will avoid some headaches in the future as switching the HDs bays and ZFS losing tracks of which disk is which. Ok, now that we have the IDs, let's wipe them to make sure we don't have any filesystems on them already.

DISK1=/dev/disk/by-id/scsi-35000c500571d23bf
DISK2=/dev/disk/by-id/scsi-35000c500964ac36f
DISK3=/dev/disk/by-id/scsi-35000c500964b5e7b
DISK4=/dev/disk/by-id/scsi-35000c500964b723b
DISK5=/dev/disk/by-id/scsi-35000c500964bbbd3

sudo wipefs -af $DISK1
sudo wipefs -af $DISK2
sudo wipefs -af $DISK3
sudo wipefs -af $DISK4
sudo wipefs -af $DISK5

Now the last bit missing is the partition layout:

sudo sgdisk -n3:1M:+512M -t3:EF00 $DISK1
sudo sgdisk -n1:0:0 -t1:BF01 $DISK1

After this we just copy it to the other drives:

sudo sfdisk --dump $DISK1 | sudo sfdisk $DISK2
sudo sfdisk --dump $DISK1 | sudo sfdisk $DISK3
sudo sfdisk --dump $DISK1 | sudo sfdisk $DISK4
sudo sfdisk --dump $DISK1 | sudo sfdisk $DISK5

Formatting

Boot

Starting with the boot partition:

sudo mkfs.vfat $DISK1-part3
sudo mkfs.vfat $DISK2-part3
sudo mkfs.vfat $DISK3-part3
sudo mkfs.vfat $DISK4-part3
sudo mkfs.vfat $DISK5-part3

ZFS

Considering that I want two disk parity on my setup, I'm going with a raidz2 pool.

sudo zpool create -o ashift=12 \
                  -O dnodesize=auto \
                  -O normalization=formD \
                  -O relatime=on \
                  -O acltype=posixacl \
                  -O xattr=sa \
                  -O mountpoint=none \
                  -O compression=lz4 \
                  -O recordsize=1M \
                  zroot raidz2 \
                  $DISK1-part1 $DISK2-part1 $DISK3-part1 $DISK4-part1 $DISK5-part1

And the following datasets:

sudo zfs create -o mountpoint=none zroot/root
sudo zfs create -o mountpoint=legacy zroot/root/nixos
sudo zfs create -o mountpoint=legacy zroot/var
sudo zfs create -o mountpoint=legacy zroot/var/media
sudo zfs create -o mountpoint=legacy zroot/var/torrents
sudo zfs create -o mountpoint=legacy zroot/var/samba
sudo zfs create -o mountpoint=legacy zroot/home

Mounting everything together

Mounting is the easiest part of the whole process. However, we need the directories to be there in the first place.

sudo mount -t zfs zroot/root/nixos /mnt
sudo mkdir /mnt/home
sudo mkdir -p /mnt/var/lib/{torrents,media,samba}
sudo mkdir /mnt/boot

Now it's just a matter of mapping everything to the right place:

sudo mount -t zfs zroot/home /mnt/home
sudo mount -t zfs zroot/var /mnt/var
sudo mount -t zfs zroot/var/media /mnt/var/lib/media
sudo mount -t zfs zroot/var/torrents /mnt/var/lib/torrents
sudo mount -t zfs zroot/var/samba /mnt/var/lib/samba
sudo mount $DISK1-part3 /mnt/boot

Finishing up

The only step left is to generate the NixOS configuration with the filesystem layout and install the system:

sudo nixos-generate-config --root /mnt

# don't forget to get your machine id and put it on `networking.hostId`
head -c 8 /etc/machine-id

sudo nixos-install