For some time I've been thinking about getting a NAS for personal usage. However, most of the prebuilt solutions are too expensive here and they don't even come with hard drives... I then decided to research a cheap way to build it my own.
One famous guide for home-built NAS is the NAS KILLER series by u/JDM_WAAAT. I tried to find most of the parts shown there but I always missed one as it was either not available or couldn't be shipped here. This way my only choice was to look at the prebuilt options they mention on the series. I saw a mention of some models of Dell PowerEdges and decided to take a look at the local second-hand market.
There it was, a Dell PowerEdge T410 for R\$400 (equivalent of \$75) including shipping. Such a steal considering that they go fora bout R\$3.5k here! The specs are the following:
Host | Dell PowerEdge 410 |
---|---|
CPU | Intel Xeon X5660 (12) @ 1.596GHz |
GPU | Matrox Electronics Systems Ltd. PowerEdge T410 |
Memory | 32GB DDR3 ECC 1600MHz |
You can see the whole cost of this setup below:
Date (in days) | Item | Price (R$) |
---|---|---|
Dell PowerEdge T410 1 | 400 | |
Dell PERC H200 [^fn:2] | 166.96 | |
5x Seagate ST4000NM0023 4TB [^fn:3] | 4455 | |
32GB 1600MHz RAM DDR3 ECC (16x2) | 161.51 | |
43 | 5183.47 |
Upgrading the PowerEdge RAID Controller (PERC)
Unfortunately my server came with a Dell PERC 6/I which only supports disks as big as 2TB. Doing some research over the internet I found out that I had two options of upgrades here: H200 or H700.
As I'm going to use ZFS as my filesystem, I went with H200 because I can just use it in IT mode (as JBOD) making it possible to pass all the drives directly to my ZFS pool without the hardware interfering much.
Now something that I want to confess here... I was afraid to buy this server and have to pay enterprise prices for hardware or even restrict my ability to expand/replace the system. However, I learned that I can use my new H200 PERC on a regular desktop[^fn:4] with some special cables you can buy for cheap so I might even be able to build a smaller machine with the same amount of disks and a more balanced power/comsuption ratio.
Setting up SSH
The first thing I do on the NixOS installation media is to change the nixos
user password to proceed the installation in another computer:
$ passwd
# over the other computer
$ ssh nixos@<ip>
Configuring the disks
Fortunately, using ZFS with NixOS is a breeze. It has a really good support and I can even boot from a ZFS pool. Let's start by listing the disks and getting their IDs:
[nixos@nixos:~]$ ls -al /dev/disk/by-id/
total 0
drwxr-xr-x 2 root root 340 Nov 2 15:12 .
drwxr-xr-x 7 root root 140 Nov 2 14:54 ..
lrwxrwxrwx 1 root root 9 Nov 2 15:12 scsi-35000c500571d23bf -> ../../sdb
lrwxrwxrwx 1 root root 9 Nov 2 15:12 scsi-35000c500964ac36f -> ../../sdd
lrwxrwxrwx 1 root root 9 Nov 2 15:12 scsi-35000c500964b5e7b -> ../../sdc
lrwxrwxrwx 1 root root 9 Nov 2 15:12 scsi-35000c500964b723b -> ../../sdf
lrwxrwxrwx 1 root root 9 Nov 2 15:12 scsi-35000c500964bbbd3 -> ../../sde
We should use the disks IDs on our ZFS pool, this will avoid some headaches in the future as switching the HDs bays and ZFS losing tracks of which disk is which. Ok, now that we have the IDs, let's wipe them to make sure we don't have any filesystems on them already.
DISK1=/dev/disk/by-id/scsi-35000c500571d23bf
DISK2=/dev/disk/by-id/scsi-35000c500964ac36f
DISK3=/dev/disk/by-id/scsi-35000c500964b5e7b
DISK4=/dev/disk/by-id/scsi-35000c500964b723b
DISK5=/dev/disk/by-id/scsi-35000c500964bbbd3
sudo wipefs -af $DISK1
sudo wipefs -af $DISK2
sudo wipefs -af $DISK3
sudo wipefs -af $DISK4
sudo wipefs -af $DISK5
Now the last bit missing is the partition layout:
sudo sgdisk -n3:1M:+512M -t3:EF00 $DISK1
sudo sgdisk -n1:0:0 -t1:BF01 $DISK1
After this we just copy it to the other drives:
sudo sfdisk --dump $DISK1 | sudo sfdisk $DISK2
sudo sfdisk --dump $DISK1 | sudo sfdisk $DISK3
sudo sfdisk --dump $DISK1 | sudo sfdisk $DISK4
sudo sfdisk --dump $DISK1 | sudo sfdisk $DISK5
Formatting
Boot
Starting with the boot partition:
sudo mkfs.vfat $DISK1-part3
sudo mkfs.vfat $DISK2-part3
sudo mkfs.vfat $DISK3-part3
sudo mkfs.vfat $DISK4-part3
sudo mkfs.vfat $DISK5-part3
ZFS
Considering that I want two disk parity on my setup, I'm going with a raidz2 pool.
sudo zpool create -o ashift=12 \
-O dnodesize=auto \
-O normalization=formD \
-O relatime=on \
-O acltype=posixacl \
-O xattr=sa \
-O mountpoint=none \
-O compression=lz4 \
-O recordsize=1M \
zroot raidz2 \
$DISK1-part1 $DISK2-part1 $DISK3-part1 $DISK4-part1 $DISK5-part1
And the following datasets:
sudo zfs create -o mountpoint=none zroot/root
sudo zfs create -o mountpoint=legacy zroot/root/nixos
sudo zfs create -o mountpoint=legacy zroot/var
sudo zfs create -o mountpoint=legacy zroot/var/media
sudo zfs create -o mountpoint=legacy zroot/var/torrents
sudo zfs create -o mountpoint=legacy zroot/var/samba
sudo zfs create -o mountpoint=legacy zroot/home
Mounting everything together
Mounting is the easiest part of the whole process. However, we need the directories to be there in the first place.
sudo mount -t zfs zroot/root/nixos /mnt
sudo mkdir /mnt/home
sudo mkdir -p /mnt/var/lib/{torrents,media,samba}
sudo mkdir /mnt/boot
Now it's just a matter of mapping everything to the right place:
sudo mount -t zfs zroot/home /mnt/home
sudo mount -t zfs zroot/var /mnt/var
sudo mount -t zfs zroot/var/media /mnt/var/lib/media
sudo mount -t zfs zroot/var/torrents /mnt/var/lib/torrents
sudo mount -t zfs zroot/var/samba /mnt/var/lib/samba
sudo mount $DISK1-part3 /mnt/boot
Finishing up
The only step left is to generate the NixOS configuration with the filesystem layout and install the system:
sudo nixos-generate-config --root /mnt
# don't forget to get your machine id and put it on `networking.hostId`
head -c 8 /etc/machine-id
sudo nixos-install
-
This hardware is actually overkill for this build but I couldn't find anything better for such a price. [^fn:2]: Unfortunately my machine arrived with a Dell PERC 6/i that has a 2TB per disk limit. [^fn:3]: Unfortunately hard drives are really expensive in Brazil due to taxes... a ~$35 drive costing near $200 is just insane! [^fn:4]: A lot of people use the LSI 9240-8I HBA on regular desktops.↩