Self Hosting

Storage server, part 5: migrating from mdadm to btrfs and unencrypted to encrypted

Fri, 10 Jul 2026 14:00:00 +0200

In part 3 and part 4 I explained how I configured my NAS with encrypted drives and btrfs in RAID1 mode. Strictly speaking, it wasn’t true. When I originally built this server a few years ago, I used mdadm RAID1 and no encryption. I soon regretted the decision.

Some time later, when I replaced some old hard drives with higher-capacity ones, I decided now it’s a good time. Changing the setup requires moving the data around. Having some drives less than 50% full simplifies it. This post is the actual migration, done live on my NAS (“firefly”) without buying any new hardware.

The starting point

Before the change, sdb and sde (2x3TB) were an mdadm RAID1 array (md0), with LVM (datavg) on top providing two mounted logical volumes:

othervol at /data/other
noshare at /data/noshare

Four HDDs (sdc, sdd, sdf, sdg) were ext4, managed by SnapRAID, unencrypted. System drive sda is left untouched.

Target state:

sdb + sde: LUKS on each disk individually, btrfs RAID1 on top, so bitrot gets corrected automatically
every SnapRAID disk: LUKS on each disk individually too, ext4 on top of it
one password unlocks everything, via a script run manually over SSH after boot, never at boot itself

The process

Step 1: make sure nothing writes to the drives

A few things on this NAS (Grafana, Jellyfin, Syncthing, Samba) read or write to /data/other or /data/noshare. I didn’t want files to change mid-process. And later, those paths will be empty until the unlock script runs. I stopped them and disabled autostart:

for container in "${DOCKER_CONTAINERS[@]}"; do
 docker stop "$container" || true
 docker update --restart no "$container"
done

for service in "${SYSTEMD_SERVICES[@]}"; do
 systemctl stop "$service" || true
 systemctl disable "$service"
done

I also disabled IDrive scheduler, so it wouldn’t back up my local copies.

Step 2: local copy

Reformatting sdb/sde destroys everything on them, so I needed a temporary local copy (I have an offsite backup on IDrive, but restoring would be slow, plus 2 copies are safer than 1). I decided to use one of the SnapRAID video volumes which had enough free space:

cp -a /data/other /data/filmy/backup/
cp -a /data/noshare /data/filmy/backup/

Before wiping the old array, I verified the copy was bit-for-bit correct. There went another few hours.

rsync -ac --dry-run --stats /data/noshare/ /data/filmy/backup/noshare/
rsync -ac --dry-run --stats /data/other/ /data/filmy/backup/other/

No files listed means both copies are identical. You might wonder why I used rsync for verification only, but not for doing the actual copy. Rsync has many features - it can resume, transfer over a network, copy only changed parts etc. But it also means for a simple task of copying files to an empty directory on the same machine, it’s noticeably slower. And, when you’re moving back and forth hundreds of gigabytes, a slower tool could mean an extra few hours.

Step 3: tearing down the old mdadm/LVM

Now that I had a local copy, sdb/sde could be wiped. I prepared a script, supported by Claude Code. Nobody reviews my home scripts and there’s no test environment. Which is why I don’t trust the AI agent, but I don’t trust myself either (I don’t hallucinate, but I make typos and can mix up device names). Here’s the relevant part.

for mp in "${OLD_LV_MOUNTPOINTS[@]}"; do
 umount "$mp" 2>/dev/null || true
done
if vgs "$OLD_LVM_VG" &>/dev/null; then
 vgchange -an "$OLD_LVM_VG"
 vgremove -f "$OLD_LVM_VG"
fi
if [ -e "$OLD_MD_DEVICE" ]; then
 mdadm --stop "$OLD_MD_DEVICE"
fi
for dev in "${BTRFS_RAID1_DEVICES[@]}"; do
 mdadm --zero-superblock "$dev" 2>/dev/null || true
 wipefs -a "$dev"
done
sed -i '/ARRAY \/dev\/md0/d' /etc/mdadm/mdadm.conf 2>/dev/null || true

The few || true parts and the if check give a bit of idempotency: if the script fails, it can be rerun. Not as safe as an Ansible playbook would be, but easier to create for a one-off task.

What it does is: unmount the filesystems, deactivate and remove the volume group, stop the array, zero the mdadm superblock on each member disk, wipefs to clear any remaining LVM/mdadm signatures, and remove the array from mdadm.conf so it doesn’t try (and fail) to reassemble it on the next boot.

Step 4: LUKS on each disk

The order of layers is important: physical disks, LUKS, then btrfs. For the “one password for everything” requirement, I generated a single random keyfile once and used it directly as the LUKS key material on every disk in the whole setup (this pair and the SnapRAID ones later):

if [ ! -f "$KEYFILE" ]; then
 dd if=/dev/urandom of="$KEYFILE" bs=512 count=4
 chmod 600 "$KEYFILE"
fi

LUKS can have multiple keys unlocking the same device. I added a second key slot on each disk with a passphrase that I saved in my password manager, in case the keyfile is ever lost or corrupted.

read -r -s -p "Recovery passphrase: " RECOVERY_PASSPHRASE
echo
read -r -s -p "Confirm recovery passphrase: " RECOVERY_PASSPHRASE_CONFIRM
echo
if [ "$RECOVERY_PASSPHRASE" != "$RECOVERY_PASSPHRASE_CONFIRM" ]; then
 echo "Passphrases don't match." >&2
 exit 1
fi
unset RECOVERY_PASSPHRASE_CONFIRM

Then created the encrypted devices and added the recovery passphrase as a second slot:

mapper_devices=()
i=1
for dev in "${BTRFS_RAID1_DEVICES[@]}"; do
 name="${BTRFS_MAPPER_PREFIX}${i}"
 cryptsetup luksFormat --type luks2 "$dev" --key-file "$KEYFILE"
 printf '%s' "$RECOVERY_PASSPHRASE" | cryptsetup luksAddKey "$dev" --key-file "$KEYFILE" -
 cryptsetup luksOpen "$dev" "$name" --key-file "$KEYFILE"
 mapper_devices+=("/dev/mapper/$name")
 i=$((i + 1))
done
unset RECOVERY_PASSPHRASE

Step 5: btrfs RAID1 on top of LUKS devices

With both disks open as /dev/mapper/luks-btrfs1 and luks-btrfs2, creating the actual RAID1 pool is a single command:

mkfs.btrfs -m raid1 -d raid1 -L "$BTRFS_LABEL" "${mapper_devices[@]}"

Btrfs replaces the LVM layer I used to have, so instead of logical volumes I created subvolumes:

mkdir -p /mnt/btrfs-root
mount "${mapper_devices[0]}" /mnt/btrfs-root
for entry in "${BTRFS_SUBVOLUMES[@]}"; do
 subvol="${entry%%:*}"
 btrfs subvolume create "/mnt/btrfs-root/${subvol}"
done
umount /mnt/btrfs-root

for entry in "${BTRFS_SUBVOLUMES[@]}"; do
 subvol="${entry%%:*}"
 mountpoint="${entry#*:}"
 [ -z "$mountpoint" ] && continue
 mkdir -p "$mountpoint"
 mount -o "subvol=${subvol}" "${mapper_devices[0]}" "$mountpoint"
done

Notice both subvolumes mount through mapper_devices[0] (the first disk). With a multi-device btrfs filesystem you only mount one member - any of them - and btrfs deals with finding the other one and keeping them in sync.

Finally, the script added /etc/fstab entries for both mountpoints, but with noauto so they’re not mounted at boot. I removed the old entries from fstab manually.

for entry in "${BTRFS_SUBVOLUMES[@]}"; do
 subvol="${entry%%:*}"
 mountpoint="${entry#*:}"
 [ -z "$mountpoint" ] && continue
 echo "/dev/mapper/${BTRFS_MAPPER_PREFIX}1 $mountpoint btrfs subvol=${subvol},noauto,nofail,nodev,noexec,noatime 0 0" >> /etc/fstab
done

Step 6: moving the data back

With the new pool mounted and empty, I copied the data back from its temporary home. This time I used a tar pipe instead of cp -a. Since both directories have a lot of small files and tar avoids the per-file overhead, I was hoping it would be faster. It was, actually, a bit slower. Oh well.

tar -cf - -C /data/filmy/backup/other . | tar -xf - -C /data/other
tar -cf - -C /data/filmy/backup/noshare . | tar -xf - -C /data/noshare

Once both finished, I verified with rsync again:

rsync -ac --dry-run --stats /data/filmy/backup/other/ /data/other/
rsync -ac --dry-run --stats /data/filmy/backup/noshare/ /data/noshare/

Only after both came back clean did I delete the temporary copy on /data/filmy.

Step 7: verifying the RAID1 pool

Btrfs hides some of its internal structure from standard Linux tools (though not as much as ZFS). The lsblk command that’s very useful on a machine with many hard drives only shows mount points on one of the underlying devices:

sdb └─luks-btrfs1 /data/noshare
 /data/other
sde └─luks-btrfs2

That is expected. The real check is:

btrfs filesystem show /data/other
btrfs filesystem usage /data/other

They confirm the RAID1 setup. Another command you might be used to: df -h, is not accurate on btrfs, the command you need is btrfs filesystem df /data/other. For a simple setup like this - just a bunch of files copied all in one batch, no snapshots, no compression - the numbers were very close, but it’s not always the case.

Also worth noting, the btrfs commands above on /data/noshare and /data/other give exactly the same results. That’s because these are subvolumes of the same filesystem.

Step 8: the deferred, single-password unlock script

The LUKS keyfile gets GPG-encrypted with a human passphrase, and the plaintext copy gets shredded once every disk (this pair, and eventually the SnapRAID disks) is formatted with it.

The unlock script, run manually over SSH after boot, asks for the password once:

read -r -s -p "Password: " PASSPHRASE
echo
echo -n "$PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 --decrypt "$KEYFILE_ENC" > "$KEYFILE_TMP"
unset PASSPHRASE
chmod 600 "$KEYFILE_TMP"

The decrypted keyfile is saved to /dev/shm (tmpfs, never written to disk), and gets shredded on exit via a trap, whether the script succeeds or fails. From there, it’s the mirror image of the setup script - open both LUKS containers by UUID, scan for the btrfs devices, mount the subvolumes:

for uuid in "${BTRFS_RAID1_UUIDS[@]}"; do
 name="${BTRFS_MAPPER_PREFIX}${i}"
 cryptsetup luksOpen "/dev/disk/by-uuid/$uuid" "$name" --key-file "$KEYFILE_TMP"
 btrfs_mapper_devices+=("/dev/mapper/$name")
 i=$((i + 1))
done

btrfs device scan

I was testing this in stages: the SnapRAID disks weren’t encrypted yet, so their section of the unlock script was commented out for now. After mounting the filesystem, the script also starts the dependent services again:

for container in "${DOCKER_CONTAINERS[@]}"; do
 docker start "$container"
done

for service in "${SYSTEMD_SERVICES[@]}"; do
 systemctl start "$service"
done

Step 9: encrypting the first SnapRAID disk

Similarly to the mdadm device, I had to make local copies using free space on the other drives. This means working one disk at a time.

First up: /data/seriale on /dev/sdg1. Once its contents were safely copied to /data/filmy/backup, the disk itself could be wiped and re-encrypted. I turned the relevant part of the setup script into something reusable, since I’ll be running it four times (the remaining video disks, plus the parity disk):

umount "$MOUNTPOINT" 2>/dev/null || true
awk -v mp="$MOUNTPOINT" '$2 != mp' /etc/fstab > /etc/fstab.tmp && mv /etc/fstab.tmp /etc/fstab
wipefs -a "$PARTITION"

I switched the fstab cleanup from a sed pattern match to an awk filter on the exact mountpoint field - a substring match could accidentally catch something like /data/seriale1. In the past, I used more, but smaller drives; old paths were still present here and there.

Same key setup as the btrfs pair: the same master.key, and the same recovery passphrase added as a second key slot.

cryptsetup luksFormat --type luks2 "$PARTITION" --key-file "$KEYFILE"
printf '%s' "$RECOVERY_PASSPHRASE" | cryptsetup luksAddKey "$PARTITION" --key-file "$KEYFILE" -
cryptsetup luksOpen "$PARTITION" "$MAPPER_NAME" --key-file "$KEYFILE"

I then recreated ext4 with the same options that I used before:

mkfs.ext4 -m 0 -T largefile4 -L "$LABEL" "/dev/mapper/$MAPPER_NAME"
mkdir -p "$MOUNTPOINT"
mount "/dev/mapper/$MAPPER_NAME" "$MOUNTPOINT"
echo "/dev/mapper/$MAPPER_NAME $MOUNTPOINT ext4 noauto,nofail,nodev,noatime 0 0" >> /etc/fstab

Next, the data goes back onto /data/seriale, verified the same way as step 6, and its LUKS UUID gets added to the (still commented-out) SNAPRAID_DISKS block in the unlock script.

Step 10: encrypting the rest of the SnapRAID disks

Then it’s the same routine again for filmy, video2 and parity1. Only once it was done, I uncommented the SnapRAID block from the unlock-mount.sh script. I ran snapraid --force-uuid check | tee snapraid.log 2>&1 to confirm the array’s fine. SnapRAID refuses to run when more than one disk changes, so the force parameter is needed. Then I shredded the unencrypted master.key. Finally, a reboot, to confirm everything was working as planned. It was!

Summary

The whole process took three days. Most of that was waiting for the file copy or verification (or idle time when it was done but I was away). Actual time spent at the terminal - maybe 2 or 3 hours.

As a result, I got bitrot protection on my main array, plus all the nice features of btrfs. I’m not enabling compression (most space is used by already compressed files, such as JPEG photos, MP3 audio or archives). I’m surely going to use snapshots.

Encryption was the most important goal. Most files on the NAS aren’t confidential - I wouldn’t care if my music collection leaks. But some are personal. My laptops have been encrypted for years, by encrypting the NAS I closed a major hole. Even if it was unlikely to be exploited, it didn’t feel proper.

Tips

I doubt anyone would be doing exactly the same thing, but maybe something similar?

Drive letters can change

After a reboot, /dev/sda can become /dev/sde. There are several ways to deal with it:

UUIDs are the best permanent solution to use in /etc/fstab or scripts. They don’t change, unless you reformat the filesystem.
lsblk is your friend. For the commands you type manually, short drive paths are convenient, lsblk is the quickest way to check all drives. Or, a less handy lsblk -o NAME,TRAN,UUID,FSTYPE,MOUNTPOINT gives you UUIDs and transport type (usb, sata, nvme and others).
Filesystem labels can be used in fstab or mount commands, but on an encrypted machine, they’re only visible once the drives are unlocked.
LVM, if used, can scan all devices for PVs, with the same caveat about encrypted drives.

Use screen or tmux

All the copying or verification commands take hours to run. If your server is running headless and you log in with ssh from another machine, you need to make sure the commands won’t be stopped if your connection is interrupted.

Have more than one copy

I only worked on local copies during the migration. But it’s easy to make a slip and type “wipefs /dev/sdf1/” when you actually meant “sdg1”. It didn’t happen to me, but I was prepared for it. Off-site backup gives peace of mind, local backup is faster, have both.

Monitor hard drive throughput

Two useful commands: iotop shows which process reads/writes most data. If the throughput is lower than expected, iostat -x 1 will show whether source or target is to blame - look for high %util and w_await.

In my case, the first attempt to make a local copy started at 250MB/s, but in a few seconds dropped to 30MB/s. Iostat showed that the destination disk was the bottleneck. Turned out the disk I’d picked was an SMR model. And that’s exactly what’s expected from SMR drives - once their write cache fills up, write speed drops to unacceptable values. I checked my HDDs, luckily one of my CMR drives (WD Red Plus) also had enough free space. Copying speed was a more sensible 100MB/s (the bottleneck was now the source drive, and I couldn’t achieve anything better from this hardware).

Later in the same copy job, speed dropped to just a few MB/s for a different reason: a scheduled mdadm RAID consistency check had started and was competing for reads on the same array. That was more tricky to find, since the kernel task didn’t show up on iotop. cat /proc/mdstat showed what was happening and that the task would take 10 hours to complete; echo idle > /sys/block/md0/md/sync_action paused it. I didn’t need the check, in a moment I’d be wiping the array anyway.

Storage server, part 3: ZFS and btrfs vs traditional filesystems

Thu, 09 Jul 2026 14:00:00 +0200

What I have now

I use btrfs RAID1 for all miscellaneous data filesystems on my NAS and ext4 with SnapRAID for video files. Previously, I used ext4 for all. Both approaches have their merits. But first, one important definition.

What is bitrot (silent data corruption)?

Digital data, in general, is resistant to errors. There’s always a large enough difference between 0 and 1, on any storage or transmission media, that background noise and other random factors won’t flip one to the other. Unless the noise is so high it corrupts many bits, which is at least immediately detectable, but it can’t flip a single bit.

Except it sometimes can. It’s not impossible, just very improbable, and if your device contains billions of bits, it will happen - especially on modern high-density media, and especially when it ages. The causes are varied: magnetic domains weakening on HDDs, charge leaking from NAND cells on unpowered SSDs and USB drives, a short voltage spike, tiny movement of HDD head during write, cosmic rays (this one seems to get the most coverage, but it’s the least probable).

The effect is the same. The device continues to function, but some individual bits are changed. No error is reported by the OS or drive electronics. Sometimes, when you read files from many years ago you might notice a weird rectangle in a JPG photo, or an MP3 file starts skipping. You begin to wonder whether the file was corrupt from the beginning, after all, digital files don’t age. Now you know: they do.

It’s more of a concern for a NAS than for a regular computer. It’s a matter of probability: the more data you have, and the longer it sits there without being rewritten, the more likely the bit flip is.

Protecting against bitrot

Standard RAID doesn’t protect against bitrot, neither does standard backup. They might detect that two copies are different, but have no way to tell which one is right. You can store checksums of your files and regularly verify if they changed.

And that’s exactly what ZFS and btrfs do. They both store a checksum alongside every block of data. On every read, the filesystem verifies the checksum; if it doesn’t match, and you have redundancy (mirror, RAID-Z, RAID1), it automatically reconstructs the correct data from the good copy and can rewrite the bad block.

They also have a “scrub” command, that you should run regularly. It reads part of your dataset and verifies checksums, to also detect errors on those files that you don’t read regularly.

SnapRAID also stores checksums of all files. But they are only checked if you run snapraid scrub (or snapraid check, which runs full verification of the whole drive), not on every read. All the more reason to schedule regular scrubs.

Modern filesystem

ZFS and btrfs have some common features. It’s not a coincidence - btrfs was inspired by ZFS and despite not sharing code, they ended in a similar place. There are other filesystems like that, on Linux and other platforms, though they don’t have a full set of features.

Both ZFS and btrfs combine RAID and a volume manager with filesystem functionality. While old-time sysadmins (me included) may frown on this approach - so against the traditional Unix spirit of one tool doing one thing - there are reasons for it:

Bitrot correction, as described earlier. If you separated the layers, RAID would have 2 copies of data but doesn’t know which one is correct; filesystem would detect bitrot, but would have no way to retrieve the correct copy.
Array rebuild efficiency. Traditional RAID copies all blocks, because it has no way of knowing which are used, ZFS and btrfs skip empty blocks.
Better performance without manual tuning. You don’t need to align the filesystem’s block size to the RAID layer’s stripe width, you don’t even need to know what that means.

Both also feature Copy-on-Write (CoW) snapshots. Snapshots look like an exact copy of a filesystem, but they only need extra space for the data that’s different. Once you get used to working with snapshots, you’ll wonder how you could have lived without them:

Creating “system restore points” before a risky operation such as a distro upgrade and instantly rolling back if something goes wrong.
Fast local backup with the ability to browse through old versions of a file (note, it doesn’t replace regular backup, it lives on the same computer).
Efficiently storing multiple VMs that are mostly identical.
A read-only snapshot ensures the data is not modified in the middle of a backup session.
Snapshots can be sent and received over a network, very efficiently (only changed blocks) allowing fast off-site backup or remote replication.

ZFS

Older of the two. Started on Solaris in 2001. Later, when large parts of Solaris were published under an open source licence, ported to FreeBSD. The “ZFS on Linux” project began in 2008, but wasn’t considered stable before version 0.6.0 from 2013.

Licence doesn’t allow integrating it directly into the Linux kernel, the module needs to be built separately. Some distros ship ZFS kernel module and tools. Some don’t, but a determined enough user can build the module themself. Yet, it feels a bit like a second class citizen in Linux. You can’t boot the OS from it (except recent versions of Ubuntu that have experimental support), native Linux tools don’t work with ZFS - you need to use ZFS tools that use a different vocabulary. Due to being out of kernel, it’s often difficult to use it with Secure Boot. Other downsides include high resource usage (depending on features used, it might need a few GB of RAM for itself plus a noticeable amount of CPU time) and difficult performance tuning.

On the plus side, it’s the better tested of the two and has more features. It can do equivalents of RAID1, 5 and 6 - and do them in a safer way. A known problem with RAID5/6 is called write hole. If the machine shuts down uncleanly in the middle of a write (e.g. due to a kernel bug or power outage), it’s possible that the data was written, but metadata wasn’t, leaving inconsistent state. In ZFS, write operations are atomic, they either fully succeed or fully fail.

I tested ZFS and decided against it, mostly due to friction with Linux (though appetite for RAM was a factor too). If your goal is something more akin to a commercial NAS, where you’re more interested in file storage than tinkering, you might be satisfied with ZFS. Or if you don’t have many years of Linux experience and therefore a strong bias.

btrfs

The filesystem was developed in the late 2000s as a response to ZFS licensing problems. It has most (but not all) features of ZFS and is superficially similar, even if under the hood it’s quite different.

Btrfs was merged into the mainline Linux kernel before it became stable. When it stabilised a few years later, it continued adding new, experimental features. Some users didn’t understand which parts of btrfs were safe to use. This is the source of the bad reputation of btrfs.

In fact, these days it’s safe to use btrfs on a single disk or in RAID1/10 mode. The features that distinguish it from traditional filesystems (data checksumming, snapshots, subvolumes, compression) work reliably. It is widely used in production e.g. at Facebook, on Synology devices, as a default filesystem in several Linux distros.

The remaining experimental part is RAID5 and 6. Stability is improving with each new version, but it’s still considered not safe for production. Most posts about btrfs problems are from people who used the RAID5 mode, or from the very early years.

Compared to ZFS, btrfs is a bit more flexible - handles adding/removing disks in an existing array more easily. It also avoids licensing problems and all the “not invented here” problems that come from it. No extra modules that can lag behind the current kernel version or fail to build. Last but not least, for a greybeard sysadmin - less different from what I’m used to.

Traditional filesystems

Old, boring filesystems such as XFS and (especially) ext4 still have some upsides. They are simple and therefore predictable. They don’t have hundreds of parameters for performance tuning (they have a few, but defaults mostly work). Decades of fsck, forensic tools, and “how to recover an ext4 filesystem” knowledge exist, including commercial data-recovery services. We can safely assume that all edge cases around unusual configurations were already discovered and fixed. Decoupling RAID, LVM and the filesystem means that if something isn’t right, you’re debugging one layer at a time. For simple use cases, reading and writing files (which, after all, are the main tasks of filesystems) they are usually faster. They use less RAM and CPU and don’t need that much disk space for metadata.

Copy-on-write filesystems have an additional failure mode. They hate running out of disk space, and sometimes you can be hit by “write error, filesystem full?” message when df shows plenty of space left. Occasionally, you need to run btrfs defrag, btrfs rebalance or their ZFS equivalents (in my experience, in some unusual cases “occasionally” turns to “every night from cron”, but these were real servers with high utilisation, not a home NAS).

That is why I used ext4 for my SnapRAID filesystems. SnapRAID handles bitrot protection, I don’t need two layers doing the same, and I don’t need snapshots for my video files - there was no reason for adding complexity. For the regular data, the balance was different. I decided that self-healing of btrfs is worth the additional hassle of setting up something I’m less familiar with.

Green IT

Tue, 07 Jul 2026 00:00:00 +0200

I try to keep my IT usage reasonably environmentally friendly. Not in an extreme way - not using the computers at all would be the best for the planet, but quite impractical - but I do think about the footprint of what I run, and I think most of the community talks about the wrong part of it.

Energy usage is the secondary concern

Whenever “green IT” comes up in self-hosting circles, the conversation almost always goes straight to electricity. People tend to focus on how little power their new hardware uses, but forget about the impact of manufacturing. It takes a lot of energy to make a computer, not to mention scarce resources. Recycling old products is mostly non-existent. Or if it’s ever done, it’s usually done in the cheapest, but harmful ways, e.g. burning plastic to keep the metals. Companies often advertise their products as environmentally friendly due to reduced energy usage, but conveniently forget about those other costs.

Note: power usage is secondary, not irrelevant. Especially if you’re running the device 24/7. I’ll get back to it later.

Reuse what you already have

Before buying anything, I look at my pile of old hardware. An old laptop that’s too slow for daily use is often still perfectly capable of being an experimental machine. An old Android phone or tablet, too slow for a multi-purpose device or with a dead battery, still has a microphone, good quality camera, high-DPI screen and a multi-core CPU in GHz range. You can repurpose it as a dashcam/security camera/wildlife camera, a wall-mounted dashboard for a smart home or a media player.

Buying second hand

When I do need something I don’t already have, second hand is the default, not new. There’s a market of off-lease laptops - mostly business-class devices such as ThinkPads and Dell Latitudes. “Business” in that case means dull colours, a non-gaming GPU, but a good build quality. They tend to have fewer mechanical problems than “home” series laptops. Corporations replace their hardware every few years to reduce their operational costs, but at home there’s a different economy. Older hardware is more at risk of failing, but that’s acceptable. If you continue using your hardware until it breaks and buy second-hand, you’ll end up with a mixture of different manufacturers and technologies. A logistical nightmare? Not really, why would I care if my service is running on a Lenovo laptop, Dell terminal or Raspberry Pi board if it’s running exactly the same? There is some value in unification, e.g. it’s convenient if your laptops share the same power bricks, or if you can scavenge one computer to repair another one. But don’t sweat too much about it.

Don’t go too old

That doesn’t mean any old hardware will do - there’s a balance. A 20-year-old machine will have a fraction of the processing power of a newer one and you would likely end up using several of them. The risk of failure is high and parts are hard to buy. At some point the electricity and hassle cost outweighs the manufacturing cost you’re avoiding.

Specifically, for a daily-use laptop, I look for one powered by USB-C. It’s really convenient to use the same charger for laptops, phones and 90% of other rechargeable devices. Many, but not all, laptops made after 2018 have one. For experimental devices that I only use at home, I might accept a proprietary barrel connector, but not for the one I travel with.

If you use Windows, remember that Windows 11 requires a newish CPU (8th gen Intel Core, AMD Ryzen 2000). There’s a lot of compatible used devices available (most laptops from 2018 onwards), just be aware when buying. And if you use Linux, that means you can buy a perfectly usable laptop with a 7th gen Core really cheaply.

The same balance applies to routers and switches. A second-hand enterprise switch or router that’s around five years old could be a good buy - it’ll run current firmware, handle the speeds you actually need at home, and draw a sensible amount of power. A switch or router from 20 years ago is a different matter: it doesn’t support current protocols, it draws more power for a fraction of the throughput, and the vendor stopped shipping security updates a long time ago.

What’s actually worth buying new

PC Power supplies. They don’t last as long as most computer hardware. They have fans and large capacitors which wear down with use, so an old PSU might not have much life left. Then there’s an issue of power efficiency. Old PSUs were horrible when running at low power, when the computer needed 40W, they drew twice as much. New ones are usually certified, 80+ means they are at least 80% efficient. Higher certifications such as 80+ Gold or Platinum are even better, but also considerably more expensive, so do your maths - if they save you $10 per year but cost $100 more, that doesn’t look like a good deal.
Hard drives. Hard drives have a limited lifespan. Considering that their platters spin at 5400 RPM or more, their heads fly just some nanometers above them and have to be positioned with an almost atomic-level precision, it’s a wonder they work at all! But inevitably they fail. If you can get them very cheaply (free is a good price) and are prepared for failure (you should be anyway, even with new drives) then go ahead. There’s also a matter of capacity. When you need 2TB of storage, would you rather use a 2TB drive or 10x 200GB? Multiple disks take more power and use precious space in the case. Anything smaller than 1TB is probably useless for a NAS. Though an old and small disk might still be useful for an experimental device.
Flash-based storage. Such as SSDs, SD cards, thumb drives. You need to be even more careful than with HDDs. While they don’t wear down mechanically, they have a limited number of writes. An old SSD might be already useless. And unlike HDDs which usually show some bad sectors but continue to work, they often fail rapidly.
Batteries. Batteries degrade even sitting unused on a shelf. When you buy a used laptop, it always has reduced capacity. It could be acceptable. But when it isn’t and you want to replace the battery, get a new one. It can be a 3rd party, doesn’t need to be original - as long as it comes from a trusted supplier.
Small components. I wouldn’t bother trying to find a used SATA-to-NVMe adapter or a 3.5" to 2.5" HDD caddy. It’s not worth the hassle and I doubt it’s even possible in most cases.

Reducing power usage

Power usage is the second concern after manufacturing impact. Many self-hosters choose single-board computers such as Raspberry Pi for their projects. Some models of Pi Zero use below 1W. They’re a good choice if you need a single-purpose device.

But you will need proper PCs for some stuff. Companies discovered the value of server consolidation in the early 2000s. Virtualization and later containers allowed running multiple separate workloads on one server. For this reason, you might want to stop using very old hardware even if it still works: it’s cheaper to run 2 reasonably modern PCs than 10 very old ones.

One way to use old or low-power hardware (or fewer computers) is to choose your software carefully. 20 years ago computers were used in a very similar way despite being considerably slower. Sometimes you don’t have a choice, because the company standardises on one product. Or maybe the software you’re choosing has high hardware requirements, but also some features you really need. That’s OK. But especially for server software that you only configure once and then it runs quietly in the background doing its job, there’s sometimes an old alternative that needs just a few megabytes of RAM and a new, trendy one that takes 2GB.

Proper configuration matters, too. Modern CPUs scale down when not in use and that should happen automatically with any Linux or Windows system unless you happen to have some buggy combination of hardware, firmware, drivers and OS.

You might also consider powering down HDDs when not in use. Spinning up and down reduces the lifetime of disks, but staying on all the time also has some impact. If you power cycle your HDD every 5 minutes you will kill it in a few months, but one cycle per day shouldn’t be worse than always on. Disks in the home NAS are often unused for hours or days in a row (especially for larger setups with multiple disks). Low power devices might run from SD cards or thumb drives in some cases.

A regular server in the office or datacentre runs 24/7, but that isn’t always necessary at home. You’re not going to use your NAS when you sleep or are away from home, right? It might use some of the time to do backups or other maintenance tasks, but probably doesn’t need that much time. Consider powering up the server only when required to save power, at an inconvenience of having to wait a few minutes when you need it. It’s a tradeoff with different answers for different people.

Choosing computers for self hosting

Raspberry Pi

A default choice for many self-hosters. It’s small and uses very little power. But it’s never been cheap for the processing power you get - an old PC can do the same work as a stack of Pis, but costs next to nothing. But sometimes you need a separate machine and a small form is an important factor. I have two Pis and could certainly use more.

Other single-board computers

Some producers also use Pi in their names, e.g. Banana Pi or Orange Pi, to jump on the bandwagon, but be careful, they have nothing in common with the Raspberry Pi. Others don’t, I think that’s more honest. I experimented with a few and the results were mixed. They offer similar or sometimes better hardware for a fraction of the price, but also get a fraction of the support. Whatever you try to do on the Pi, someone already did that and wrote a blog or posted a video tutorial. On another board, you’re usually on your own. Raspberries will get the newest software in a few days and it’s going to be tested by thousands, on other boards you have to use a 3-year-old version or compile it yourself. If that’s OK for you then go ahead, if you’re new to Linux, you better stick to Raspberry Pis.

Dell Wyse thin clients

They are often cheaper than single-board computers and have similar processing power. They have Intel Atom processors or AMD equivalents, 1 or 2 GB of RAM, some kind of storage such as SD, MMC or even a small HDD. They need a few watts, depending on the exact hardware. Their intended usage is a terminal, usually they run a built-in operating system called ThinOS whose only job is to connect to a virtual desktop server such as Citrix. But it can be replaced with Linux.

Old laptops

You probably already have a few. If not, you can buy them really cheaply. Most only take about 5W when idle and maybe 40W under high load. A broken screen or keyboard is not an issue for server usage. A battery, even if it lost 95% of the original capacity, is useful as a built-in UPS: it’s still good for a few minutes, enough to move around the house or shut down safely during a power outage.

The downside of a laptop is its inconvenient form. If you need to run several at a time, they take up a lot of space if you keep them open - and many will overheat if you run them closed (some don’t mind and you can just pile them). Laptops are hard to repair, especially old ones. If you have several of the same type and know which way to hold a screwdriver, you might use some as organ donors. Or you might get unlucky and all of them will need the same part.

Small form factor PCs

For example Lenovo ThinkCentre. They share many traits with laptops. Just like them they are small, hard to upgrade, moderately power efficient and can run standard PC software. Generally cheaper than laptops when comparing the same generation hardware, easier to fit several in a small space, but lack the built-in UPS functionality.

Proper PCs in tower cases

They have one great feature: the ability to upgrade or replace hardware. You probably need one to use as a storage server (unless you prefer an off-the-shelf NAS). Do you need one for your daily work? If you want some serious processing power, you can get a powerful PC for 1/3 of the price of an equally powerful laptop.

Second-hand servers

I don’t recommend them, unless you know what you’re doing. It’s a surprisingly cheap way to buy really powerful hardware - you can buy one with multiple CPUs, huge RAM and a rack-mountable case for the price of one Pi. But that’s because nobody wants them at home. Some old servers take a few hundred watts even when idle and possibly more than 1kW under load. That’s about the same as a hairdryer, no wonder they make a similar noise! You might consider one if it’s only going to be on occasionally, e.g. if you need a powerful machine for rendering videos or compiling. Having said that, some newer servers are more efficient. Another problem with servers is hardware compatibility: you can get replacement parts for standard PCs everywhere, servers have proprietary power supplies, need hard drive caddies etc.

Creating a photo gallery with Hugo and Mapael

Mon, 06 Jul 2026 11:30:00 +0200

The photo gallery was my initial reason for switching from Pelican to Hugo. Pelican didn’t have a good gallery theme. Plus, customising themes in Pelican was hard, and I had a specific goal in mind.

Years ago, I posted my photos on TrekEarth, but this site is no more. I missed it, so I thought about publishing my travel photos first and having a map interface. Of course, I wouldn’t get the community this way, but you can’t have everything.

So I decided to give a chance to another static generator, Hugo. I considered it before, I chose Pelican because it was written in Python (but I never touched the source code, so it’s irrelevant) and I perceived it to be simpler to use (maybe, not sure about it now).

Getting started with Hugo Gallery

I looked at available themes and chose one aptly named “Gallery”, created a repo on GitHub and initialised the site with:

hugo new site web-gallery --force # force required because the directory was not empty - it contained git files
cd web-gallery
git submodule add --depth=1 https://github.com/nicokaiser/hugo-theme-gallery.git themes/gallery

Now some configuration in hugo.toml. I copied it from the example website with obvious modifications:

baseURL = 'https://photo.too-many-machines.com/'
languageCode = 'en'
title = 'Photo'
theme = 'gallery'
copyright = "© Igor Wawrzyniak"
disableKinds = ["taxonomy", "term"]
defaultContentLanguage = "en"
enableRobotsTXT = true
timeZone = "Europe/London"


[params]
 title = "Photo"
 defaultTheme = "dark"

[author]
 name = "Igor Wawrzyniak"

[outputs]
 page = ["HTML"]
 home = ["HTML", "RSS"]
 section = ["HTML"]

[imaging]
 resampleFilter = "CatmullRom"
 quality = 75
 [imaging.exif]
	disableDate = false
	disableLatLong = true
	includeFields = "ImageDescription"

[module]
 [module.hugoVersion]
	extended = false
	min = "0.112.0"

Adding content

Photos go into subdirectories of “content”. I created directories for countries (Poland, UK, Ireland, Belgium…) then for cities or regions (Poland/Wroclaw, Poland/Bieszczady). Unless I only have a few photos from the country and don’t plan to add more soon, in which case I stopped at the country level. But, Hugo will ignore directories until I add an index file (which could be useful - I can add some photos, but if I’m not ready, just skip creating the index file).

There are two types of pages in Hugo, branch and leaf. Leaf is the end node - a post or article on a normal website, a gallery in this case. Branch contains a list of posts/articles/galleries. They use different template files, but from the user perspective the most important thing to remember is: for branch you need to create an _index.md file, for leaf it’s index.md without an underscore.

The only relevant part of the page is the front matter - the metadata section on top. It can be written in several formats (YAML, TOML, JSON and ORG). Again, I followed the examples from the Gallery theme:

---
title: Belfast
linktitle: Belfast
description: In 2005, when I lived in Dublin, I made a weekend trip to Belfast. It made a huge, if not always positive impression on me. Several years after the violence subsided, you could still see the marks it left. Barbwire everywhere, schools looked like prisons and police stations looked like fortresses. Murals praising terrorists from both sides - although some peaceful murals began to appear.

---

That’s it, all my index pages look like this. Now I can type “hugo server” and click the local URL to test.

Creating the map

Instead of a list, I wanted to have a zoomable, clickable map on the main page. Front end development is not my speciality, I didn’t even know what software to use, so I asked ChatGPT (today I would use Claude). It told me to use Mapael library and generated HTML and JavaScript for me. The code was almost correct (imports didn’t work and the HTML was missing one important div) but it was easy to fix. Now I needed to replace just one page in my generated website with my custom HTML. I had to learn a bit more than I intended about the way Hugo templates work, but it wasn’t so bad and might come in useful in future.

First thing to understand, all files from /themes/gallery/layouts can be overridden by files in /layouts - no need to replace the whole theme. Second, the main page by default uses the standard template for a branch page. But, if a template for “home page” is present, it takes precedence. So, I created a file /layouts/home.html.html (yes, double extension) and for the first attempt, I just put my crude HTML in there. It worked! Now, two things left to do: improve my map (set proper initial zoom, colours, add links to countries) and replace the manually created page with a proper template. The goal was to add extra imports in the head section (for mapael JavaScript files) and replace the standard gallery list with my map, but keep the rest (menu, footer, CSS) for the consistent look.

I ended up creating several more files (everything’s on my GitHub):

I put all Mapael JavaScript files in /static
in the same place I put map.js with my map configuration
/layouts/partials/mapael_head.html contains all the code from the theme’s /layouts/partials/site_head.html plus my extra JavaScript imports (not perfect, if the file changes in the theme I’ll have to manually replace it, but the theme didn’t provide any hook for custom imports)
/layouts/home.html.html is based on the theme’s /layouts/_default/baseof.html, but instead of importing site_head.html it imports mapael_head.html and instead of “block main” it has mapael divs.

In the end, I got a world map that’s zoomed on Europe (if I travel somewhere further, I’ll just have to zoom out), that automatically adjusts to the screen size, the countries with galleries are clickable and marked by a different colour than the rest.

Configuring email: MX, SPF and DMARC

Mon, 06 Jul 2026 03:00:00 +0200

The mikr.us post mentioned that Outsider bundles a free mail server. But these days, if you just run the mail server without setting a few special DNS records, few servers will accept emails from it. And definitely not the big ones like Gmail and Outlook. We can all thank spammers for it.

A note on record types

I’m assuming you’re using some kind of admin panel for DNS (if you can edit zone files directly, I envy you). In most places, they default to A record when you add new entries. Make sure to choose the right type, MX for the first one, TXT for the other two.

MX record: where mail is delivered

Other mail-related DNS records are a relatively new addition (early 2000s for SPF and DKIM, 2012 for DMARC), are intended to prevent spam and viruses and are still optional. MX has been with us since ancient times (1987), it’s required and it has a different purpose: it tells the world which server accepts mail for your domain.

In my case, the mail server is outsider.mikr.us and priority can be anything. A domain can have multiple mail servers, in which case the priority field decides which one is main and which are fallback.

SPF: saying who’s allowed to send as you

SPF (Sender Policy Framework) is a TXT record listing the servers that are allowed to send mail claiming to be from your domain. Receiving mail servers check it. If a message claims to be from you but comes from a server not on the list, it can be flagged as suspicious (and likely classified as spam by the mail client) or rejected. Without an SPF record at all, there’s no check - anyone can send mail pretending to be you@your-domain.

Outsider’s panel shows the exact value to publish. For my VM, it looks like this: v=spf1 a mx ip4:95.217.59.141 ip6:2a01:4f9:4a:1384:0:0:0:2 ~all. Check with your own provider, you might need something else if you use mikrus, you definitely need something else if you use a different server.

Breaking that string down: a and mx allow the server(s) pointed to by your domain’s A/AAAA and MX records to send mail, the explicit ip4/ip6 add the VM’s addresses directly, and ~all means anything else, not from the list, should be soft-failed (recommends the receiving servers to treat the mail as suspicious but don’t reject it). Soft-fail is a safer option while testing, when you’re reasonably sure your setup is correct, change ~all to -all which means hard-fail: ask the receivers to reject all mail that doesn’t come from your servers.

Note: if you want to run your own mail server on the VPS and use this domain as the “from” address, don’t forget to add its IP to SPF.

Checking if it worked

Give DNS a few minutes to propagate, then check both records resolve:

dig +short MX your-domain.com
dig +short TXT your-domain.com

One thing to watch for: dig results can lag behind reality if your local resolver cached an earlier, empty answer. If a record you just added doesn’t show up, try querying a public resolver directly (dig @8.8.8.8 ... or dig @1.1.1.1 ...) before assuming something’s wrong.

You should see outsider.mikr.us for the MX query and several TXT records, one of them being v=spf1 .... That’s enough to get basic mail flowing with a minimal anti-spoofing check in place.

You can also use one of many services that check mail security. I don’t have any preferred one, I just search for “check mail domain” whenever I need it. Or, if you use an AI agent such as Claude, you can ask it to check your domain and recommend further steps.

DMARC: policy and reporting

DMARC builds on SPF (and DKIM, but I don’t have it) by telling receiving servers what to do when a message fails those checks, and where to send reports about it. It’s just another DNS TXT record, published at _dmarc.your-domain.com, so you can add it yourself regardless of what your mail provider supports.

Rather than setting up my own mailbox to receive the aggregate reports, I used Cloudflare’s built-in “Email DMARC Management” feature, which generates the record for you and collects the reports on its own domain:

v=DMARC1; p=none; rua=mailto:c45b1153cbda491795b7b2f1c5d429f4@dmarc-reports.cloudflare.net

p=none means monitor-only for now - nothing gets rejected or quarantined because of DMARC, it just starts collecting data on who’s sending mail as your domain and whether it’s passing SPF. Once I’ve seen a few reports and I’m confident nothing legitimate is failing, I can tighten it to quarantine and then to reject.

DKIM: the missing piece

DKIM (DomainKeys Identified Mail) works differently from SPF. Instead of listing which servers are allowed to send your mail, the sending server cryptographically signs each outgoing message with a private key, and adds a DKIM-Signature header to it. The public half of that key is published in DNS, as a TXT record at selector._domainkey.your-domain.com (the “selector” is just a name your mail provider picks, so it can rotate keys later). Receiving servers fetch that public key and verify the signature matches - which proves two things: the message really was sent by something holding your private key, and it wasn’t altered in transit.

That second part is what SPF can’t do. SPF only checks where a message came from; it says nothing about the message itself, and it breaks entirely when mail is forwarded through a third party (the forwarding server isn’t in your SPF record, so the check fails even though the message is legitimate). A DKIM signature travels with the message itself, so it survives being forwarded and still verifies correctly at the final destination.

Unfortunately, there’s no DKIM available for outsider.mikr.us, or at least I wasn’t able to find it in the docs.

Consequences of not having it set up:

Forwarded mail is more likely to be rejected. DMARC passes if either SPF or DKIM passes. If SPF ever fails (say, because a message got forwarded), the message fails DMARC entirely, with no fallback.
Slightly worse spam-filtering reputation. Big providers (Gmail, Outlook) use DKIM as one more signal when scoring incoming mail. Having only SPF and DMARC isn’t disqualifying, but it’s one less thing working in your favour.
No tamper-evidence. Without a signature, nothing proves the message body wasn’t modified between leaving the sending server and arriving at the recipient’s inbox.

None of that is catastrophic for a low-volume personal domain like mine - SPF plus DMARC already stops trivial spoofing, which was the main goal. But DKIM is the piece that would close the gap.

Cloudflare: webmaster's little helper

Mon, 06 Jul 2026 01:00:00 +0200

It’s not exactly self hosting. It’s very much against the spirit of self hosting. But I put Cloudflare in front of my external services: for one important reason and several small conveniences.

The good thing is the free tier service is more than enough for amateurs experimenting with personal websites and apps. In fact, it’s enough for many commercial applications.

The main reason: fixing the IPv6 problem

My photo gallery lives on mikrus, which has no public IPv4 address - only IPv6. Cloudflare’s reverse proxy makes it easy to reach it from any standard, IPv4-only client. It might sound complicated, but it really took only a few minutes. Here are the steps:

Register a Cloudflare account if you don’t have it yet.
Login to Cloudflare control panel, choose “Add domain”.
Next, login to the domain registrar (OVH in my case) and set the nameservers to those provided by Cloudflare. Note, I had problems with DNSSEC, can’t confirm it now but probably it would be easier to disable DNSSEC before the DNS change.
Cloudflare will automatically fill some of the DNS zone, not everything though (zone transfer is disabled by default). In my case I had proper records for the main domain + www + MX, but it was missing subdomains (advent, selfhosting…) so I added them manually. Make sure proxy is disabled initially, the goal is to recreate the old configuration first - don’t introduce too many modifications at the same time.

DNS change might take some time to propagate (minutes, hours, depending on how you set the TTL). But if all went well, you won’t notice any downtime, since old servers provide exactly the same information as new servers. Now, if you’re sure the domain’s DNS is now served by Cloudflare (check with host -t NS your-domain-name), here comes the best part: I added “photos.too-many-machines.com” AAAA record with IPv6 address of my VPS and this time turned the proxy option on. The magic happens in the background:

I only declared AAAA record, but Cloudflare added both AAAA and A (IPv6 and IPv4),
and they don’t point to the address I provided, they point to one of Cloudflare servers,
which will proxy the connection and connect to my VPS on IPv6 while serving clients on both protocols.

Nice things on top

Once I moved my domain to Cloudflare DNS, a few other things come along for free:

TLS termination. Cloudflare gets you a certificate automatically and terminates HTTPS at the edge. I don’t need to run Certbot or renew anything on my own server for the domains behind the proxy.
DNS management. The dashboard is simple, changes propagate fast, and it’s one place to manage records for every domain, regardless of which registrar I bought it from.
Reverse proxy for multiple sites. I later enabled the proxy option for all other subdomains. Proxying works very well for static websites. My websites load faster and I use much less of Outsider’s bandwidth quota.
DDoS protection. I doubt anyone is seriously targeting my personal blog. But these days, some bots (e.g. for training new LLMs) scan websites so aggressively that they could unintentionally DDoS my tiny VPS.
Web Application Firewall. Irrelevant for static websites, useful if you want to run web applications. Cloudflare will inspect the traffic and stop known attack patterns.

Downsides

It’s not all upside, though:

You’re relying on an external service. Have a way out in case Cloudflare goes out of business (unlikely) or terms of service change (a bit more likely). In my case, the photo gallery would be the only problem, everything else could be reverted in a few minutes.
You’re trusting a third party with your traffic. TLS gets terminated at Cloudflare’s edge, which means Cloudflare can see your traffic in plaintext before it re-encrypts (or doesn’t) the connection to your origin. For a personal blog where all the content is supposed to be publicly available anyway, that’s not an issue. For a real app, it might be.
One more outage point. If Cloudflare has a bad day, your site can go down even though your own server is perfectly healthy. Cloudflare is very reliable, but it had a famous outage in 2025.
Occasional edge-case weirdness. Once traffic goes through a caching proxy instead of hitting your server directly, it could be harder to debug a problem with your app. Never happened to me, but it is a possibility.

Self hosting on the internet: mikr.us

Mon, 06 Jul 2026 00:00:00 +0200

If you read the rest of this site, you probably noticed a clear split. Everything under “Self hosting at home” runs on my NAS and stays on the local network - no exposing it to the internet. That’s on purpose. The services don’t need to be reachable from the outside world. And every open port is one more thing that can be attacked.

But there are services that I want to live on the internet: this website, for example. My photo gallery, some experimental apps. For that, you need a VPS: Virtual Private Server. A virtual machine which you can configure to your needs and easily recreate if you break something.

These days, many admins automatically think of AWS, Google Cloud Platform or Azure when they need a VM. They have nice features such as replication and auto-scaling. But, if you need exactly one VM running 100% of the time, traditional hosting providers are much cheaper.

Why mikr.us

mikr.us is a Polish VPS provider aimed at learning and experimentation. It shows in two things: pricing and Service Level Agreement:

A whole year costs about as much as a single month (sometimes just a few days) of a typical cloud VM. I pay 75 PLN a year, which is roughly 13 GBP / 15 EUR / 16 USD.
For that price you obviously don’t get an enterprise SLA. But in practice it has been about as reliable as the more expensive services I’ve used.

Note that you can run commercial services from it, if you like. I know that many companies use it, especially for testing or to run monitoring scripts.

There are also some quirks you have to accept:

The server only has an IPv6 address. Reaching it over IPv4 needs port redirection (you automatically get one for SSH and can configure some more, all on high ports), or a proxy in front of it if you want standard ports and your own domain (which is easier than it sounds and explained in the next post).
The resources are modest, this is not where you run anything resource-hungry.
You can’t run anything other than Linux and you can’t choose the kernel - it is shared.
Service agreement specifically prohibits several applications that put too much load (media streaming, some game servers), plus the usual illegal or grey-area activities.
Support is community-driven rather than enterprise-grade.

None of that has been a real problem for me. It’s meant to be a bit of a hands-on, DIY experience, and you end up learning something in the process - which is exactly what I wanted.

Outsider

A useful feature of mikr.us is called Outsider. It’s an extra service bundled in for free, and it gives you an easy, ready-to-go shared hosting setup: web server, PHP support, databases, mail server, automatic SSL certificates - all without touching your own server configuration.

It works like a charm for static content (and would work fine for small PHP apps too), and it’s where I keep most of my websites. This one and a few other small pages. The real limits are 500MB of space and a fairly low bandwidth quota, which is more than enough for text and a reasonable number of images, but not for something like a photo gallery with hundreds of full-size photos. That’s the one thing I run on the actual VPS instead, behind Cloudflare, exactly like I described in the photo gallery post.

Other conveniences

There are loads of them, I haven’t even used half.

Strych (attic) is a backup solution where you can rsync files from your mikr.us VM
NOOBS is a set of scripts for installing and configuring many popular applications
A shared instance of WireGuard if you need a VPN but don’t want to configure your own
Pusher sends mail notification to the VPS owner
M.A.R.I.A.N is a tool for diagnosing problems with the VPS
Amfetamina is a temporary boost (more RAM and CPU) e.g. when compiling software
Comes with a free subdomain but you can assign your own domains

One inconvenience

The registration site is only in Polish, so is the documentation and the main VPS admin panel. These days, when all browsers can translate websites, it’s not a deal breaker.

Outsider admin panel, where you’d probably spend more time, has multiple language versions. And if you reach out to the community (on Discord or Facebook) someone will surely answer in English.

Getting started with mikr.us

Convinced already? Go to the mikr.us website (full disclosure: it’s an affiliate link, if you use it I’ll get 2 months free) and select the version you need.

Frog is an almost-free option - only a one-time payment of 5 PLN (that’s about $1 or €1), but it’s very limited
Mikrus 1.0 has only 384MB of RAM and can’t run Docker - still, a good choice for a single-purpose machine, e.g. to run monitoring scripts.
Mikrus 2.1 is a sweet spot for me: 1GB of RAM and 10GB of disk space is enough for a Linux server. I used to run commercial services on less than that.
If you need more, there are higher plans available, up to Mikrus 4.2 with 16GB RAM and 160GB disk space.

Register, pay and in a moment you’ll have your machine up and running. You’ll get your initial password in the welcome mail and the admin panel will show the exact SSH command (you need to connect on a high port, not the usual 22).

What’s next? It’s a Linux server, you can do anything with it. I’ll show some ideas in later posts. Remember to keep it updated and backed up.

Generating websites with Hugo

Sun, 05 Jul 2026 12:00:00 +0200

Static Website Generator vs. Content Management System

If you’re a dinosaur like me, you probably wrote your first websites manually, in HTML. In the 1990s and early 2000s, that was the default choice. Then came Content Management Systems, mostly based on PHP and MySQL: many competing systems first, until Wordpress dominated the market. Everyone shifted to them.

I want to convince you to go back from server-generated pages. But it doesn’t mean you need to edit HTML manually like it’s 1998. Static website generators use a different concept:

your content lives in plain text files (usually Markdown)
after editing a post or adding a new one, you use the generator to “build” the site
it will then combine your content with the theme (that decides how you site looks) and configuration
and will output HTML, CSS and images, that you can upload to the server

The build and upload phase can be automated, if you prefer.

Who is this guide for

I’m assuming you’re familiar with CMSes. It’s not really necessary, but I’m going to compare the two approaches. I’m also assuming you’re not afraid to work with the terminal. But it doesn’t mean you need to be an IT professional (though if you are one, everything here will be easy, if you’re not, it might take some effort).

Static websites often go together with git. If you’re familiar with it, I highly recommend to put your contents under version control. If you’re not, you can skip this part, it’s not required.

Or let the AI do it

The first time I used Hugo, I did all the steps below manually. This time I used Claude. It turned out it could competently handle most of the steps on its own: install the theme, change configs, rebuild, check the result, correct if needed. It worked especially well for migrating from Pelican to Hugo, since it required some boring steps: converting front matter and fixing image paths. I also used it to check for typos and other language errors (English is not my first language).

That doesn’t mean you can skip reading this guide, though. Even if you don’t run the commands, you need to understand the concepts. AI agents make mistakes and need pointing in the right direction.

Advantages over CMS

Security - Since there’s no server-side processing, there are far fewer ways to attack your website.
Server availability and cost - Again these come from no server-side processing: every web server can easily serve static pages. Which means you can use any cheap hosting provider and don’t even need to check what features are available.
Speed - On a CMS, the page is generated the moment someone accesses it, which takes time - could be a few seconds or more for a complex page on a busy server. Serving static files skips this step entirely.
Maintenance - There’s next to none. If you ever used WordPress, you know you need to constantly update your plugins and themes and it always introduces incompatibilities. With a static website generator, you can just continue using your setup as-is. (Fine print: you probably want to update the theme and the generator itself occasionally to get new features, but it rarely requires any changes to your config or content.)

Ease of use?

CMS is easier to use, at least once someone set it up for you. Everything is from an admin panel on the web. Website generators have a steeper learning curve.

But they’re only hard in the beginning. Once you’ve learned the concepts and configured it, adding more content is easy.

Pelican vs. Hugo

I had used another website generator in the past - Pelican. In 2023 I wanted to build a photo gallery and discovered its limitations. There are few people maintaining the app and creating themes, the themes themselves are hard to customise. I used Hugo for my gallery and it worked perfectly. Next time I needed to create a site, I used Hugo too. I’m now in the process of migrating my old websites.

Step by step guide

Some new vocabulary

Markdown: a simple, text-only file format that offers some formatting: headings, bold/italic, tables etc. Used by many static website generators (plus GitHub, Jupyter and many other places, if you’re in IT, you use it everyday).
Front matter: a block of metadata at the top of a content file. Here you set title, date, tags, draft status etc.
Theme: a separate package of layout and styling files that decides how your content looks, without touching the content itself. Swapping themes shouldn’t require rewriting your posts.
Archetype: a template used when you create a new page, so it starts with some sensible front matter already filled in.
Shortcode: a small reusable snippet you can call from inside a Markdown file for things plain Markdown can’t do on its own, such as embedding a video.
Page bundle: probably the concept that trips up new Hugo users the most, so it gets its own section below.

Installing Hugo and creating a site

I use Debian on my home laptop, so a simple sudo apt install hugo is enough to install. Other platforms have their own packages. Or you can download from the project’s website - Hugo ships a single binary. Make sure you get the “extended” edition if your distribution offers a choice - some themes need it for SCSS support.

Once installed, creating a new site is one command:

hugo new site my-website
cd my-website

This generates a skeleton directory: content for your pages, layouts for any template overrides, static for files copied verbatim (favicons, robots.txt), archetypes for new-page templates, and a configuration file (hugo.toml by default, or .yaml/.json if you prefer).

Choosing a theme

Browse the theme gallery on the Hugo website and choose one. There are several ways to install the theme, the recommended one is to use git submodule:

git submodule add https://github.com/some-author/some-theme themes/some-theme

Then point your configuration file at it:

theme = 'some-theme'

Most themes ship an exampleSite folder with a sample configuration - copying the relevant bits into your own hugo.toml will save you a lot of guessing about which parameters the theme actually reads.

Configuring the site

At a minimum, your hugo.toml needs a baseURL (the production URL of your site), a title, and the theme name. This is where you also configure what your sidebar/top menu will contain and hook up external systems (e.g. for comments). Here’s how hugo.toml for this website begins:

baseURL = 'https://selfhosting.too-many-machines.com/'
locale = 'en'
title = 'Self Hosting'
theme = 'hugo-theme-stack'
summaryLength = 40
copyright = "© Igor Wawrzyniak"
defaultContentLanguage = "en"
enableRobotsTXT = true
timeZone = "Europe/Stockholm"
disableKinds = ["section", "taxonomy", "term"]

Most variables are theme-specific and live under [params]. The first time I created a Hugo website, I spent some time reading the docs, next time I simply copied the file and changed what was relevant.

Page bundles: keeping content and images together

In many older static site setups (Pelican included), your Markdown files live in one directory and your images live in another, and you glue them together with a path like {static}/images/photo.png. In Hugo, another approach is possible: every page is a directory, containing an index.md plus all images and other files that the page needs.

content/internet/hugo/
 index.md
 screenshot.png

To use the image in a page, use just the filename - ![Screenshot](screenshot.png) - no need to know the final URL. Hugo will take care of it. It will also automatically generate resized versions of your images.

This kind of bundle (one page, no children) is called a “leaf bundle”. A “branch bundle” is the same idea but for a section front page - a directory with an _index.md that can have further pages underneath it, such as a section listing page.

Adding your first page

You can create a new page bundle with the command: hugo new content posts/my-first-post index.md.

This creates the file with front matter based on archetypes/default.md. By default it has draft: true setting in the front matter so it won’t accidentally get published. Open it, write your content in Markdown below the front matter, drop any images into the same directory, and change draft to false when you’re ready to publish.

You don’t have to use this method though. You can just create a directory and write index.md by hand, or copy and modify an existing page, whatever is convenient for you.

Previewing and building

Hugo has a built-in development server that runs on http://localhost:1313/. It rebuilds and refreshes your browser automatically when you save the markdown file.

hugo server

When you’re done, generate the final static files with a simple hugo or hugo --minify to reduce size of CSS and scripts. This writes everything to the public directory, ready to be uploaded to your web server. You can automate building and uploading the website e.g. using a Continuous Deployment system, but that’s a story for some other day.

Sidenote 2 - Ansible fact gathering

Fri, 22 Mar 2024 17:40:00 +0100

If you look closely at the playbook’s output, you will see that it always begins with “Gathering facts”. It takes a few seconds, which annoys many users - especially when you’re developing and therefore running the same playbook many times.

How does it work?

Ansible runs a special built-in task at the beginning of each playbook. Facts are then available as variables. You can print them all with a simple playbook:

---
- name: Print all available facts
 hosts: localhost
 tasks:
 - name: Print all facts
 ansible.builtin.debug:
 var: ansible_facts

Or even simpler: ansible <hostname> -m ansible.builtin.setup.

Why?

Ansible can manage Linux, other Unix-like operating systems, Windows, network devices and a plethora of other things. Even if you stick with Linux, there’s a huge difference between RHEL 6.0 and Ubuntu 23.10.

You can use facts like any other variable in your playbook or template. Some facts are more useful than others. For example I often use ansible_distribution_release when adding apt sources.

You can even gather and use facts about one system when configuring another system. For example, you can get the IP address from your DB server and use it when configuring a DB client.

Disabling fact gathering

Sometimes you don’t need to gather facts. Be careful though, even if you don’t use them directly, your Galaxy roles or built-in tasks might rely on them. But there are some cases when you can be 100% sure you won’t need the facts. For example, when working with clouds you then specify localhost as your playbook’s target, but it’s only the host that runs the cloud API client, you don’t really care about its configuration. You can then begin your playbook with:

- name: Do something on the cloud
 hosts: localhost
 become: false
 gather_facts: false

Caching facts

If you want to make your playbooks run a few seconds faster, there’s a better way than disabling fact gathering: fact caching. For more complex setups you can use the DB, but a simple file is usually enough. Add this to your ansible.cfg:

[defaults]
fact_caching = jsonfile
fact_caching_connection = $HOME/.ansible/fact-cache
fact_caching_timeout = 3600

How long should you cache facts? That depends on you and your infrastructure. If it’s highly dynamic, the machine can change at any time, but in many places servers stay the same for months. Personally, I use something between a few minutes and one hour.

Backup, part 1: IDrive, backup to the cloud

Mon, 20 Nov 2023 17:40:00 +0100

That is against the spirit of self-hosting, but keeping data safe is more important than keeping purity. Your computers can back up to your NAS (I’ll cover that in the next step), that protects you from disk failure, ransomware, accidental deletion and losing your laptop on a train. But a fire or flood can destroy both at the same moment. A proper disaster recovery plan requires 3 copies of data: 1 working copy, 1 on-site backup for quick restore and 1 off-site backup if the previous backup fails. One option is to find another self-hoster and arrange for mutual backup to each other’s NAS, another is to use a cloud service.

Cloud storage choices

Popular options include Google Drive, Microsoft OneDrive and Dropbox. They are convenient for sharing files, but you can also use them for backup. That is, if the space is large enough for you (you can buy more, but it will cost you) and you trust that your files will be private enough for your needs (best way - encrypt them before sending).

Another way is to use an object storage such as Amazon S3, Google Cloud Storage, Azure Blob Storage or many S3-compatible offerings. Some providers have a special tier called cold storage. It’s much cheaper than the standard tier, but be careful: you also pay for data retrieval. For an archive or backup that you’ll probably never need, this might not be a big issue. A big advantage of such a service is that you can use any backup tool that works with cloud storage (Duplicati, Restic and many others).

IDrive

Probably the cheapest way to store several TB of data at the time of writing this (November 2023) is IDrive. Their personal plan starts from $99 per year for 5TB, $149 for 10TB, but you can find some affiliate links with a special offer: $4 for the first year. I got mine at https://www.tomsguide.com/reviews/idrive-cloud-storage-review.

Downside of IDrive: this service doesn’t provide S3-compatible API, REST interface or any other standard way that’s supported by various backup programs. You have to use their own client. In the case of Linux, it’s a set of scripts. They have limited functionality compared to a Windows version. There’s also no way of automating installation and configuration. Good sides? It’s cheap, it’s reasonably fast, it offers pretty good security (2FA, encrypted storage and transfer), and it works.

Sending or receiving terabytes of data takes days, even on a fast broadband. IDrive gives an option of using USB drive for a restore or for initial backup. I haven’t used this option, so I can’t rate it.

Getting started

Sign up for an account. It’s also wise to turn on 2FA, preferably with a TOTP app. Now, the Windows and Mac clients are available for download, but for the Linux scripts you need to contact support. I’m not sure why, do they need to enable any special options for the account? Anyway, I asked on the live chat and got a download URL in 2 minutes. Beware of IDrive scripts you can find on the internet - I found some, they were so old they failed to connect - and even failed to auto-update.

Following the documentation at https://www.idrive.com/readme I unpacked the client to the directory of my choice. I used /opt and ran the client as root. If you don’t need to back up files belonging to different users, you can use a standard account. After unpacking you need to make the scripts executable: chmod a+x *.pl.

First script to run is account_setting.pl. It detected some missing dependencies, tried to install them - some with apt, some using Perl CPAN - and failed. I ran the script again and chose not to install dependencies, but list them instead. I tried manual installation to see what was wrong. The first Perl module tried to create a directory in /usr/local/man, but on my system it wasn’t a directory, but a dead symlink. I fixed it, tried automatic installation again - it worked. I then logged in to my account and answered a few questions - default choice should work most of the time.

First backup

Next script to run is edit_supported_files.pl. It is used for editing backup sets, exclude lists etc. I chose a reasonably sized directory (few GB) for a first test and added more directories later.

Manual backup is started with Backup_Script.pl. Remember to run it under cron or tmux if you expect your job to run longer than a few minutes. It shows progress, has an option to pause/resume backup and to set bandwidth limit.

Make sure you can restore your data. You can use Restore_Script.pl on your Linux machine or the web interface.

Last thing to do is to run scheduler.pl to schedule automatic backup. Choose your frequency (hourly/daily/weekly) and time to start. Once the scheduled job starts, you can watch its progress and pause/resume/throttle using status_retrieval.pl.

Sidenote 1 - Using Ansible

Sun, 19 Nov 2023 17:40:00 +0100

I use Ansible to configure my home computers. I started doing it to learn the tool, but is it really worth using it at home other than for education? Maybe.

On the downside, it’s generally faster to configure a server manually. The syntax is quite awkward for beginners. If you need to change a few lines in a config file, you can do it in 30 seconds with a text editor or spend an hour retrying the playbook. But it can save you time as well if you’re going to repeat the task.

There are several ways to use Ansible. One is to automate ad-hoc tasks, such as patching, gathering some information, repairing common problems. It’s quite usual if you’re introducing Ansible into a pre-existing infrastructure.

But Ansible really shines if you use the Infrastructure as a Code approach. That is, every server (or group of identical servers) has one playbook containing the whole configuration: software, user accounts etc. Parts of the playbook that are the same on different servers can be shared (see below about roles). And if you ever need to rebuild your server, you can do it with one command. It happened 3 times with my NAS (I experimented with different hardware). For more experimental devices or VMs, you can intentionally rebuild them often.

Proper Ansible playbooks are idempotent, meaning they can be run multiple times. Ansible will check the state of the server and skip tasks that wouldn’t change anything.

Think what should be automated

Since it’s a home network, I didn’t do several things I would do in an enterprise setting. The most important omission is provisioning. I just started from the base system already installed. Provisioning can be automated, but I decided not to do it. I’ve got different types of machines: Raspberry Pis, PCs, VMs, VPS. Each needs to be provisioned in a different way: PXE, writing image to the card, cloning base image or using Terraform. That would mean lots of work for not much use (usually once per machine) and not much educational value (I know how to do it). My setup does 80% of the job with 20% of the effort.

Sometimes one action you do with Ansible changes the way you interact with the server later. Example: I change SSH port on all my servers. That means I have to login with a default port on the first run, but on each subsequent run I need to use my custom port. Solution? I just manually change the port in the Ansible inventory. If I had a highly dynamic infrastructure with new machines coming every day, that wouldn’t do, but in my case it only happens once per machine.

Using linter

Playbook syntax can be hard in the beginning. A wrong indentation can turn it into an incorrect YAML. Even worse, there are some real gotchas, for example if you don’t put quotes around numeric file permissions, they might end up other than expected. A tool called ansible-lint checks all these and enforces some best practices.

In fact, the linter is really picky and you might have to ignore some rules. I’ve got this in my .ansible-lint:

skip_list:
 - yaml[line-length]
 - package-latest

Which means I allow long lines (linter thinks you should split them, I disagree) and installing latest version of software instead of a specific version.

I started using Ansible several years ago. In the meantime, the idea of how an Ansible playbook should look changed a lot. Running ansible-lint revealed many problems with my code. For example, it is recommended to use fully-qualified collection names even for built-in commands (e.g. ansible.builtin.file instead of file).

Using roles

Roles are reusable pieces of configuration, which make my playbooks really short. There are some things I want to do with all Linux machines on my network:

create a user account and install SSH key,
configure SSH server the way I like it,
upgrade packages
install some packages that I use everywhere.

Which is why I prepared a role “linux_common”. Then there’s role “linux_dev” that installs code editors, C++ compiler and some tools (I mostly code in Python, but it is already included in the base system). I want it on my laptop and on my desktop, but not on the server. Similar with the office and multimedia software included in the role “linux_desktop”.

My home server Firefly gets many roles. There’s one called firefly which contains settings specific to this machine, but there are also roles for Jellyfin and Prometheus. In future I might decide to move them elsewhere, so I’ll remove those roles from Firefly and include them in another machine’s playbook.

Ansible uses some conventions about directory structure and file naming. You can ignore them, but adhering to them will make life easier for everyone. This is an example for a moderately complex role.

The most important directory is “tasks”, it stores YAML files with all task definitions. Handlers are special kind of tasks - they are run after all the standard tasks if a specific condition was met. For example, you can restart your service if its binary has changed or if the configuration file was modified. If both events happened, Ansible is smart enough to only run the handler once. In both of these directories, only file “main.yml” is read automatically. But if it gets too long, you can import other files from it.

Directory “defaults” stores default values for variables, which are overwritten by custom values from “vars”. Last but not least, “files” contains all files that will be copied to the server without any change, those from “templates” are filled with some generated content first.

Using Galaxy

You can install roles prepared by others using Ansible Galaxy. They often save you hours of writing and testing playbooks. Some of them are so well-prepared that you can use them without changing the code, only modifying the configuration. Others though are of lesser quality, or your infrastructure is so specific that a generic role can’t solve all of your problems. In that case, you can just edit them like you would edit your own role.

Prometheus and Grafana, monitoring - part 2

Fri, 17 Nov 2023 17:40:00 +0100

For the reference, here are the default ports of various components of Prometheus/Grafana stack:

3000 Grafana
9090 Prometheus
9100 Prometheus Node Exporter
9116 Prometheus SNMP Exporter
9093 Prometheus Alertmanager

Generating node list for Prometheus

In the default config file there are already two exporters configured: Prometheus for localhost and node exporter also for localhost. Yes, Prometheus server also exposes some metrics about itself and then consumes them. The first item is OK, I only have one Prometheus instance, but I want to monitor all Linux hosts on my home network. Let’s put the default config file in the role’s templates subdirectory and add a .j2 extension (so the full name is prometheus.yml.j2). Now just change a few lines at the end of the file:

 - job_name: node
 # If prometheus-node-exporter is installed, grab stats about the local
 # machine by default.
 static_configs:
 - targets:
 {% for host in groups['linux'] %}
 - '{{ host }}:9100'
 {% endfor %}

My Ansible inventory contains a group called linux, with all Ansible-managed Linux machines. Now, to generate the config file, change tasks/main.yml by adding these lines right after package installation:

- name: Generate config for Prometheus
 ansible.builtin.template:
 dest: "/etc/prometheus/prometheus.yml"
 src: prometheus.yml.j2
 notify:
 - Restart Prometheus

We’re now only missing the restart handler - a task that will run only when “Generate config for Prometheus” changes anything. Add handlers/main.yml:

---
- name: Restart Prometheus
 ansible.builtin.service:
 name: "prometheus"
 state: restarted

Configuring Grafana

Grafana is a visual tool. You can programmatically generate the config (it’s all YAML) if you know what you want to achieve, but if you’re exploring, GUI is a better option. Login to Grafana and click on “Add data source”. Select “Prometheus” as the type, then enter the hostname (either host.docker.internal or a real external hostname, localhost won’t work) with the port (9090 is the default). Select “No authentication” as we didn’t configure any. Grafana will confirm it can read metrics from Prometheus.

Let’s get the graphs already. There are two options. You can create your own dashboard by manually adding graphs of various types and entering PromQL queries. Or you can import an existing dashboard from Grafana. The first way is useful if you need a custom dashboard to quickly see selected metrics from different sources. If you want a complete view of each node, importing a high-quality existing dashboard will save you hours. I chose “Node exporter full”, a template so famous that many template designers compare their work to it, referring to it by id (“My template is similar to 1860, but…”) and “Prometheus 2.0 Overview”, number 3662.

Where to go from here

We barely scratched the surface. Prometheus is widely used for gathering all sorts of metrics, from low-level system stats like we collect here, through HTTP/DB/other server stats, up to business metrics (e.g. click-through rate or order value from an e-commerce app) or industrial data. Grafana can plot all of these, plus it can use data sources other than Prometheus - next by popularity is probably Loki, a log aggregation system.

Then there’s alerting. Is it useful for a home network? Definitely not as much as in the enterprise, some of your hosts may be down most of the time (e.g. a laptop), but notification about low disk space on your NAS or a failed backup might be nice.

Prometheus and Grafana, monitoring - part 1

Thu, 16 Nov 2023 17:40:00 +0100

As the infrastructure grows, the need for monitoring becomes more apparent. Servers fail, and when they do, you want to be notified and you want to have information for debugging.

Every decent monitoring system is modular. It can gather metrics from a wide variety of hardware and software, it can discover new services in any environment (e.g. LAN, Kubernetes, AWS), it can send alerts in many different ways, it can interface with a ticketing system. It can form a hierarchy, where low-level systems gather all data about their part of the infrastructure and send only summary statistics to the higher level. Most of these options are not needed at home, but it’s still worth choosing one of the popular and comprehensive systems. First, because there’s a high chance you’ll use it later for your work. Second, because a popular monitoring system will likely have plugins for everything you might imagine. Some systems, despite being very advanced, are not particularly overwhelming - you can just focus on the core system and easily ignore the modules you’re not using.

Choosing a monitoring software

Some of the popular monitoring systems are surprisingly old. Nagios, created in 2002, is still widely used - mostly because many admins find it already installed and working just fine. It has a huge library of community plugins, including very rare devices not available elsewhere, that’s why many later monitoring systems can use Nagios plugins (with a varying amount of success).

Other somewhat popular and equally old (+/- 2 years) software includes Zabbix, OpenNMS and Cacti. They have some strong points too, e.g. Cacti makes it very easy to monitor network devices. But I wouldn’t recommend any of these for a new installation. Infrastructure today is very different than in the early 2000s and we need to monitor containers, web services, ML platforms and other things that weren’t even invented back then.

Why Prometheus

Prometheus was created in 2012, when cloud infrastructure was already similar to what we have today. It has all the standard features of a modern cloud-native software:

all network communication is HTTP pull,
written in Go,
configuration in YAML,
multiple independent instances can be used to reduce the risk of failure.

A data source for Prometheus is called an exporter. There’s a huge range of official exporters to choose from and even more written by the community, some popular ones include:

node exporter, gives standard hardware and system statistics (CPU, memory, disk, load avg etc.) about a Linux host,
SNMP exporter for monitoring network devices,
exporters for server hardware, sensors, GPUs,
exporters for HTTP servers and proxies which provide statistics such as latency, number of 4xx/5xx errors and many others,
exporters for databases,
exporters for cloud services (AWS, Azure, DigitalOcean, GitHub…)

Even better, Prometheus became so hugely popular that a lot of cloud infrastructure software (e.g. Docker and Kubernetes) can directly expose metrics in a Prometheus format.

Prometheus collects metrics from the configured (or auto-discovered) exporters. Then it can use another module called AlertManager to notify about an urgent problem. An important feature is inhibition of alerts. Have you ever received hundreds of notifications when one network error caused many checks to fail? Prometheus can be configured to prevent that.

Prometheus has a web interface (obviously). You can use it to write ad-hoc queries against the stored metrics using PromQL, a specially designed and easy to learn query language. You can also view graphs, but it’s not really recommended - Grafana, a visual dashboard which can get data from Prometheus, provides a much better experience.

Installing Prometheus with Ansible

There are several ways to install Prometheus. You can use a package from your distribution, you can run it in a container or get the latest binary from the project’s website. Debian and Ubuntu, my distros of choice, have Prometheus packages available. Of course, I’ll use Ansible.

But wait, there will be only one Prometheus/Grafana server, but every Linux system in my home network should run a node exporter. I already have a role linux_common which, among other things, installs a few packages I like to have on all my machines. Let’s add one thing to the list: sys_packages: [ 'curl', 'vim', 'git', 'screen', 'mc', 'lsof', 'aptitude', 'rsync', 'gpg', 'prometheus-node-exporter' ]

I’m going to run Prometheus on Firefly, my NAS. Not much choice, I only have two always-on systems: one for DNS/DHCP and one for everything else. But it might change in future, so I’ll prepare a role “prometheus_server” and firefly will include this role.

First I’m installing packages: Prometheus core, alert manager, and SNMP exporter for monitoring network hardware.

- name: Install prometheus packages
 ansible.builtin.apt: name=['prometheus', 'prometheus-snmp-exporter', 'prometheus-alertmanager'] state=latest

Let’s point the browser to http://firefly.home.arpa:9090 - it works!

Installing Grafana with Ansible

Grafana is a bit more complex, as there’s no official Debian package. I decided to use a Docker container. Here’s a Docker Compose file:

version: '3.5'
services:
 grafana:
 image: grafana/grafana-oss:latest
 container_name: grafana
 environment:
 - TZ=Europe/London
 ports:
 - '3000:3000'
 extra_hosts:
 - "host.docker.internal:host-gateway"
 restart: 'always'
 volumes:
 - /data/noshare/grafana:/var/lib/grafana

This is fairly standard, except for extra_hosts directive. Grafana in a container wouldn’t connect to Prometheus on the host. That is, it would connect with a real (not 127.0.0.1) IP, provided that Prometheus listens on something else than a loopback interface, treating it like any other external server. Usually, the whole point of using Docker is to have some isolation. But it’s a home server, I’m OK with mixing container-based and host-based workloads. After adding those two lines we can connect from the container to services running on the host using a special name “host.docker.internal”.

If we don’t configure any volume, Grafana will store data inside the container - meaning it will lose it when the container is stopped. We can either create a Docker-managed volume, or bind-mount a specific directory. I chose the second option since I want to keep the data on a specific disk. Before we start the container, we need to make sure the directory exists and has proper access rights - Grafana runs as a user with UID 472. Such a user doesn’t exist on my system, but that’s not a problem. Here’s how the complete Prometheus/Grafana role looks:

---

- name: Install prometheus packages
 ansible.builtin.apt:
 name: ['prometheus', 'prometheus-snmp-exporter', 'prometheus-alertmanager']
 state: present

- name: Chown directory for Grafana data
 ansible.builtin.file:
 path: /data/noshare/grafana
 state: directory
 owner: 472
 recurse: true

- name: copy Grafana docker-compose.yml
 ansible.builtin.copy:
 src: files/docker-compose-grafana.yml
 dest: "/home/{{ create_user }}/compose/docker-compose-grafana.yml"
 owner: "{{ create_user }}"


- name: start containers
 community.docker.docker_compose:
 project_src: "/home/{{ create_user }}/compose/"
 state: present
 files:
 - "docker-compose-grafana.yml"
 register: result

- name: print compose output
 ansible.builtin.debug:
 var: result
 verbosity: 2

Run the playbook: ansible-playbook firefly.yml --start-at-task "Install prometheus packages" and check if the container is running: docker ps. Then, point the browser to http://firefly.home.arpa:3000 - if you don’t see anything, wait a moment, Grafana needs to initialise its database on the first start.

Syncthing, synchronising files between the devices

Wed, 10 May 2023 17:40:00 +0200

Sharing files between your devices using a cloud service such as Google Drive or Dropbox is incredibly convenient. Wouldn’t it be nice to do the same locally, with complete privacy and without using your internet connection? Sure it would.

Syncthing works on Linux, Windows, macOS, FreeBSD and Android. There are also packages for commercial NASes from Synology, QNAP, WD and others. It doesn’t require a server - you can directly sync two computers, or a computer with a phone, or 3 devices can all sync with each other. That’s what I did before I got the NAS. But if you have a home server which is always on, it makes sense to treat it as a central point and have all other devices synchronise with it.

Installation

On my Linux laptops and desktop, I just installed the package and set syncthing to autostart using my user account (no need to run it as root and a dedicated account wouldn’t have access to my files):

apt install syncthing
systemctl enable syncthing@igor
systemctl start syncthing@igor

On the NAS, it was similar, but I used Ansible. This is the task file I added to the firefly role:

---
- name: Install syncthing
 apt: 
 name: ['syncthing'] 
 state: present

- name: Create directory for syncthing
 ansible.builtin.file:
 path: /data/noshare/syncthing
 state: directory
 owner: igor

- name: Restart syncthing
 service:
 name: syncthing@igor
 state: restarted
 enabled: true

I could run syncthing in a Docker container, but I don’t see much value in it. It’s just one package, data is all contained in one place and an up-to-date package is widely available for Linux distros.

On Windows I installed the SyncTrayzor package, which includes Syncthing and a system tray widget. Finally, on Android it’s just installing the package from either Play Store or F-Droid.

Configuration

Syncthing isn’t really well suited for automated configuration. It stores its settings in ~/.config/syncthing/config.xml. You could generate this file if you really wanted to, but a more sane option is to configure using the web GUI and then back up this file so you can recreate the setup if needed. For configuring your computer, just point your browser to http://localhost:8384.

It’s a bit more tricky with the NAS. By default, syncthing only listens on the local interface. You can change this option in your config file or, since it’s only needed for a moment and the default is better for security, you can set up an SSH tunnel. On my laptop, I typed ssh -L 8385:localhost:8384 firefly and pointed the browser to http://localhost:8385. How does the tunnel work? It means that ssh will forward local port 8385 through firefly to localhost:8384 - but that’s localhost from the firefly’s point of view. Note that I can open the tunnel through firefly to another server, sysadmins often use this technique to connect to servers using IPs through one jumphost available on the internet.

Back to the point. In one of your browser tabs, click “Add Folder” and choose what you want to share. In my case, I’m sharing ~/CloudStation between all my computers. The folder name is due to historical reasons, a few years ago I used Synology CloudStation for the same purpose. Don’t share your whole home directory as some config files are system-specific.

Next, click “Add Remote Device”. You can then paste the ID of another device (or scan the QR code) or very likely, you’ll see a list of other devices on the network. Remember to type a name that will be displayed in the GUI instead of the ID. Then, on your other machines, confirm you want to pair with this device. A moment later you’ll see a message: this device wants to share a folder. Click to confirm. Note that the path of the shared folder doesn’t need to be the same, on my computers I use ~/CloudStation, but on the NAS I chose /data/noshare/syncthing/CloudStation.

Syncing the phone

There are some things I like to sync between my computers - code snippets, financial stuff, home related documents - that I would never access on the phone. I decided to share only a subset of files, inside my ~/CloudStation folder, I have ~/CloudStation/na-telefon (which translates as “to the phone”). Syncthing warns that one shared folder is contained within the other, but allows such setup.

One other use for syncthing is to auto upload my photos from my phone to Elsa, my graphics workstation. On the phone, I chose to sync the camera folder with Elsa, but there’s one twist: the synchronisation is only in one direction. The phone will send files, I can view them on the computer and delete those I don’t need, the changes won’t be synced back.

Dealing with conflicts

Most of the time, nothing will stop you from modifying the same file on multiple devices at once. Some applications create lock files, those warn about opening the same file more than once and often suggest opening it read-only. Depending on your workflow, this might be a huge problem (“I’m getting conflicts all the time”), none at all (“I’m always editing this file on this computer anyway, others only read it”) or something in between.

When you do get a conflict, syncthing will keep both versions of the file, adding “syncconflict” plus timestamp and device ID. It is then up to you - and your applications - to resolve it. For example, KeePassXC will merge two DBs without any issues, text files (source code etc.) can be compared with diff, vimdiff or similar tools.

Jellyfin, a streaming server for home

Fri, 14 Apr 2023 17:40:00 +0200

We made a lot of hard work in the previous steps, time for relax. My NAS holds, in addition to other things, a small collection of videos. Mostly some old movies and TV shows not available in streaming services or my favourites that I want to keep no matter what. The PCs can easily play video files from an SMB share using VLC or another player, but mobile devices, smart TVs etc. need streaming. Plus, commercial streaming services made us used to features like easy searching, categories, recommendations, resuming on another device. Fortunately it’s easy to run your own streaming server.

There are two main contenders in this field in addition to a plethora of less popular solutions. Plex is a well-established one with 15 years of history. Recently many users are annoyed by the GUI changes, mostly done to promote Plex Pass, an optional subscription service extending features of Plex. Jellyfin is a fully open-source alternative that doesn’t include any subscriptions or outside components, it’s fully self-contained. It has fewer plugins and clients than its main competitor, but it’s growing rapidly and already supports most platforms.

Jellyfin has a client-server architecture, meaning you install server software that reads your media files and streams them to the clients, and client software on the devices you want to use for viewing. There are clients for Android, iOS, popular smart TVs and digital media players, there’s also a web client. Installing a client is a completely no-frills experience, just install it from an app store, open it and it will automatically detect the server on the LAN.

Installing the server

There are several ways to install Jellyfin server, I chose the simplest for my setup: run it in a Docker container on my NAS. That way, Jellyfin doesn’t need to access media files using the network, I can bind mount media directories inside the container. I used an image from https://docs.linuxserver.io/images/docker-jellyfin and a very simple Ansible role.

Here is the task:

- name: Create directory for docker-compose.yml
 ansible.builtin.file:
 path: "/home/{{ create_user }}/compose"
 state: directory
 owner: "{{ create_user }}"


- name: copy docker-compose.yml
 ansible.builtin.copy:
 src: files/docker-compose.yml
 dest: "/home/{{ create_user }}/compose/docker-compose.yml"
 owner: "{{ create_user }}"


- name: start containers
 community.docker.docker_compose:
 project_src: "/home/{{ create_user }}/compose/"
 state: present
 register: result

- name: print compose output
 ansible.builtin.debug:
 var: result
 verbosity: 2

And the main part is a Docker Compose file:

version: '3.5'
services:
 jellyfin:
	image: lscr.io/linuxserver/jellyfin:latest
	container_name: jellyfin
	environment:
	 - PUID=1001
	 - PGID=1002
	 - TZ=Europe/London
	network_mode: 'host'
	volumes:
	 - /data/noshare/jellyfin/config:/config
	 - /data/noshare/jellyfin/cache:/cache
	 - /data/filmy:/media1:ro
	 - /data/video2:/media2:ro
	 - /data/seriale1:/media3:ro
	 - /data/seriale2:/media4:ro
	restart: 'unless-stopped'

And that’s it. The things that need customisation are:

PUID and PGID - the user and group that owns the media files,
TZ is my timezone
Volumes: first two are for config and cache. I have a hard drive mounted at /data/noshare which, as the name suggests, is not shared by the NAS. The rest are media directories, the same ones I also share using SMB. Naming here can be anything you want, in hindsight I should have used something more descriptive than media1, media2 etc. but it’s not a big deal.

Configuring Jellyfin server

Point your browser to port 8096 of your server, in my case it’s http://firefly.home.arpa:8096 and go through the settings. First you need to create an admin account. Next important thing is to add media sources, it’s not enough to make the volumes available to the container. It is recommended that a media source contains movies, TV shows, music etc. but not a mixture of different media types. Note that media sources in the Jellyfin GUI don’t need to be the same as volumes in the container:

you can have one volume split into multiple data sources, e.g. a volume called /videos containing directories /videos/movies and /videos/shows
or the other way around, you can have volumes /movies1 and /movies2 combined into one data source.

There are some options specifying how should Jellyfin check for changes, where to get metadata etc. - usually you can leave the defaults. Scanning the data source will take between a few minutes and a few hours but the media will start appearing immediately.

In the beginning I wrote that Jellyfin doesn’t need online services. Well, it was a small lie. When it detects new contents, it will download metadata such as cover image, movie description, actor list etc. It’s optional, but on by default and uses free data sources.

More configuration

You can create multiple user accounts. They can have limited access to the data sources, e.g. child account might only have access to child-friendly shows (or vice versa, adult account without access to cartoons).

Transcoding options have a huge impact on CPU usage. First, use hardware acceleration if possible. This setting is available on the server. See Jellyfin manual for details, unfortunately my hardware doesn’t support acceleration so I can’t speak from experience. I also have a rather weak CPU. More than adequate for all other purposes, in fact it rarely goes above 1% usage, but it’s sometimes too slow for transcoding. If you find yourself in my place, you need to tune the transcoding settings on each client (not the server):

on Android, choose “integrated player” instead of “web player” in “client settings” - it supports more codecs, meaning it will do more processing on the client instead of the server
in “Playback”, choose option to prefer fMP4 Media Container - again, it means more direct playback, less transcoding.

Storage server, part 4: putting it all together

Fri, 14 Apr 2023 17:38:00 +0200

(partly based on old post from 2023.04.14, rewritten on 2026.07.09)

Choosing the OS

There are some distributions of Linux or FreeBSD specially designed for NAS use. Or you can install ordinary Linux (or Windows, I won’t judge). NAS distribution simplifies management, you get something similar to a Synology or QNAP device. But you need to play by their rules, while ordinary Linux can be customised any way you like.

I don’t have a problem configuring Linux servers without a GUI and I wanted a place for experiments in addition to plain NAS. Plus, I already used a vanilla system (Debian, stable release) on my previous attempts to run a NAS. I configured it with Ansible. I was able to reuse the playbook for my 3rd NAS. That’s the nice thing about Ansible: configuring my NAS for the first time took me more time than doing the same thing by hand, but the next attempts were almost fully automated.

Not for everyone

I don’t necessarily recommend everyone to follow these steps. I’m OK if my NAS is a bit difficult to configure, if it means I’m honing skills I need for my job. I don’t go as far as having a dedicated homelab, I just mix the two uses for home servers.

In particular, Ansible doesn’t make sense for a server that you’re not going to rebuild. Unless you want to write a playbook because you haven’t done it at your day job recently and you don’t want your skills to deteriorate. Despite my enthusiasm for Ansible, I advise everyone to skip it, unless they have a good reason.

Base installation

In my case, that was just a simple Debian install. Write the ISO to a thumbdrive, plug it into the NAS, connect screen and keyboard for the initial setup. Make sure the installer writes to the system drive only and doesn’t touch the data HDDs at all (you can disconnect them to be sure).

Installing the base system and configuring the hard drives was the manual part. It could be automated as well, but it doesn’t make much sense to automate a one-time task.

Formatting the drives

Main data drives

Btrfs RAID1 on top of two LUKS-encrypted HDDs. I created 2 subvolumes, they got mounted at:

/data/noshare - stuff that won’t be shared by SMB or NFS, mostly space for services running on the NAS (Syncthing, Jellyfin, Prometheus, photo archive)
/data/other - stuff (other than videos) that’s shared over LAN - my music and audiobook collection, old projects, files copied from old computers that I’m going to sort in mythical free time

I haven’t used any extra options for btrfs and didn’t do any performance tuning. So far, the defaults work well.

Video drives

As I wrote before, I keep video files on several drives protected by SnapRAID. It does checksum-based bitrot protection and I see no reason to snapshot the filesystem, so there was no need for btrfs. I used ext4, but this time with some extra options when creating the filesystems: mkfs.ext4 -m 0 -T largefile4 -L 'video1' /dev/mapper/luks-video1

-m 0 disables reserving space for root - by default 5% is reserved, that’s only possibly useful on a system disk.
-T largefile4 changes the inode size to 4MB from the default of 16KB. An inode is an allocation unit. 4MB is a good value for video files which are hundreds of MB to several GB in size. Fewer inodes use less disk space for metadata, larger inodes increase performance for large files. But there’s a tradeoff: the size taken by each file is rounded up to the inode size, and if you run out of inodes you can’t write anything to the disk even if it has free space. For filesystems containing text documents, photos, applications, etc. - pretty much everything other than videos or filesystem images - you should stick to the default.
-L video1 adds a filesystem label. A convenience on a machine with multiple drives, one more way to select the right filesystem.

The two main contenders are Samba and NFS. NFS allows using Linux-specific attributes such as file ownership and ACLs while Samba does the same for Windows. You can run both services on the same machine and even share the same files, if you need both Windows and Linux file permissions then go for it. If you have only one OS on your network, then the choice is obvious too.

The less common choice is iSCSI, which exports a block device instead of a filesystem. From the point of view of the connected client, it will look a lot like a local drive which can be formatted and used in any way. Very useful for many professional settings, for example for live migrating VMs between hosts. Not so much for the home, unless you mostly want to hone your sysadmin skills.

For now, I went with Samba only. I don’t really need any permissions, I’m OK with making all files on the share accessible to all systems and users on my LAN. If I need it later, I can get NFS running in a few minutes.

Configuration with Ansible

I installed the base system, set the static lease on the DHCP server and launched Ansible. Main playbook firefly.yml (that’s how I called my NAS) is very simple, because the real deal is in the roles. Most are shared with my other computers:

linux_common is the basic configuration for all Linux systems, be it physical or VMs,
docker_host and qemu_host are for running containers and VMs
firefly is the main thing.

I’ll describe the shared roles separately, now let’s focus on the nas role. I based the files on several configurations I found in Ansible Galaxy. It does quite a lot, so I split it for readability. The main file just imports the other four:

- import_tasks: pre-tasks.yml
- import_tasks: mountpoints.yml
- import_tasks: samba.yml
- import_tasks: post-tasks.yml

File tasks/samba.yml is mostly taken from https://github.com/bertvv/ansible-role-samba with only minor modifications. I added three files with tasks required before and after configuring Samba.

Before:

create a system user for samba, it will own the shared files
create mount points: make directories where the new filesystems will be mounted, modify fstab, mount filesystems - change ownership of files

After:

install and configure wsdd, or Web Services Dynamic Discovery, software that allows Windows to find Samba shares,
install and configure hd-idle, software that turns off inactive USB drives (video disks are going to be unused most of the time)

Historical notes

USB drives

My first attempt ran on Odroid and used only USB drives. It didn’t work well. Later when I moved to a PC case, I still used several USB drives. When they failed or ran out of space, I replaced them with internal drives. I don’t have any USB drives connected to the NAS anymore.

The only reason I used USB drives is that I already had a large number of them.

June 2023 update: SnapRAID recovery

It was inevitable that a disk would fail. In fact, two disks failed at once (at that time, I had more USB drives and didn’t trust them, so I used two parity drives). How I dealt with it:

I unplugged the failed disks (they were both USB drives).
Plugged new disks of the same size (could be bigger, could probably be smaller if they weren’t filled to capacity, but I had the same size disks).
Formatted them the same way (again, not necessary, but since the setup worked before, I didn’t change anything), mounted in the same place.
Checked the disk number in /etc/snapraid.conf, in my case these were d3 and d4.
Ran the command to recover files: snapraid -d d3 -l fix.log fix for one disk and snapraid -d d4 -l fix2.log fix (I could have skipped ‘-d DISKNAME’ to fix everything at once). Note: you should run it under screen or tmux as it will take a very long time - hours, maybe days.
Ran the command to check the recovered files: snapraid check (again, many hours).
Finally, re-synchronised the array just to be sure: snapraid sync (that was, as expected, quick).

Alternative to SnapRAID

Before I discovered SnapRAID, when I tried to create a NAS only with USB drives, I used a different solution to keep my data safe from drive failure. I added the drives in pairs, e.g. if a main drive was called “video1” there was also “video1-backup”. And I generated a script to copy all data to the backup disk. That’s where Ansible really shone. All I needed was a few lines in the playbook:

 name: Generate backup script
 template:
	dest: /etc/cron.daily/rsync-backup.sh
	src: rsync-backup.sh.j2

And a very short template:

#!/bin/bash
# rsync backup file -- Managed by Ansible, please don't edit manually
#
# {{ ansible_managed }}

{% for share in samba_shares %}
mount /data/{{ share.name }}-backup && \
 rsync -va /data/{{ share.name }}/ /data/{{ share.name }}-backup/
umount /data/{{ share.name }}-backup

{% endfor %}

Where “samba_shares” contained the list of filesystems. When I added another disk pair, I only needed to edit “samba_shares”. Ansible modified fstab, smb.conf and the backup script.

Storage server, part 2: disk drives, RAID and SnapRAID, encryption

Fri, 14 Apr 2023 17:36:00 +0200

(first published on 2023.03.14, rewritten on 2026.07.08)

What I have now

Again, before going into details, this is my current setup.

System drive: 256GB SSD (no redundancy)
Array for miscellaneous data: software RAID1 (btrfs) on 2x3TB drives
Array for video files: SnapRAID on 4 HDDs, mixed sizes (1TB, 3x4TB)
Data drives encrypted with LUKS, using method 2: providing the password later

Do you need RAID?

RAID stands for Redundant Array of (Inexpensive|Independent) Disks. Which means the data is stored on more than one disk. If one HDD fails, the array will continue to operate, your data will be still available. When you replace the failed drive, the array will rebuild.

RAID vs backup

RAID protects you from drive failure.

Backup protects you from drive failure, controller failure, accidentally deleting a file, ransomware encrypting all your drives, fire, theft, flood and pretty much any other reason of data loss.

If you’re going to choose between the two, choose backup. One good reason to have RAID in addition to backup is recovery time. Restoring terabytes of data from backup takes hours or days, RAID just continues to work. Which is why RAID is necessary for professional servers, to keep the business running, for home use it’s optional.

Which RAID is right?

Hardware vs software

If you read very old articles, they might argue that software RAID is slow. That is not the case anymore. Modern CPUs (as in: made in the last 15 years or so) are so fast you won’t notice any performance impact under normal conditions. Maybe during the array rebuild, but even then HDD speed will be the bottleneck.

Proper RAID controllers, e.g. MegaRAID or Dell PERC, offload everything from the CPU. I worked with them on real servers (in those unusual cases when you still use a physical server), but I’m not buying one for home. But if you happen to have one, you can consider using it.

Some home motherboards offer hardware RAID, but do it so poorly that it’s actually slower than software RAID. And I wouldn’t trust them with my data.

Hardware RAID gives an abstraction layer between your drives and your OS. You configure the array in BIOS setup tool or with an additional utility (e.g. megacli), the OS sees a single SCSI drive. It simplifies the view - either you work with physical drives or with filesystems, never mixing the two - but it also means you can’t use standard Linux tools to test and monitor the hard drives.

For most people, the choice is between:

standard Linux software RAID (built into kernel, configured with mdadm)
modern filesystems (ZFS, btrfs) that combine filesystem, RAID and volume management
RAID-like solutions such as SnapRAID

There are, as usual, tradeoffs, more on that later.

RAID levels

RAID0 - striping

R in RAID stands for Redundant. RAID0 isn’t - it spreads the writes across all drives to increase performance. If you lose just one drive, you lose the whole array.

RAID1 - mirroring

The most basic form of RAID. The same information is written to each drive in a pair. If one fails, the other continues to function as if nothing happened. When both work, read throughput is increased compared to a single drive (data can be read from both simultaneously), write throughput is similar to a single drive, maybe just a bit slower (need to write on both and it’s mostly done simultaneously). Rebuilding the array after replacing the drive is a simple process of copying data. It’s a very safe level, but wastes half of the disk space.

The nice thing about Linux software RAID is you can take out one disk from a pair, install it in a machine without RAID support and just use it like a regular drive, all the files would be accessible.

RAID5 - parity

RAID5 stores parity information. Think of it like this. Disk 1: data set A, disk 2: data set B, disk 3: A+B. If drive 2 fails, it can be recreated by calculating “disk 3 - disk 1”. Except the data and parity are distributed between the disks, not written on a dedicated one (that was RAID 3 and 4, but nobody uses them anymore). And you can have more disks than 3. And the operation is not the sum. But you get the idea.

RAID5 can survive a failure of one drive in the array, if the other one fails before the rebuild is done, all your data is lost.

A huge advantage of RAID5 is the efficient use of storage. RAID1 always has a 50% capacity of disks. RAID5 on 3 drives - 66%, on 4 drives - 75%, etc. But there are also disadvantages. Performance is not as good as in RAID1. Read throughput is about the same as from a single disk, write throughput is worse and modifying small files is the biggest problem: the controller (or OS) needs to read the original sector, recalculate parity, write new data and new parity.

But the worst thing is what happens when a drive fails. Every read requires calculating the data from parity, hurting performance. Rebuilding requires recalculating the whole data contained on the failed disk from the information stored on all other disks. This means the process is long and requires a lot of operations on the remaining disks, increasing the chance that one of them fails as well.

For this reason, RAID5 is not recommended for professional use. For home uses, the opinions vary. I think that for small arrays (3-5 disks) the efficiency might be worth the risk. And you have a backup anyway, right? If you want to be extra safe, choose RAID1.

RAID6 - parity, but more of it

RAID6 is like RAID5, but it stores each parity block twice, on different drives. That means the array survives the failure of 2 hard drives.

Four disks are the minimum number of hard drives for RAID6 and would give you the same storage efficiency as RAID1, but with more complexity. It’s a level of choice for large arrays in professional settings (that usually also have hot spares and replace disks every few years even if they don’t show any problems, all to minimise the risk of a second drive failure before the rebuild is done). But if you’re managing one, you shouldn’t be getting your knowledge from some random guy’s blog.

RAID10 - 1+0

Combines mirroring and striping to give you both redundancy and performance. Sometimes used in professional settings, if throughput is more important than storage efficiency. Not much use at home.

RAID levels at a glance

RAID	Min. disks	Usable capacity	Fault tolerance	Read perf	Write perf	Notes
RAID 0	2	100% (all disks)	None - any disk failure loses everything	High	High	Striping only, no redundancy
RAID 1	2	50% (1 disk’s worth)	1 disk (per mirror pair)	High	Normal	Simple mirroring
RAID 5	3	(n-1)/n	1 disk	High	Reduced (parity calc)	Write hole risk on unclean shutdown
RAID 6	4	(n-2)/n	2 disks	High	Lower (double parity calc)	Safer than RAID5 for large/many disks
RAID 10	4	50%	1 per mirror pair (up to half the disks, if spread right)	High	High	Mirror + stripe, best perf/redundancy balance, costly on capacity
RAID-Z1 (ZFS)	3	(n-1)/n	1 disk	High	Reduced	Like RAID5, no write hole (COW)
RAID-Z2 (ZFS)	4	(n-2)/n	2 disks	High	Lower	Like RAID6, no write hole

Flexibility of Linux RAID

One good feature of mdadm - if you’re into DIY solutions and are not afraid of running strange configurations for some time - is the flexibility. If you plan to use RAID1 in future, but only have one drive now, you can create an array and declare one drive missing. It will show as degraded - obviously - but you can add the drive later and “rebuild”. It works the other way too - you can remove the drive if you temporarily need it for something else. Even better, put it into another computer with no RAID configured and it will still show all the data.

If you need more space and decide to change RAID1 to RAID5, that’s just two commands: one to add a 3rd drive and one to change RAID level. Similarly, you can convert RAID5 to RAID6.

Note that reshaping the array takes hours or days and your data might be at risk if the process is interrupted or if a drive fails mid-process (depending on the operation, many are safe). Always check mdadm documentation, consider testing on VMs first and make sure you have a tested backup.

But still, such operations are occasionally useful for home scenarios when you need to manage limited disk space. And no other tool offers that kind of flexibility.

Modern filesystems

Some modern filesystems - in particular, ZFS and btrfs - combine RAID, volume management, filesystem and other disk-related tools in one package. This has both advantages and disadvantages compared to the traditional way. The answer is not simple and will take a separate post. Before creating RAID with mdadm, read part 3 - you might want something else.

SnapRAID

A relatively new option for home NAS is SnapRAID. It’s an interesting software that combines the advantages of RAID5/6 with the advantages of no RAID at all. How is that possible? The name SnapRAID comes from Snapshot (as in: snapshot backups) and RAID, because it works like something in between the two:

you have any number of data disks, they can have any filesystem such as ext4, XFS, JFS; could be different filesystems, different sizes,
you have between 1 and 6 parity disks, they need to be as large as the largest data disk.

During normal operation, you only use the data disks, parity drives are not touched until you run the snapraid sync command (which is run either by cron, e.g. at night, or manually after you changed something on the data drives, depending on the use case). That works best for large files that are written once and never change - such as video files. Compared to proper RAID, there are some interesting advantages:

you can add and remove disks whenever you like and combine sizes,
the drives can already contain the data when added, they would still contain the data when removed,
you can also use it to restore files that were accidentally deleted (if you’re fast),
there’s very little disk space overhead (e.g. if you use 1 parity disk for 4 data disks, that’s only 20%),
it protects against bitrot, while RAID doesn’t (see part 3 about bitrot),
there’s no slowdown, no CPU usage and no extra I/O during normal operation, only during sync,
you can set up your disks to power down and only the one that’s currently accessed will spin up (RAID5 or 6 would spin up all disks),
if multiple disks break, you can still access the data on the other drives - unlike RAID5 and 6 when they would all be useless,
while it is not recommended to use SnapRAID on USB drives, it is possible.

The downsides: your files are not protected from drive failure until you run “snapraid sync” and the technology is not as well tested as Linux RAID.

Testing the drives

Some of the hard drives I use are old, maybe even 10 years old. But even with the new drives it’s a good idea to test them. Especially if you can replace them under warranty. I had some old USB drives too and did the following to quickly check them:

connected them to my laptop one by one,
ran the short self-test: smartctl -t short /dev/sdb
if there was already some data, checked the filesystem: fsck -f /dev/sdb1
short self-test takes about 2 minutes, after that time I would see the results plus all other SMART data: smartctl -a /dev/sdb

Two drives were showing quite alarming results (failed self-test, high values in Reallocated_Sector_Ct). One of them had some old backups on it, it was also showing filesystem corruption and some files could not be read (nothing important though). I copied the data, wiped the drives and threw them away.

Should you immediately throw away the drive showing any errors in SMART data? It’s up to you, if you trust your RAID and backup, you might take your chances. Remember that a drive can fail without a SMART warning anyway. I’d say that one-digit numbers in Reallocated_Sector_Ct or Current_Pending_Sector are no cause for alarm. Some drives develop a few bad sectors early and then continue to work fine for years. But if you see the numbers rising or SMART overall health self-assessment goes bad, you better get a new drive soon.

In fact, two other drives were showing some “not great, not terrible” SMART attributes. I marked them as suspicious and made sure they are backed up. It’s handy to have a label writer when you’re dealing with 20 USB drives.

Then after connecting the drives to the target system it was time for a proper checkup. First, a long SMART self-test which takes several hours: smartctl -t long /dev/sdX

Another thing I did was a badblocks test overwriting the whole disk: badblocks -b 4096 -wsv /dev/sdX. To reiterate, it overwrites everything on the disk! Only use it with an empty drive. And it takes days on a large and slow drive. It was quite a pain, especially since some drives already had data, so I had to copy it around. But once it was done, I could be reasonably sure that every sector of the disk works properly. And if any errors were discovered and corrected by the drive firmware, they should show up in the SMART data.

Disk encryption

I wouldn’t use a laptop without an encrypted hard drive. Laptops often get lost or stolen. If you’re lucky, a thief will just reformat the laptop for sale. If you’re not, your data will be leaked, which means potential embarrassment or identity theft. Companies can even be fined for neglect. Both Windows (with BitLocker) and Linux (with LUKS) make it easy and almost transparent to encrypt a personal device. With Windows, you can even switch back and forth, with Linux, it’s better to choose during the installation whether you want encryption, changing your decision later means a manual process.

The balance isn’t the same for home servers. In many places, home burglaries are unlikely, even if they happen, a bulky PC is not a first choice for thieves. And servers usually run headless (without screen and keyboard). Which means the usual way of unlocking encrypted hard drives - typing a password during boot - is not possible.

Three ways to deal with it:

Method 1: no encryption

A valid choice for some. If your risk vs inconvenience assessment says so, I’m not judging.

Method 2: providing the password later

You set your system NOT to mount the encrypted filesystems on boot, and NOT start any daemons that depend on them. Instead, you create a script that will prompt for a password, unlock the devices, mount the filesystems and start the services. After the server boots, log in with SSH and run the script.

Upsides: easy to implement, no complex setups with many potential failure points, decent security.

Downsides: still requires manual intervention, system drive can’t be encrypted this way.

Overall, it can be a good choice for a home server that’s rarely booted. Not encrypting the system drive means some information is at risk (configuration in /etc, logs in /var, list of installed software) but for most home users it’s a reasonable tradeoff.

Method 3: passwordless LUKS

Password is the most common type of LUKS key, but there are other possibilities.

LUKS key on a thumbdrive. You can use a file as the LUKS key. Store it on a thumb drive, make sure it’s present during boot, then remove it and keep it somewhere safe, away from the server. Least secure of all options here.
systemd-cryptenroll with YubiKey or smartcard. Similar to the previous version, but slightly more secure: a thumbdrive can be copied, those devices can’t. Requires additional hardware that most people don’t have and a more complex setup. Useless if you leave your device in or next to the server.
LUKS key in initramfs + Secure Boot. LUKS key is present in the initial ramdisk, so during normal boot, the disk is automatically unlocked. If you enable Secure Boot, password-protect the bootloader, UEFI setup and prevent selecting an alternative boot device, you’ll also protect the system during not-so-normal boot. Allows unattended boot and encryption of the system disk (but not /boot), but has one weak point: an attacker can move the hard drives to another machine, running their own OS, thereby bypassing your passwords. They can then extract the key from initramfs (which needs to be unencrypted in that setup), unlock and mount the drives. Probably above the pay grade of a common thief, but not beyond the skills of an experienced Linux user, not to mention a security professional.
LUKS key in the TPM. TPM will only release the key if certain conditions are met. You can also configure it to always release the key, which works similar to the previous point, but addresses the weakness: if you move the drives to another machine, the TPM won’t contain the key anymore. Downsides: it’s the most complex option.
Authorization server on the network. You’ll be surprised how much you can pack into initramfs to manage the way the system boots. One way is to contact Tang server to authorize unlocking the disk. If the server is moved away from the LAN, it won’t reach it. Tang server (and related Clevis client) are free and open source, in theory you could run it at home, but I’ve never heard of anyone doing it.

Are these methods practical? First two rely on removing your key after the system boots, leave it with the servers and you completely defeated the encryption. The others are much more complex. There’s a real risk of making your system unbootable due to configuration error or a failed update - you should have an additional LUKS key (e.g. a complex password you keep in your password manager) for recovery. There’s also a risk of leaving a way in if you miss one step (and the 3rd method already leaves a huge known vulnerability).

I don’t recommend passwordless LUKS unless you’re a seasoned sysadmin. Actually, I am a seasoned sysadmin and decided it’s not worth it for home use.

Storage server, part 1: hardware

Fri, 14 Apr 2023 17:35:00 +0200

(first published on 2023.03.14, rewritten on 2026.07.07)

A house is not a home without a NAS. OK, maybe I’m not really that emotionally attached to my file server, but it’s one of the first things you need if you want to rely less on the cloud. What can you use it for?

sharing files between the devices,
storing large data you create, such as photos and videos - laptops and even desktops tend to have smallish SSD drives these days, not really suitable for media content,
backups,
storing music, movies and TV shows.

You can also use the same machine as an all-purpose server. With virtualization and containers, it’s easy to separate different workloads (you could run everything on the same OS, but the system might get messy).

I’m trying to keep my IT green (see separate post for details), which in short means:

the fewer boxes running 24/7, the better,
prefer old hardware to new one.

What I have now

Before going into details and considerations, this is my current setup

Part	Model	source
Case	Corsair, unknown (huge) model	used
Motherboard	Asus H87I-PLUS, Mini-ITX	used
CPU	Intel Pentium G3220T	(came with MB)
RAM	8GB	(came with MB)
HDD	6 different ones	new and already owned
SSD	Plextor PX-256M6S	already owned
PSU	Thermaltake Smart BM2 450W	new
other	extra SATA card	new

My NAS history

Commercial vs DIY

The easiest option is to buy an off-the-shelf NAS such as Synology or QNAP. I had one in the past (Synology DS216se). The good thing about them is you can get them up and running in less than an hour and without much knowledge. Just install disks, plug it into the network and answer a few questions in the web UI. It’s also more compact than even a smallest PC.

But the cheap models only have 2 disk bays, very little RAM and a pathetic CPU. They work fine as basic file servers, but doing anything in the GUI is an exercise in frustration. Click, wait for 10 seconds until it responds, click, wait again. You’ll quickly miss the command line. If you want to run any extra services, you’re limited by both hardware resources and software available. I used RAID1 for storage so when I ran out of space, the only option would be to replace both disks with the bigger ones, old disks would be useless (or add a USB drive, but that means no RAID). Enough, back to the DIY.

To USB or not to USB?

I have a drawer full of USB HDDs. A few were mine, most I literally inherited. Many of them are small and not really worth bothering, but about a dozen were 1-2TB. I experimented with a NAS running on a single-board system (Odroid C1+) and a small-form PC (Dell Wyse) and decided I REALLY don’t want a NAS on USB drives. You can run one or two semi-reliably, but the more you have, the probability of problems approaches 100%. Sometimes the plug moves in the socket and you end up with a corrupt filesystem. Or the cable cannot handle the power. Or the disks run fine when you initially connect them one by one, but during the next reboot they will start all at once drawing too much current (yes, I tried a powered USB hub) and only half of them will show up. Of course, you can’t have RAID on USB (technically, it is possible, the system wouldn’t stop you, but it’s a terrible idea if your drives can disappear at any moment).

Searching for a case

For a NAS, you want a case that can hold many drives: preferably, more than you intend to install at the moment, to have space for extension. Such cases are relatively easy to buy in the USA, but not in Europe. It’s quite a niche product after all, very few PCs use more than 1 or 2 drives. Those that can be found are twice as expensive as in the States, buying straight from the US vendor means paying for shipping and long waiting.

But I found one interesting option: Kolink Satellite Cube. According to the specs, it had 4x 2.5" and 3x 3.5" bays. And it held a mini-ITX motherboard - not perfect, but OK. It was cheap and quite small. I ordered the case and searched for the MB. Mini-ITX are hard to find, but I found a great second-hand option, with 8GB RAM, a Pentium CPU that’s really a 4th generation Intel Core and best of all, 6 SATA ports!

When I got the case, I discovered the specs were wrong in two ways. One, it could hold a mini-ITX or a micro-ATX motherboard, the latter is slightly larger and much more popular. Fine, if I knew that maybe I would have found the motherboard quicker, but I’m satisfied with the one I got. But the other thing was worse. Turned out I can have 4x 2.5" OR 3x 3.5", but not both.

I lived with this limited space until I got tired of it and searched eBay. I found a person nearby selling Corsair gaming case and the specs specifically mentioned multiple drive bays. I don’t even know how many hard drives it can take, but I put 6 and there’s some space left. Plus, the 5.25" drive bays can be reused for HDDs if I need even more.

That also means I can switch to a full-size ATX motherboard in future. Currently, the mini-ITX one I have is enough and can be installed in a huge case, even if it looks funny sitting in one corner of it.

Hardware considerations - mine and yours

CPU

If you want only a storage server, or just a few small services on top of it, you don’t need much processing power. Any reasonably modern (say, less than 15 years old) mainline CPU from Intel or AMD is already way more powerful than the puny processor in a commercial NAS. Rule of thumb for Intel CPUs: if it’s called “Core i-some number”, it should be OK.

If you want more processing power (e.g. for video transcoding), look for something slightly newer - not necessarily new and top of the line. Still more? If you have funds, you can pack an unreasonable amount of CPU power in one box. I used to work with servers that had 2x128 cores, they were used for physics simulations. But I don’t have enough imagination to think why anyone would need that at home.

On the other end of the spectrum, there are Intel Atom (for low power usage devices), Pentium and Celeron (budget versions of mainline CPUs). I really wish Intel used a more consistent naming convention. These brands have been in use for decades and without checking the exact model, you won’t know if it’s just slightly slower than a modern laptop, or a completely unusable 30-year-old junk.

My CPU is Intel Pentium G3220T, which is a slightly slower (no HT, no AVX) version of a 4th generation Core i3. Slow by today’s standards, yet most of the time it’s only used at 1%.

Two things worth checking when buying used hardware:

Does the CPU have virtualization extensions? You need them to run VMs at reasonable speeds. All mainline CPUs since roughly 2015 have them, if it’s older or unusual - check.
Intel Xeon and AMD EPYC are server CPUs. Some are power hogs, others scale down just as well as home models. If you find a good offer on them, there’s a chance to buy some real processing power cheaply, but check idle power usage. Note that server motherboards might not fit standard PC cases.

I also don’t recommend buying CPU and motherboard separately, unless you know what you’re doing and how to check the compatibility.

RAM

For a storage-only server, even small RAM like 1 GB is OK for the start, Linux would only need a few dozen MB anyway and use the rest for disk cache. If you want to run extra services, the answer can vary from “1GB is still enough” to “I have 128G and ran out of memory anyway”. But realistically, 4GB should suffice if you don’t run VMs. I have 8GB and use less than 2 most of the time.

The situation changes if you want to use ZFS - a modern filesystem with some interesting features, popular among self hosters. 4GB is a minimum, 8GB is better. Personally, I don’t like ZFS, more on that in later posts.

Network

You should connect the NAS using Ethernet, not Wi-Fi. 1Gbit is a reasonable choice, 100Mbit is too slow these days, higher speeds are not necessary for most home users (if you’re the one that needs it, you know).

If you really insist on 10Gbit, you should use fibre. While the standard Ethernet cables (CAT6 and higher) theoretically support 10Gbit, the transceivers/NICs often overheat under high load, causing unstable connections. I’m speaking from experience of using 10Gbit at work.

SATA ports

You’ll need a lot of them. Most motherboards have 2 or 4, if you’re lucky you might get 6. It’s not a big issue: you can install additional controllers. Just make sure you have enough empty PCI Express slots. This is only a concern for small cases and mini-ITX/micro-ATX motherboards, full-size PCs can easily have an unreasonable number of SATA ports.

Hard drives

3.5" are cheaper, faster, available in larger capacities and are considered more reliable (though it’s mostly anecdotal evidence).

2.5" are smaller (obviously). In some PC cases, you can fit more of them. In others, you just have to install a caddy around the 2.5" drive and you still use the same 3.5" bay.

SMR controversy

Traditional hard drives use CMR (Conventional Magnetic Recording), where each data track is written independently with a small gap between tracks. SMR (Shingled Magnetic Recording) drives instead overlap tracks like roof shingles. This allows packing the data more densely, and therefore lower the price of high capacity disks.

But there’s a tradeoff. Writes to SMR drives are 3-10x slower than to CMR. Hard drives use write caching to offset this. In normal use, you might not notice the difference. But if you try to write gigabytes at a time, you’ll notice write speed begins at reasonable levels such as 250MB/s, but in a few seconds, once the cache is full, slows down to 40MB/s. It’s not only an issue for copying large files, but also for RAID rebuilds.

Most budget large-capacity HDDs these days are SMR. Whether you want to pay more for CMR, depends on your use case.

There are several types of SMR, the most popular in consumer hardware is called Drive Managed (DM-SMR). It is invisible to the host, there’s no attribute that flags it. Some drives might have SMR in their identification that you can read with dmesg | grep sda or smartctl -i /dev/sda, but most don’t. The only semi-reliable way to check your HDDs is to read the model number (from smartctl data) and look it up on the internet. Semi-reliable, because several manufacturers were caught quietly swapping technology in previously-CMR product lines.

Personally, I have a mixture. Some drives are CMR, because they predate SMR technology. New ones are a mix of both SMR and CMR, when I bought them, I didn’t know the difference.

Brands and reliability

You’ve probably met someone who says never to buy Seagate and get Western Digital instead. Or the other way around. The truth, if you look at reliability data (e.g. famous Backblaze stats), is that all manufacturers had some models that failed early and some that exceeded expectations. If you want to shop for reliability, look for specific model, not just the brand. Even that doesn’t guarantee that 2026 WD Red 4TB is the same thing as 2024 WD Red 4TB.

Personally, I don’t care much. I’m just prepared for any drive to fail at any moment. Instead, if I buy two HDDs at the same time - especially if I plan to use them in RAID1 - I choose two different manufacturers. It’s not optimal for array performance (ideally, you would have 2 drives with the same specs), but I’m reducing the chance of both drives failing at the same time.

USB drives and shucking

External USB disks are often cheaper than internal 2.5" drives, despite the extra cost of enclosure and electronics. Some can be removed and used as regular drives. The process is called shucking and it’s quite popular among self hosters. But there’s a reason for lower price: they are built with lower quality components (or they failed QA tests), they might live for years if they are only used occasionally, but won’t last long with more intensive use.

Since I had many USB drives, I shucked a few. I used others as external drives (they couldn’t be shucked, they are soldered directly to the USB electronics and have no SATA port). About half of them failed in a few months, the other half worked just as reliable as regular HDDs for a few years. I don’t use them with the NAS now, though some are still usable. I don’t recommend buying USB drives for a NAS, but if you already have them, you can reuse them, just be prepared for failure.

The system drive

Should you place your OS with the data or use a separate disk? Both ways have some pros and cons.

If you use an old motherboard, it might not support booting from large disks - but once the operating system loads, it will access the drives just fine.
Most, if not all, NAS-specific distros require a separate system disk.
Decoupling data from OS will generally make your life easier in the long run: you can replace data disks if they are too small while keeping the system. Or you can move all your data drives to another machine.
You can also run the system from an SSD which is much faster - it doesn’t make much difference if you use it only for NAS, but it could be noticeable if you also want to host VMs or containers.

On the other hand, keeping OS with the data means you don’t waste precious drive bay in the case. And if your data is on RAID, the system is also protected from the drive failure.

All things considered, I’d recommend a separate drive.

Tip: if you want to use a separate drive, but don’t want to waste drive bay/SATA port/money on the extra disk, you can install Linux on a USB stick. Just remember they are way slower even than HDDs and wear out quickly. You should try to use them read-only (possible but tricky, requires combining read-only fs with a writable ramdisk on top of it, e.g. aufs) or almost read-only (e.g. disable swap, redirect logs to an external system). You can even install the stick internally - motherboards have USB connectors for use with the case’s front ports.

PSU (power supply)

Buy the lowest power PSU you can find. It won’t be easy. These days few people build PCs, it seems that these are mostly gamers and crypto miners, both use power-hungry GPUs. A NAS, even with multiple hard drives, will likely not exceed 100W at full load, much less at idle. Old PSUs were very inefficient at very low load, modern ones are better, but still waste a considerable amount of power.

There are certifications aimed exactly at this problem: 80+ means the PSU is at least 80% efficient at any load higher than 20%. The higher the level - they go from White (or just 80+ without a modifier) through Bronze, Silver, Gold, Platinum to Titanium - the better, but more expensive.

Other things worth paying for are reliability and low noise level. Good news: all three factors often go together. Bad news: they also go together with higher price.

Brands recommended in self hosting circles: Seasonic, Corsair, be quiet!, FSP. The one I have (Thermaltake) is considered “not great, not terrible”.

Pi-hole - DHCP and DNS

Fri, 14 Apr 2023 17:30:00 +0200

DHCP is a necessary component of the home network. You probably don’t want to manually assign IPs. It would be quite inconvenient on the laptop that travels between several networks, even more on the smartphone. On IoT devices lacking proper screens and keyboards, it’s plain impossible.

If all you have are client devices, you don’t need to know what IPs are assigned. Perhaps the address of the router to occasionally log in to its web interface, but that’s it. Most people don’t know the IP of their computer or tablet and rightly so.

It all changes when you want to run servers. There are ways to find services on the local network, such as mDNS and WS-Discovery, but let’s face it: at some point, you’ll need to point your browser or SSH client at a specific IP. So it’s important that it doesn’t change.

Of course, every home router contains a DHCP server which allows setting some addresses as static. It’s often a good choice. There’s one important advantage of keeping your DHCP server on the router: fewer points of failure. DHCP is needed for the network to function and so is the router, if they’re the same device, that’s only one device that really needs to work all the time. But a good reason to move DHCP out of the router is the possibility to back up configuration. Are you going to keep using the same router? I certainly don’t, as the one I got from my internet provider is terrible and I’m going to replace it soon. I don’t want to configure all the addresses manually again and home routers don’t allow you to export/import DHCP configuration, certainly not between different brands.

So I decided to run my own DHCP server as a first step before I start running servers. But why would I type IPs, when I could use local DNS? And if I have a DNS for the local network, it can also cache internet addresses. A typical website fetches content from 10 or more different servers, so there’s a noticeable difference when the DNS replies in 2 milliseconds as opposed to 200 milliseconds. I could use bind and isc-dhcpd like I did in the past when configuring “real” networks, but there are servers more tailored to the needs of home users.

Pi-hole was designed as a DNS sinkhole - a server that can block some content by replying “address unknown”. Typical use is blocking ads and tracking: on websites, but also in mobile applications and IoT devices that can’t use traditional ad-blocking software. It can also be used to increase security by blocking some malware that relies on DNS, blocking fraudulent websites, etc. - it all depends on the blocklists loaded and there are many to choose from. Non-blocked queries are forwarded to external DNS servers and then cached. There’s also a possibility to add local addresses and an optional DHCP server. So it’s the extra functionality that first caught my attention and the core functionality is for me a nice addition.

Choosing hardware

Pi in Pi-hole comes from Raspberry Pi, but the software can run pretty much everywhere where you can run Linux. I did my first experiment using Docker on my homelab server, but I quickly thought that Raspberry Pi is indeed a better option. Not because there was anything wrong with the Pi-hole itself, but DHCP and DNS are such important pieces of infrastructure they really should run on a separate device. A homelab server is, by definition, something for experiments, you often need to reboot it and sometimes feel a need to rebuild it from scratch. Plus, if Pi-hole crashes (it never happened so far and I’ve been running it for months) when I’m away, it can be rebooted by pulling a plug.

So, Pi it is. They are currently hard to get or outrageously priced, but luckily I had one unused Pi 2 in the drawer. It’s 7 years old, with a performance way below current models, but more than enough for the job.

Installing OS

The official operating system is Raspberry Pi OS, formerly called Raspbian. It’s a port of Debian, my favourite distro. Perfect. I chose a 32-bit version (64-bit requires Pi 3 or 4), Lite edition, meaning no GUI. Unlike the PCs, on machines like this, you don’t use the installer. Instead, you write the image to the micro-SD card using another computer. There is a special tool for that called Raspberry Pi Imager. It simplifies downloading and writing and even allows you to do some customisation. I chose the old school way - downloaded the image myself, wrote it with dd and customised the system later, but that’s because I like the old way.

Initial configuration

My system is going to run headless, but I have to do the initial configuration. There are two options: you can temporarily connect a keyboard and monitor, during the first boot you’ll answer a few questions. Or you can mount the micro-SD card on the PC and modify a few config files. There’s a third option of using Raspberry Pi Imager, but I already skipped this. Since I like editing config files and I’m too lazy to go to another room and fetch an HDMI cable, I chose the second option. Again, feel free to use the other two if you don’t feel like manually editing configs.

There are two partitions on the micro-SD card, bootfs and rootfs. All I had to do was these 3 steps:

on bootfs, create a file called ssh or ssh.txt, it can be empty or have any content, just the presence of this file informs the system to start the SSH server,
also on bootfs, create a file called userconf.txt containing a username and an encrypted password separated by a colon, to encrypt the password you use the command: openssl passwd -6
on rootfs (not bootfs this time!), edit the file etc/dhcpcd.conf to provide network information (since the Pi is going to run the DHCP server, it obviously cannot get the configuration from DHCP)

cd /media/igor/bootfs
touch ssh
openssl passwd -6
vim userconf.txt # add the line below
igor:I'mNotShowingMyPasswordHereEvenIfItIsEncrypted/
cd /media/igor/rootfs/etc
vim dhcpcd.conf
# add lines
interface eth0
static ip_address=192.168.1.3/24
static routers=192.168.1.1
static domain_name_servers=192.168.1.1 1.1.1.1

Note that I’m not using localhost as the DNS server - not yet!

I replaced the micro-SD card, connected Pi to LAN and powered it up. A few seconds later it’s available on the network. I could log in with SSH and update packages. It takes much longer than on the PC (SD cards are slow).

Installing Pi-hole

The recommended method is:

curl -sSL https://install.pi-hole.net | bash

but I prefer to see the script before running it, so I chose alternative method 2:

wget -O basic-install.sh https://install.pi-hole.net
less basic-install.sh
sudo bash basic-install.sh

The installation script supports several operating systems and uses the distro’s package manager to install most of the software: lighttpd and a few PHP packages for the web interface, git to download the core part of Pi-hole and some supporting tools. We need to confirm the system uses static IP and answer some very important questions (but don’t worry, you can change all the settings later):

Privacy versus security

Do you want to keep logs of the DNS queries? It can be useful to see the activity on the network, e.g. to debug something or to find malware. But it also means you’re logging all websites visited by you and other people in your home. Do you want it? I don’t. Another problem with query logging is the size of logs - you’re going to run out of disk space soon and you’ll quickly wear out the SD card.

My recommendation: answer “yes” during installation, but only keep this option on for a few minutes, to see that everything is running fine. Then, log in to the web interface, choose Settings/System, “Disable query logging”, then “Flush logs” to delete those already written. Pi-hole will only keep completely anonymous statistics, e.g. how many clients are connected and how many DNS queries were sent.

Warning, there’s another option in Settings/Privacy to disable displaying some private information, but if you don’t disable logging, it will still be written.

Choosing upstream DNS

Another important option during installation is the upstream DNS or the server to which the Pi-hole will forward the queries. Again you should consider privacy. A popular option is Google, the server is fast and its IP is easy to remember (8.8.8.8), but don’t you think Google already knows too much about you?

I think there are two good options. One, choosing “custom” and entering the servers of your internet provider. After all, the provider already knows where you connect and the servers are close to you (I’m talking about network distance here, how many devices are between you and the server), so quite fast. However, in many countries providers are required by law to filter some domains and you might not like it. Some providers even use DNS for more controversial purposes - if you type a non-existent domain, instead of returning an error, they will redirect you to a website showing ads. It’s annoying in the browser and could completely mess up other software.

That’s why the other option is even better: Cloudflare and the famous “1.1.1.1” server. It’s blazingly fast, faster than Google and likely faster than your provider, despite being further from you. What’s even more important, Cloudflare doesn’t log DNS queries. They have regular external audits to prove it.

Configuring DHCP

There needs to be exactly one DHCP server on the home network. Less than that and the devices won’t get an IP address and won’t connect. More than that, they will get confusing replies and likely also won’t connect. Luckily, once the device gets the network configuration, it will keep it until it expires, so changing the DHCP server is simple and safe, provided that you do it in the correct order:

log in to the router and Pi-hole web UI,
check the DHCP settings on the router, but don’t disable it yet,
enter the same settings on the Pi-hole (except for DNS, which should be the address of the Pi-hole), but don’t enable DHCP yet,
now, disable DHCP on the router and quickly, before someone complains that the network doesn’t work, enable it on the Pi.

It’s a good idea to test. Get another device (not the one connected to the router and Pi) and force it to renew the IP, for example by enabling and disabling airplane mode. You should see a client appearing on the Pi-hole DHCP server and DNS queries showing up on the dashboard.

Some time later, you will see the “Currently active DHCP leases” list getting longer. On the right of every entry, you can see two icons. One to remove the device and the other to set a static lease (permanent address). It’s a bit unintuitive: clicking this will move the entry to the list below (Static DHCP leases configuration), but you need to click the + button next.

Filtering

Filters are configured in the Adlists tab. The default list is quite lax. I think that is right. If you use a browser extension such as uBlock Origin, you can be aggressive in your filtering: if it causes a website to stop functioning properly, it’s easy to unblock. In Pi-hole that’s more hassle, plus it’s filtering for the whole network which is probably used by less-technical users as well.

What can we do if Pi-hole breaks the website (or we suspect that it does)? There are two options. Quick one: click “disable blocking”. Precise one, see option “domains” in the left menu. You can use it to add a domain to a whitelist or blacklist, using ordinary text or regular expressions.

Filtering with DNS is not as effective as using a browser extension. You can’t block https://example.com/ads/ while leaving the rest of https://example.com/ . Of course, you can use both. Pi-hole can also increase security. Some malware connects to a specific domain. There are also lists of domains used by scammers - fake lotteries, SMS subscriptions, etc. Since I also visit Polish websites, I use a list that also includes Polish domains: https://raw.githubusercontent.com/MajkiIT/polish-ads-filter/master/polish-pihole-filters/KADhosts.txt

Local DNS

Recommended domain for local use is home.arpa - not localdomain or example.com that many people use. To be fair, in most cases nothing bad would happen, but let’s do it by the book. Simply add your devices in Local DNS -> DNS Records. Pity there’s no option to automatically populate local DNS with DHCP entries.

Backup

I usually use Ansible to configure services, but this time I decided it was not worth it. You only configure Pi-hole once, no configuration can be shared with other servers. It’s much faster to use a web UI.

But it’s always a good idea to have backups. There are two ways to do it, for slightly different purposes. One is the option Teleporter in Settings. It allows you to save or revert configuration. It generates a small file containing things like DHCP and DNS settings, blocklists URLs (but not the blocklists), manually added entries. It’s useful if you mess up the configuration or need to move the config to another device.

The second option is to make an image of the micro-SD card. Take it out of Raspberry Pi, insert it into the reader, check (e.g. with lsblk) which device name it got (I’ll use sde in the example):

dd if=/dev/sde bs=10M status=progress conv=fsync | gzip > raspbian-pihole.img.gz

It writes a backup of the whole operating system. Useful if you break something on this level or your micro-SD card stops working. You can restore it with:

zcat raspbian-pihole.img.gz | dd of=/dev/sde bs=10M status=progress conv=fsync

If you don’t know dd, the options are:

if is the input file, stdin by default
of is the output file, stdout by default
bs stands for block size, set it to something large like 1M or 10M (probably no difference between the two, but the default is only 0.5KB) to increase throughput
conv=fsync ensures that data and metadata are really written before the program exits,
status=progress shows progress.

It’s enough to write it once, maybe repeat it occasionally (e.g. once a year or before/after an OS upgrade).

Why Self hosting?

Fri, 14 Apr 2023 17:30:00 +0200

I think we’re on the verge of a “back to the past” moment in IT. For the last 15 years or so, we heard that on-premises infrastructure is dead, desktop software is not needed, everything should run on the cloud. Preferably as a collection of microservices, utilising NoSQL databases and message queues.

Recently it changed. I keep seeing articles about the advantages of monolithic architecture, relational DBs, applications that don’t embed the browser, etc. Last but not least, more and more companies realised that the cloud is expensive, on-prem servers are back. There is also a growing trend among hobbyists to run self-hosted services at home.

Now, don’t get me wrong. All of these technologies have good uses, but they are not a solution for everything.

Good and bad reasons

Education is one good reason. Many IT professionals maintain a “home lab” to experiment with technology. Isn’t it convenient when the lab can also do something useful?

Privacy is another good reason. I see some people declaring that privacy is lost anyway and they might as well share all their personal information. I see others that meticulously control all their data to make sure it never leaves a system they control. I’m neither. I don’t believe fully reclaiming your data is even possible and the more you try to control it, the more you hit the law of diminishing returns. But some things can and should be done.

How about owning your data? Cloud service can be discontinued at any moment. Small companies go out of business, corporations decide to cut the margins. Especially Google is famous for killing many of their projects, even if they achieved a moderate success. It is also a known problem with IoT devices - many people discovered their smart bulb or socket is no longer smart (or doesn’t work at all) when it can’t connect to the manufacturer’s servers.

Less reliance on the internet. OK, I’m not making any plans for living offline, that would be way too uncomfortable. But if I have most of my data and software locally, I can live with slow or unreliable connection. And yes, it is an issue even in the most developed regions of the world. Perhaps you would like to live in a village or spend some time in an RV or a boat one day? Outside the towns, mobile internet is often the only option. Just a few kilometres from the cell tower and you don’t even get that.

Cutting cost is not a good reason. You may get some savings, but it’s not guaranteed. You might cut down on cloud subscriptions, but add the cost of hardware and electricity. Not to mention the time spent on configuring all this.

Where am I now?

I plan to write this as a step-by-step guide of my journey to self-hosting, like I’m starting from scratch. That is not strictly true. I’m already running some of the services or I did that in the past. I’m going to start with the basic infrastructure first and then add new services or better, tidier versions of those I’m already running.

(OUTDATED) Generating websites with Pelican

Mon, 27 Mar 2023 17:30:00 +0200

July 2026 update

I no longer use Pelican for my websites. I use another static generator: Hugo. It’s much faster and actively maintained. The results look much better too, especially for pages with many or large images. For text-heavy sites like this one, there’s not much difference.

I decided to keep the old contents. All the generic information about static generator vs. CMS are still (or even more) valid for Hugo. And maybe someone who needs to use Pelican stumbles upon this site.

Old contents from here

I use static website generator called Pelican to build all my websites, unless I have a good reason not to. Why? Since laziness is one of the virtues of a good programmer, I asked ChatGPT why a static website generator is better then a CMS. Here’s a slightly edited answer:

Speed and Performance: Static website generators offer incredible speed and performance compared to CMS. With a static website, there is no server-side processing required to generate pages, resulting in faster load times

Security: Static websites are more secure than CMS because they have a smaller attack surface. Since there is no server-side processing involved in generating static sites, there are fewer points of entry for potential hackers.

Cost: Static websites are generally cheaper to host and maintain than CMS. With CMS, there are ongoing costs associated with hosting, database maintenance, and plugin/theme updates. In contrast, static websites can be hosted anywhere.

Version Control: Static website generators are typically built with version control in mind, which allows developers to track changes, revert to previous versions, and collaborate with others on the same project.

So there you have it, ChatGPT got right. Out of many static website generators, I chose Pelican because it’s easy to use and written in Python. To be frank, I never even looked at the source code because I never had to modify or extend anything, but it’s good to know that I can.

But I don’t like the look of this website

Could be, but it’s not the problem of Pelican. That’s because I like minimalistic designs and I’m not very good at them. You can have as much eye candy as you like.

Start simple

Static site generators can seem quite intimidating. You’re faced with thousands of configuration options, plugins and themes. But you don’t have to use all of these, definitely not from the start. My suggestion is to do some minimal configuration, add a bit of content, generate the website, repeat. For the developers: think of it as a one-person Scrum with 30 minutes sprint time.

Installing Pelican and initializing the website

I assume you already have Python installed. On Linux, it’s almost always true, every general purpose distribution comes with Python preinstalled. On Windows, you can get Python from python.org or Windows Store. Or you can use Anaconda on any platform if you prefer.

A recommended way to install Pelican is using venv and pip. However, I had good results on Debian and Ubuntu with an ordinary system package - a simple “sudo apt install pelican” gives me a working setup in seconds.

Optional step: create a GitHub repository for your website. Login to GitHub, click the “New repository” button, then clone it on your computer, either with your favourite IDE or “git clone” plus the URL pasted from GitHub. Remember to add output/ to .gitignore.

If you don’t want to use GitHub, simply make a directory and enter it. You can alway create a repo later if you change your mind.

In the new directory, run “pelican-quickstart” and answer a few questions. Make sure to change the title, author, URL prefix, and timezone. I chose not to automatically upload the site, I wanted to test it locally first.

Choosing a theme

Have a look at https://github.com/getpelican/pelican-themes. At first I chose Bootstrap2, a simple theme with no dependencies. Soon I discovered I should switch to similar looking, but much more powerful Pelican-Bootrstrap3 which requires several plugins. Fine, I’ll need plugins anyway. Changing a theme is simple, so don’t worry that you choose wrong.

Recommended way is to use git submodule to install the theme and plugins, but again I did it my way. Here’s my setup:

I keep all my websites in ~/Documents/web, eg. this one is ~/Documents/web/web-random
In the same directory, I keep themes and plugins, in ~/Documents/web/pelican-themes and ~/Documents/web/pelican-plugins
In each website’s directory, I created symlinks to themes and plugins

cd ~/Documents/web
git clone --recursive https://github.com/getpelican/pelican-themes ./pelican-themes
git clone --recursive https://github.com/getpelican/pelican-plugins ./pelican-plugins
cd web-random
ln -s ../pelican-themes .
ln -s ../pelican-plugins .

Then I just added a few lines in configuration file pelicanconf.py:

PLUGIN_PATHS = ['pelican-plugins']

THEME = 'pelican-themes/pelican-bootstrap3'
BOOTSTRAP_THEME = 'flatly'

JINJA_ENVIRONMENT = {'extensions': ['jinja2.ext.i18n']}
PLUGINS = ['i18n_subsites']
I18N_TEMPLATES_LANG = 'en'

Adding content

Every file with .rst (ReStructuredText) or .md (Markdown) extension in the directory content will be parsed by Pelican. I prefer Markdown since it’s also used on Github, Jupyter and many other places. Pelican considers every file an article - something that is indexed by a publication date, has category, tags etc. Unless the file is in the content/pages subdirectory - this one is for non-chronological content. So, if your website resembles a blog (like this one), you would have no or maybe several pages (eg. About) and many articles. If you don’t want to see newest content first, you would put everything in pages.

Every article or page starts with a short metadata section. Here is an example taken from this post:

Title: How I build this site
Date: 2023-03-27 17:30
Status: published
Category: it
Tags: it, python, website
Slug: pelican

Most of this is self explanatory. Slug is the output filename if you want to use a non-default.

For the non-blog site, you generally skip things like tags and category. You also have to choose one page as the default. Example from my personal site:

Title: Too Many Machines
Date: 2023-03-27 17:30
Status: published
Summary: Personal website of Igor Wawrzyniak
URL:
save_as: index.html

Then, after at least one empty line, follows the contents. You can use all of the standard Markdown syntax plus many extensions eg. from Github, such as syntax highlighting.

Previewing the site

After adding some test content, I run two simple commands: make html && make serve

igor@testvm:~/Dokumenty/web/web-random$ make html && make serve
"pelican" "/home/igor/Dokumenty/web/web-random/content" -o "/home/igor/Dokumenty/web/web-random/output" -s "/home/igor/Dokumenty/web/web-random/pelicanconf.py" 
[15:37:32] WARNING Watched path does not exist: /home/igor/Dokumenty/web/web-random/content/images log.py:91
Done: Processed 1 article, 0 drafts, 0 hidden articles, 0 pages, 0 hidden pages and 0 draft pages in 0.56 seconds.
"pelican" -l "/home/igor/Dokumenty/web/web-random/content" -o "/home/igor/Dokumenty/web/web-random/output" -s "/home/igor/Dokumenty/web/web-random/pelicanconf.py" 
Serving site at: http://127.0.0.1:8000 - Tap CTRL-C to stop

So far so good. I’ve got the working website and it doesn’t look very ugly. Articles are indexed by tags, category and date.

Linking

There is already an image on this page and I didn’t explain how to add them. Pelican has a concept of static files - they are copied to output directory without any change. You can configure different types of static files, images are already configured by default. Simply copy the files you need into content/images and use them like this:

![Pelican init]({static}/images/pelican-init.png)