Introduction
As part of my effort to reduce my dependency on the tech giants I am starting to host my own data as much as possible. Whilst this gives me the added privacy it also means I have to take care of my data a lot more and have my own means to backing up where as when I used a cloud provider this is generally done for me.
This is a guide on how to set up a raid 1 on the Turris Omnia with logical volumes to be used by the OS like physical partitions. Although it is written specifically for the Turris Omnia (Turris OS, which is OpenWRT) the guide can be adapted for most Linux based systems.
Requirements
The Turris Omnia, is a hackable kit running Turris OS (a variant of OpenWRT). The Turris board has mini pci-express slots, two of which I have wireless cards plugged into, I have one spare. There are USB ports as well as GPIO header pins. The OS runs on flash eMMC (circa 7GB) so is limited.
The router can run applications such as nextcloud and other web services but it has limited RAM (1GB) and limited disk space (eMMC) and the flash memory won’t last long when there are intensive I/O ops.
To make sure I have something I can store my data on in a resilient way I want to:
- Extend the Turris OS disk space;
- Ensure system can withstand disk failure;
- Receive alerts when there are any disk issues;
- Provide network access to the storage (NAS);
- Protect eMMC flash memory;
- Improve/optimise performance of the Turris OS wrt to solution.
Design Decisions
1. mdadm will be used for software raid
Basic design is to add more disks space to the Turris. To make sure we have continuity of data/service on disk failure we will have 2 disks in a raid 1 configuration. To do this we could use a hardware raid controller but the Turris is limited on options for this and the affordable mini raid controllers I’ve found don’t seem to be supported by Linux. We will use software raid that is well supported on Linux. The mdadm provides this.
2. Disks will be attached via a mini PCIe sata III controller/splitter
We have a number of options to attach more disks. There are two USB ports so we could attach the two disks to these. However, USB might not be the fastest option and it might present an inconvenience should we need these (external) ports for other reasons - we’d have to unplug a disk and that’s not ideal. Internally we could attach an mSata SSD but there isn’t space for two atm. The alternative is to use a mini PCIe disk controller that has two stata drive sockets. This is what shipped with the Turris Omnia NAS upgrade on Kickstarter so I am happier to go this route for now.
(note to self: there are mini PCIe cards with 4 sockets on, this could be interesting to try some other time).
3. Extend Turris OS to have /nas, /srv and dedicated swap.
To protect the eMMC we can make use of the extra disk space by mounting it to /srv. This will take any I/O intensive processes used by anything like MySQL, nextcloud or pakon away from the eMMC and onto the disks.
We also want to optimise performance of Turris OS and whilst 1GB RAM is sufficient at times this could become a bottleneck. If we create some swap space on the disk then the OS can use this for paging when needed.
The rest of the disk space can be used for a NAS, mounted at /nas and accessible from other devices/users on the network.
4. Logical volumes using LVM rather than physical partitions
Each use case (swap, srv, nas) has it’s own required file system jg. /srv has to be btrfs as that’s what Turris Omnia OS is. These are as follow:
- swap : swap
- /srv : btrfs
- /nas : ext4
To do this we could partition the disk into respective physical partitions and then format each one in the appropriate file system. However, this presents a few problems; The partitions should match in size and then each one would need to be created and assembled into a raid. If this is done then the partition sizes are static and can’t easily be extended if needed. Also when a disk fails and needs replacing the process to do so is complicated, each partition needs to be created on the disk with the right file system format first.
An alternative is to use LVM. By creating the raid on the whole disk before any partition has been written we can then configure the logical volumes on top of the raid. Logical volumes are more or less the same as physical partitions to the OS (once created) but because they are logical devices rather than physical they can be dynamically reconfigured and resized when needed. On disk failure replacing a disk is easier as the logical volumes sit on top of everything, the disk doesn’t need configuring before joining the raid.
5. Disk issue notifications will be sent via email
mdadm can be configured to send email on disk failure (there are a number of things mdadm can be configured to monitor on the disk). This will provide notifications on disk failure/issues.
An email client msmtp is included in the Turris OS distro so we will use that (others are available). Turris also provide Turris Omnia owners an smtp service for free for up to 10 or so emails a day so we will use this though alternative smtp services can be easily implemented.
6. The NAS will be accessed as a Samba share
For interoperability with Windows and other OSs we will access the /nas disk space through a samba share.
7. Everything will work on reboot
We will assemble the raid, mount the logical volumes and mount the samba shares on boot.
Hardware
- mini PCIe Sata iii 2 port card
- 2 x samsung SSD
- I also bought a cheap 2 x SSD caddy to fasten the drives into so they weren’t just resting on each other.
- Note: A power cable and sata leads shipped with the mini PCIe card though I needed to convert a cable to plug onto the Turris Omnia power connector
- Watch the Turris Omnia NAS installation on YouTube
Raid 1 using mdadm
opkg install mdadm
setting up the disks
root@turris:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
sdb 8:16 0 465.8G 0 disk
...
Then we will create the new raid device mb0
mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2
/dev/sda /dev/sdb
You’ll get notified of the disks content and asked to confirm creation of array. Type ‘Y’ when prompted to continue. The disks will then start to sync blocks.
This can be checked with cat /proc/mdstat
It will take some time for sync to complete.
Next it’s important to set up this device and assemble the array on reboot. We have mdadm running as a service under init.d, our config files are at /etc/config/mdadm.conf (sometimes conf files will be in different places and you may need to search for it).
We need to add the UUID into the config so find it in the output of:
mdadm --detail /dev/md0
The UUID needs setting in the array config /etc/config/mdadm
config mdadm
option email root
# list devices /dev/hd*
# list devices /dev/sd*
# list devices partitions
config array
option uuid c536a84a:xxxxxxxx:xxxxxxxx:xxxxxxxx
option device /dev/md0
option name turris:0
option metatdata 1.2
# option super_minor 0
# list devices /dev/sda1
# list devices /dev/sdb1
# option spares 0
# option spare_group spares
# option bitmap /bitmap.md
# option container 00000000:00000000:00000000:00000000
# option member 1
test the configuration:
/etc/init.d/mdadm restart
and then
mdadm --detail /dev/md0
finally make sure that the mdadm daemon will start on boot
/etc/init.d/mdadm enable
Partitions
* [LVM guide on archlinux](https://wiki.archlinux.org/index.php/LVM#LVM_Building_Blocks)
using lvm
lvscan
complains so remove eMMC from lvm devices by adding a filter in /etc/lvm/lvm.conf under devices section:
add filter = ["r|/dev/mmcblk0rpmb|" ]
find the raid device to partition
lvmdiskscan
lists out /dev/xxx with disk size amongst these are the md0 device.
create a physical volume (pv)
pvcreate --dataalignment 1m /dev/md0
[Creating PV on whole disk with no initial partition](https://serverfault.com/questions/356534/ssd-erase-block-size-lvm-pv-on-raw-device-alignment"read about why to use this pvcreate statement for our use-case")
- you can see the physical volume created with
pvdisplay
create a volume group which we will then put logical volumes onto.
vgcreate VolGroup00 /dev/md0
this can also be seen afterwards with
vgdisplay
Now we’re going to create the logical volumes.
We’ll create the /swap volume to be 2 GB, the /srv volume to be 13GB and then the /nas volume to be the remaining free space.
root@turris:/etc/lvm# lvcreate -L 2G VolGroup00 -n lvolswap
Logical volume "lvolswap" created.
root@turris:/etc/lvm# lvcreate -L 13G VolGroup00 -n lvolsrv
Logical volume "lvolsrv" created.
root@turris:/etc/lvm# lvcreate -l 100%FREE VolGroup00 -n lvolnas
Logical volume "lvolnas" created.
root@turris:/etc/lvm# ls /dev/VolGroup00/
lvolnas lvolsrv lvolswap
note the volumes have been created under the volume group in devices.
This can be checked by running
lvdisplay
We can now create file systems on these volumes, we’ll make volumes for swap and srv btrfs as this is the file system the Turris Omnia uses. We’ll make the nas volume ext4.
lvolsrv will mount to /srv, it is used by the main Turris OS as part of the main file system and so we will format as btrfs
mkfs.btrfs /dev/VolGroup00/lvolsrv
lvolswap will be used as swap, we will format this with the swap-space signature
mkswap /dev/VolGroup00/lvolswap
lvolnas will be set up as the network storage, for max flexibility we’ll format this as ext4
mkfs.ext4 /dev/VolGroup00/lvolnas
Mounting
Mounting the logical volumes can be done in either Luci > System > Mount points or in the fstab config found at /etc/config/fstab.
Read more about fstab and the attribute for mount points
pWe will need the UUIDs (I’ve replaced parts of them with ‘…’ in my example)
root@turris:~# blkid
/dev/mmcblk0: PTUUID="4dd..." PTTYPE="dos"
/dev/sda: UUID="c536a84a-..." UUID_SUB="afe4a919-..." LABEnL="turris:0" TYPE="linux_raid_member"
/dev/mmcblk0p1: UUID="c418d6a2-..." UUID_SUB="1d77500c-..." TYPE="btrfs" PARTUUID="4dd38318-01"
/dev/md0: UUID="A0WIOf-Jto6-..." TYPE="LVM2_member"
/dev/sdb: UUID="c536a84a-..." UUID_SUB="053d5ed8-..." LABEL="turris:0" TYPE="linux_raid_member"
/dev/mapper/VolGroup00-lvolswap: UUID="3c9bd300-..." TYPE="swap"
/dev/mapper/VolGroup00-lvolsrv: UUID="9cabced3-..." UUID_SUB="c5e3257f-..." TYPE="btrfs"
/dev/mapper/VolGroup00-lvolnas: UUID="a7c92f51-..." TYPE="ext4"
Note the UUID needed for the srv mount is btrfs and we will use the UUID_SUB.
edit the /etc/config/fstab file
config global
option anon_swap '0'
option anon_mount '0'
option auto_swap '1'
option auto_mount '1'
option delay_root '5'
option check_fs '0'
config swap
option enabled '1'
option uuid '3c9bd300-...'
config mount
option enabled '1'
option target '/srv'
option fstype 'btrfs'
option uuid 'c5e3257f-...'
config mount
option enabled '1'
option uuid 'a7c92f51-...'
option target '/nas'
option fstype 'ext4'
And then make sure that fstab is enabled
/etc/init.d/fstab enabled
and restart the service
/etc/init.d/fstab restart
we can also force the mounts to reload
mount -a
and finally check that they exist correctly
root@turris:~# df -h
Filesystem Size Used Available Use% Mounted on
/dev/mmcblk0p1 7.3G 1.9G 5.4G 26% /
tmpfs 503.3M 1.5M 501.8M 0% /tmp
tmpfs 512.0K 0 512.0K 0% /dev
/dev/mapper/VolGroup00-lvolnas
442.6G 72.0M 419.9G 0% /nas
/dev/mapper/VolGroup00-lvolsrv
13.0G 16.5M 11.0G 0% /srv
and that the swap is present
root@turris:~# free
total used free shared buffers cached
Mem: 1030732 258100 772632 1508 6112 63168
-/+ buffers/cache: 188820 841912
Swap: 2097148 0 2097148
or inspect the /proc/swaps
root@turris:~# cat /proc/swaps
Filename Type Size Used Priority
/dev/mapper/VolGroup00-lvolswap partition 2097148 0 -1
Getting email notifications
It’s all well having a raid set up but if a disk goes down and no alarms ring then we’ll never know there’s a problem until its too late when the other disk gets sick too.
We need a notification and mdadm is capable of checking and notifying via email if there are any issues.
First we need an email client that can send email via SMTP (we also need a mail server that allows smtp - I’m going to use my email providers server for that).
The mail client we will use is msmtp and can be installed
through luci or using opkg install msmtp, however, it is already
preinstalled on Turris OS.
To configure the SMTP client we need an smtp service to use, this could be from your favourite mail provider or if you’re using a Turris Omnia then you can use the Turris SMTP servers.
- Setting up msmtp on Turris Omnia with Turris SMTP service
- generic info on setting up msmtp on openwrt
Note if you are using a password then you may want to do something other than store the password in plain text in this file (though if anything compromises your router then maybe you’re in a much worse position anyway).
Note the use of option email root in the mdadm config we created
earlier and that you need to create an aliases file and reference this
in the msmtp config.
Once set up you can run a test with the following command.
mdadm --monitor --test --oneshot --scan
setting up Samba
The NAS needs to be accessed from other computers. One way of doing this so any OS (I think) can access it is to set up Samba shares.
We will use the /nas. The package we will use is called samba36-server and we are accessing the configs through the luci-app-smaba though this is preconfigured on the Turris Omnia if you select the NAS option under software in Foris.
It can also be managed through the terminal - reference the OpenWRT config for samba
I have set up a samba share with name NAS and path /nas to
match that of the mounted /nas volume.
Accessing the samba
The samba share (NAS) should be accessed by users of the router’s OS that have Samba passwords though these users do not need home directories nor should they be able to log into the OS.
We will create them under the group nobody which has a gid
65534.
There are 2 ways of adding the user. The first directly editing the user dir.
nano /etc/passwd
and creating a new line for a new user name, the first id is the user id and should be a number greater than 1000 that isn’t in use. The second is the group ID, this should be copied from the nobody user (65534).
The second way is to run
useradd --home-dir /var --gid 65534 --no-create-home --shell /bin/false newuser
where newuser is the username (a user id will automatically be
assigned).
We can then create this user a new samba password
smbpasswd -a newuser
A prompt will ask twice for the new password.
note you will want users from other systems to be able to access the share and write files and directories. This can be managed through the samba mask options for directories and files to set the permissions. - make sure the samba user has permissions on the share.
Mounting the Samba share from another device (linux)
On Linux I am using cifs to access the shamba share but there are other options too not covered here.
sudo apt-get update
sudo apt-get install cifs-utils
and then in /etc/fstab we need to add the line
//servernameorip/sambaname /mountpoint cifs credentials=/home/yourusername/.smbcredentials,,uid=1000,gid=1000,vers=1.0 0
Where the servername and share name is the ip address of the router and the NAS name we set up in the samba set up stage. The /mountpoint path is the path on the local machine to the folder we’re mounting the share to so in my case it’s /nas. ubuntutu username is your username, the home dir where you can store the credentials file.
Next we have to make the credentials file
touch ~/.smbcredentials
and add the relevant samba user credentials we created in the last step.
username=sanbausername
password=sambauserpassword
then save the file and change the permissions on it to secure it.
chmod 0600 ~/.smbcredentials
finally we can test the fstab
sudo mount -a
I had a puzzle where I could connect to a samba share through my kde file browser (dolphin) and write files and folders (once I had the permissions corrected for the samba user - see above) but I couldn’t write files through the command line and the mounted path. What I needed to do was match the user ID I created on the router with that of my current user ID on my local machine. It may be better to do this via matching up groups and setting group permissions then adding users to the groups. This is something I’ll look into.
Conclusion
In this guide we attached 2 disks onto a Turris Omnia and assembled them in raid 1 using mdadm. We created logical volumes using LVM, each with different file systems; one for swap, one for Turris OS /srv and one for a NAS. We then configured access to the NAS using samba and mounted the samba share on another device. Everything should rebuild/reassemble on reboot.
The next steps for me personally is to store a backup of my devices on the NAS and also to set up nextcloud so that I can store media assets from my mobile devices.