Series : Ubuntu Home Server (ZFS + virtual IpFire)
-->Part II : Setting up ZFS (this post)<--
Setting up ZFS
Table of Contents:
Installation of ZFSCreation of the first zpool
Having a look at your first pool
Using Datasets
Dataset properties - Network sharing
Dataset properties - Quotas
Dataset properties - Compression
Data Scrubbing
Give user permission for status polling of ZFS
Troubleshooting
Towards a next generation file system at home
This blog continues from the previous post describing the installation of Ubuntu and a virtualized IpFire dedicated firewall : http://blog.gjpvanwesten.nl/2013/11/howto-ubuntu-home-server-with-zfs-and.html. Here I will be quickly describing the installation of ZFS, setting up your first zpool, some network shares and simple quotas. I will also include a health script I found on Calomel.org.
Specs:
As a reminder, here's what I set up:- Ubuntu 12.04.03 LTS
- ZFS on Linux (6 disk RAIDZ2 , double parity)
- Virtual firewall (IpFire) running through Virtualbox
- DHCP / DNS / samba file sharing
- LAN runs in the 192.168.10.xxx range.
Running on:
- Athlon II X3
- 16 GB ECC RAM
- 3 Ethernet adapters (2 for the firewall and 1 for the host OS)
- 6 WD 2 TB disks (mix of Green and Red)
- 500 GB bootdrive (2.5 " WD)
Installation of ZFS:
Installing zfs is actually very simple when you use the native implementation by ZFS on Linux (ZoL). They actually have a PPA with binaries available which makes the process rather trivial. Before we do so make sure you have performed the necessary preparations.
Open a terminal and become root (sudo -s).
First install dkms to allow rebuilding the kernel modules for zfs after kernel updates:
sudo apt-get install dkms
I guess you need to reboot after installing these, if you already have them you can continue. Again have a terminal with su and do:
sudo apt-get install linux-headers-`uname -r` linux-headers-generic build-essential
Now to install zfs:
apt-add-repository ppa:zfs-native/stable
apt-get update
apt-get install ubuntu-zfs
I guess this requires another reboot, after which you can again open a terminal an become root.
Type :
zpool status
If you don't get an error but see "no zpools available" or equivalent, then installation was successful and we can move forward to creating the zpool.
Creation of the first zpool :
A zpool is a virtual device that contains the pooled storage space of the underlying devices. Hence a zpool can be as simple as a single drive, but it can also be a complex combination of virtual RAID5 devices that are striped and form a RAID50. For further reading I would refer you to the FAQ of ZoL or the Oracle documentation. Here I assume some prior knowledge.I will use a pool consisting of a single RAIDZ2 virtual device (vdev) as an example. This is a pool consisting of 6 2TB disks, of which 2 are used for parity. Hence this array can sustain two disk failures and still function. The total size is 6 * 2 TB minus 2 * 2TB (parity) = 8 TB.
The creation of a zpool can be done with any vali disk identifier, but is recommended to be done with disk ids. The reason for this is that pools will not break when you add new disks to the system (which would be the case when using /dev/sda) and that the zpool is not dependant on the actual hardwired connectivity. When using ids, I have successfully switched disks in my pool from being connected to a raid expansion to being connected via the onboard sata and there was not problem what so ever.
Now to make life a bit easier, ZoL allows you to define an alias that resolves to a disk by id. This is easier when creating the pool, but is also easier when reading out your zpool status. To use these aliases create a tab delimited text file /etc/zfs/vdev_id.conf:
leafpad /etc/zfs/vdev_id.conf
Now I assume you already have some dedicated disks you want to use to create the zpool (it is recommended to use full disks, but you can also create a pool from a number of partitions).
In the file enter the following (each line represents a disk, this is my current setup): # by-vdev # name fully qualified or base name of device link alias 2tb1 /dev/disk/by-id/scsi-SATA_WDC_WD20EFRX-68_WD-WMC301348703 alias 2tb2 /dev/disk/by-id/scsi-SATA_WDC_WD20EFRX-68_WD-WMC301355426 alias 2tb3 /dev/disk/by-id/scsi-SATA_WDC_WD20EARX-00_WD-WCAZAL405566 alias 2tb4 /dev/disk/by-id/scsi-SATA_WDC_WD20EFRX-68_WD-WCC300687955 alias 2tb5 /dev/disk/by-id/scsi-SATA_WDC_WD20EARX-00_WD-WCAZAL762406 alias 2tb6 /dev/disk/by-id/scsi-SATA_WDC_WD20EZRX-00_WD-WCC300740429 Now close the file and run the following to enable the aliases: udevadm trigger
The next step is actual creation of the pool, now I use a raidz2, but you can also create a raidz1 or mirror, or set of both. When creating the pool you can choose to use 512 bit sector sizes of 4096 bit sector sizes (common on modern drives). The latter should perform better but will lead to more lost space in particular when storing smaller files. I am not going into all the details but some further documentation can be found here, here, and here .
In principle when using raidz, ideally one should use an even number of data disks coupled to the amount of parity disk required by the raidz version. So
Raidz1
2 data + 1 parity = 3
4 data + 1 parity = 5
n data + 1 parity = n + 1
Raidz2
2 data + 2 parity = 4 (although wasteful)
4 data + 2 parity = 6
n data + 2 parity = n + 2
In principle when using raidz, ideally one should use an even number of data disks coupled to the amount of parity disk required by the raidz version. So
Raidz1
2 data + 1 parity = 3
4 data + 1 parity = 5
n data + 1 parity = n + 1
Raidz2
2 data + 2 parity = 4 (although wasteful)
4 data + 2 parity = 6
n data + 2 parity = n + 2
Raidz3
2 data + 3 parity = 5 (although wasteful)
4 data + 3 parity = 7
n data + 1 parity = n + 3
2 data + 3 parity = 5 (although wasteful)
4 data + 3 parity = 7
n data + 1 parity = n + 3
I am creating a 6 disk raidz2.
The command is (in a terminal with su):
zpool create -o ashift=12 -O casesensitivity=mixed -O compression=lz4 zfspool raidz2 2tb1 2tb2 2tb3 2tb4 2tb5 2tb6
Let me go through all the parts of the command in turn:
zpool (zpool is used for the administration of zpools)
create (tell zpool to create a new pool)
-o ashift=12 (I will use a 4096 bit sector size)
-O casesensitivity=mixed (I intend to create shares for windows using samba)
-O compression=lz4 (turn on block compression on all files, found to perform well and very vast)
zfspool (name of the pool to be created)
raidz2 (type of the pool, can be substituted with raidz1, mirror, etc)
2tb1 2tb2 2tb3 2tb4 2tb5 2tb6 (individual disk members)
Uppercase 'O' is for properties of datasets (children of zpools, inherited by default) and lowercase are properties of the zpool.
If you were successful again run "zpool status":Uppercase 'O' is for properties of datasets (children of zpools, inherited by default) and lowercase are properties of the zpool.
Having a look at your first pool:
root@gertdus-server:~# zpool status
pool: zfspool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zfspool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
2tb1 ONLINE 0 0 0
2tb2 ONLINE 0 0 0
2tb3 ONLINE 0 0 0
2tb4 ONLINE 0 0 0
2tb5 ONLINE 0 0 0
2tb6 ONLINE 0 0 0
errors: No known data errors
The zpool will be automounted by default under it's zpool name, hence you can find it at :
/zfspool
obviously you can change the mountpoint (further reading).
zfs set mountpoint=/foo/bar zfspool
Now lets interpret the output. the overview shows the name of your pool:
status - can be ONLINE, DEGRADED (member failed but not critical), FAULTED (critical failure), OFFLINE (manual set in offline state)
scan - indicates if a data scan (scrub) or rebuild (resilvering in zfs) is running
config - lists the different pools on the system (here 1). Shown are the pool and it's state, the underlying vdevs and their state and the state of each individual disk / partition. If all is well the values for state is "ONLINE" and the values for READ/WRITE/CKSUM are '0'.
Other states vdevs and pools can be in are DEGRADED (member failed but not critical), FAULTED (critical failure), OFFLINE (manual set in offline state).
Disks can be UNAVAILABLE (missing), FAULTED (smart error, corrup data), OFFLINE (manual set in offline state).
With the command 'zpool list' you can get an overview of some of the properties and used size by the pool. Note that it displays the raw sie and does not correct for the double parity.
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zfspool 10.9T 6.37T 4.51T 58% 1.00x ONLINE -
Shown are the name, total size (6 * 931 GB = 10.9 TB) and allocated / used space (6.37 TB , of which 4/6 is actual data (4 data disks, 2 parity disks) so 4.24 TB). Free is 4.51 (again 4/6 so 3.0 TB), meaning that the pool is in use for 58 %. Lastly it shows how much space is gained by deduplication but I have not turned that on due to the memory requirements.
After creation of the pool, it is created with owner root (given that it is created using su), so take ownership of the pool and subfolders (in a terminal with su):
chown -R username:username /zfspool
Note that you can destroy the pool with:
zpool destroy zfspool
Using datasets:
When using ZFS it is recommended to create file systems (datasets) which are nested under the zpool. The idea is that this allows for much better use of the space based. One can ideally combine files of the same type (e.g. dataset for music, documents, etc).
These datasets can have different options / properties as can the zpool. Among this are compression, deduplication, mixed case sensitivity, or network share. So the actuall network sharing can be a property of the data set (both samba and nfs).
Above we used the zpool command, this controls and manages zpools. For datasets we use the 'zfs' command which controls and manges datasets.
Creation of a dataset is easy:
zfs create zfspool/backup
creating a dataset named "backup" which will be automounted under the zpool directory:
/zfspool/backup
Obviously this can be more complicated. Remember we already set casesensitivity and compression as properties of the pool above, if you don't want that for this dataset, they can be turned off:
zfs create -o casesensitivity=sensitive -o compression=off zfspool/backup
You can now show the properties of the dataset with:
zfs list
root@gertdus-server:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zfspool 4.24T 2.89T 416K /zfspool
zfspool/backup 242G 108G 242G /zfspool/backup
Showing you the space used and available (I have defined a quota here) and mountpoints. Note that now the correct space used is shown (contrary to zpool list, where lost space due to parity was not considered). The total is identical though.
Dataset properties - Network sharing:
Setting up network sharing using zfs is extremely simple. Though on Linux it requires a configured samba / nfs system. If you haven't install samba / nfs. Make sure that the user you want to share it for is also available as samba user and that the password is correct (see part III for further info on this).Installing samba and nfs
apt-get install nfs-kernel-server samba
If your username is steve then use:
smbpasswd steve
Setting the actual share is as simple as:
sudo zfs set sharesmb=on zfspool/backup
or
sudo zfs set sharenfs=on zfspool/backup
You can then verify with
zfs get all | grep share
root@gertdus-server:~# zfs get all | grep share
zfspool sharenfs off default
zfspool sharesmb off default
zfspool/backup sharenfs off default
zfspool/backup sharesmb on local
Make sure that steve has permission to access all the files in your shared folders. This command inside your shares should help: (Change the name, obviously.)
cd /zfspool/backup
sudo -R chown username:username *
you might wish to change the workgroup name for Samba in /etc/samba/smb.conf.
Dataset properties - Quotas:
Setting a dataset quota is also very simple. Unfortunately, to my knowledge there is no support for individual users or groups, just quota's on a per dataset level. Not setting them will have the effect that each and every data set you create get's listed as havng the full zpool worth of free space (which can be messy).
Setting the quota:
zfs set quota=1862G zfspool/backup
or
zfs set quota=1T zfspool/backup
check with
zfs get all | grep quota
root@gertdus-server:~# zfs get all | grep quota
zfspool quota none default
zfspool refquota none default
zfspool/backup quota 350G local
zfspool/backup refquota none default
Dataset properties - Compression:
Compression is another property that can be set to on. In particular using Lz4 can be highly advantageous as it requires not so much CPU (a resource usually not very stressed on a file server), but translates in much more effective use of space and quicker access.
When setting compression on simple clear text files (e.g. .xml, or .txt, *.sdf) the compression can be impressive (I get 1.73, meaning the space use is 1.73 times less, or taking up 58 % of the space when it would not be compressed).
Hence I would recommend always enabling / enabling it at a pool level (as shown above). ZFS supports multiple types, Lz4, Lzjb, and gzip. When you just use 'on' this turns on lzjb (default), however Lz4 has been shown to perform better (better compression) and faster.
Again this is a property that can be set with zfs:
zfs set compression=lz4 zfspool/werk
check with (using compress gives both the property and ratio)
zfs get all | grep compress
root@gertdus-server:~# zfs get all | grep compress
zfspool compressratio 1.09x -
zfspool compression lz4 local
zfspool refcompressratio 1.00x -
zfspool/backup compressratio 1.10x -
zfspool/backup compression lz4 local
zfspool/backup refcompressratio 1.10x -
zfspool/werk compressratio 1.73x -
zfspool/werk compression lz4 local
zfspool/werk refcompressratio 1.73x -
Easy as pie... Note that the ratio for the whole pool is 1.09 , so the data takes up 92 % of the space, saving me in total 541 GB (8 % of the total 7.26 TB)!
Data Scrubbing:
ZFS provides the unique ability to validate the quality of your data via data scrubbing. Herein the data blocks are validated against their checksums and can be repaired. It is recommended to do so every once in a while to check for silent corruption. It is generally accepted that the time between scrubs can be higher using enterprise grade disks than when using consumer disks. Some people claims that the stress caused on the disks by doing a scrub leads to an lower life expectancy.However the people at Backblaze show that this life expectancy is actually pretty good (http://blog.backblaze.com/2013/11/12/how-long-do-disk-drives-last/) So I have settled on a scrub every 30 days (as included in the ZFS health script below).
Running a scrub is easy:
zpool scrub zfspool
And there you go, the time it takes differs based on the amount of disks, controller speed, cpu etc... (I get to about 230 MB/s, so a scrub runs in around 8-9 hours)..
Give user permission for status polling of ZFS (tested on Ubuntu 12 / CentOS 6):
This trick allows the user to run zpool status , zfs get (eliminates the need to sudo for quick reference, also allows you to use this cool app )Perform all of the following as root or using sudo
leafpad /etc/udev/rules.d/91-zfs-permissions.rules
insert the contents:
#Use this to add a group and more permissive permissions for zfs
#so that you don't always need run it as root. beware, users not root
#can do nearly EVERYTHING, including, but not limited to destroying
#volumes and deleting datasets. they CANNOT mount datasets or create new
#volumes, export datasets via NFS, or other things that require root
#permissions outside of ZFS.
ACTION=="add", KERNEL=="zfs", MODE="0660", GROUP="zfs"
Adjust "GROUP" and "MODE" to your needs
run the following commands
groupadd zfs
gpasswd -a gertdus zfs
again, adjusting "zfs" to your match group in udev file.
And a screenshot
Troubleshooting:
Here I will list some problems I encountered and simple solutions to them... (makes life easier)When datasets are not automounted:
It's a bit dirty, but works. I have had automount problems with 0.6.2, not with 0.6.1.add to /etc/rc.local
zfs mount -a
When kernelmodules are not recompiled after kernel upgrade:
reinstall dkms
apt-get install --reinstall zfs-dkms
Alternatively:
Get the version number of the registered modules:
dkms status
(eg: 0.6.0.90 for the daily ppa.)
Try to build the modules manually:
dkms remove -m zfs -v 0.6.0.90 --all
dkms remove -m spl -v 0.6.0.90 --all
dkms add -m spl -v 0.6.0.90
dkms add -m zfs -v 0.6.0.90
dkms install -m spl -v 0.6.0.90
dkms install -m zfs -v 0.6.0.90
If you get the same error, then reinstall the headers package:
apt-get install linux-headers-`uname -r` linux-headers-generic build-essential
Useful zpool commands:
sudo zpool list
list info on zfs pools rather than datasets
sudo zpool history zpoolname
shows history for named zpool
sudo zpool clear zpoolname
removes errors, starts resilvering if needed
sudo zdb -DD zpoolname
shows deduplication histogram
sudo zpool iostat
shows IO parameters
sudo zpool scrub zpoolname
manually starts scrub job
zpool set propertyname=value zpoolname
zpool set propertyname=value datasetname
change single property: e.g. zfs set dedup=on zfstest / zfs set sharesemb=on zfstest
Useful zfs commands
zfs mount
display mounted sets
zfs get mountpoint datasetname
get mountpoint of dataset
zfs set mountpoint=/foo_mount data
set mountpoint of dataset
sudo zfs mount -a
mounts all datasets
sudo zfs mount datasetname
mounts said dataset
zfs unmount datasetname
zfs unmount -a
unmount sets similar to above
zfs get all zpoolname
zfs get all datasetname
get all information on the zpool or dataset
zfs get propertyname zpoolname
zfs get propertyname datasetname
get information on 1 property : e.g. zfs get dedup zfstest / zfs get sharesmb zfstest
ZFS health script form calomel.org
#! /bin/bash
#
# Calomel.org
# https://calomel.org/zfs_health_check_script.html
# FreeBSD 9.1 ZFS Health Check script
# zfs_health.sh @ Version 0.15
# Check health of ZFS volumes and drives. On any faults send email. In FreeBSD
# 10 there is supposed to be a ZFSd daemon to monitor the health of the ZFS
# pools. For now, in FreeBSD 9, we will make our own checks and run this script
# through cron a few times a day.
# 99 problems but ZFS aint one
problems=0
# Health - Check if all zfs volumes are in good condition. We are looking for
# any keyword signifying a degraded or broken array.
condition=$(/sbin/zpool status | egrep -i '(DEGRADED|FAULTED|OFFLINE|UNAVAIL|REMOVED|FAIL|DESTROYED|corrupt|cannot|unrecover)')
if [ "${condition}" ]; then
emailSubject="`hostname` - ZFS pool - HEALTH fault"
problems=1
fi
# Capacity - Make sure pool capacities are below 80% for best performance. The
# percentage really depends on how large your volume is. If you have a 128GB
# SSD then 80% is reasonable. If you have a 60TB raid-z2 array then you can
# probably set the warning closer to 95%.
#
# ZFS uses a copy-on-write scheme. The file system writes new data to
# sequential free blocks first and when the uberblock has been updated the new
# inode pointers become valid. This method is true only when the pool has
# enough free sequential blocks. If the pool is at capacity and space limited,
# ZFS will be have to randomly write blocks. This means ZFS can not create an
# optimal set of sequential writes and write performance is severely impacted.
maxCapacity=80
if [ ${problems} -eq 0 ]; then
capacity=$(/sbin/zpool list -H -o capacity)
for line in ${capacity//%/}
do
if [ $line -ge $maxCapacity ]; then
emailSubject="`hostname` - ZFS pool - Capacity Exceeded"
problems=1
fi
done
fi
# Errors - Check the columns for READ, WRITE and CKSUM (checksum) drive errors
# on all volumes and all drives using "zpool status". If any non-zero errors
# are reported an email will be sent out. You should then look to replace the
# faulty drive and run "zpool scrub" on the affected volume after resilvering.
if [ ${problems} -eq 0 ]; then
errors=$(/sbin/zpool status | grep ONLINE | grep -v state | awk '{print $3 $4 $5}' | grep -v 000)
if [ "${errors}" ]; then
emailSubject="`hostname` - ZFS pool - Drive Errors"
problems=1
fi
fi
# Scrub Expired - Check if all volumes have been scrubbed in at least the last
# 8 days. The general guide is to scrub volumes on desktop quality drives once
# a week and volumes on enterprise class drives once a month. You can always
# use cron to schedule "zpool scrub" in off hours. We scrub our volumes every
# Sunday morning for example.
#
# Scrubbing traverses all the data in the pool once and verifies all blocks can
# be read. Scrubbing proceeds as fast as the devices allows, though the
# priority of any I/O remains below that of normal calls. This operation might
# negatively impact performance, but the file system will remain usable and
# responsive while scrubbing occurs. To initiate an explicit scrub, use the
# "zpool scrub" command.
#
# The scrubExpire variable is in seconds. So for 8 days we calculate 8 days
# times 24 hours times 3600 seconds to equal 691200 seconds.
#30days
scrubExpire=2592000
if [ ${problems} -eq 0 ]; then
currentDate=$(date +%s)
zfsVolumes=$(/sbin/zpool list -H -o name)
for volume in ${zfsVolumes}
do
if [ $(/sbin/zpool status $volume | egrep -c "none requested") -ge 1 ]; then
echo "ERROR: You need to run \"zpool scrub $volume\" before this script can monitor the scrub expiration time."
break
fi
if [ $(/sbin/zpool status $volume | egrep -c "scrub in progress|resilver") -ge 1 ]; then
break
fi
### FreeBSD with *nix supported date format
#scrubRawDate=$(/sbin/zpool status $volume | grep scrub | awk '{print $15 $12 $13}')
#scrubDate=$(date -j -f '%Y%b%e-%H%M%S' $scrubRawDate'-000000' +%s)
### Ubuntu with GNU supported date format
scrubRawDate=$(/sbin/zpool status $volume | grep scrub | awk '{print $11" "$12" " $13" " $14" "$15}')
scrubDate=$(date -d "$scrubRawDate" +%s)
if [ $(($currentDate - $scrubDate)) -ge $scrubExpire ]; then
emailSubject="`hostname` - ZFS pool - Scrub Time Expired. Scrub Needed on $volume"
problems=1
fi
done
fi
# Notifications - On any problems send email with drive status information and
# capacities including a helpful subject line to root. Also use logger to write
# the email subject to the local logs. This is the place you may want to put
# any other notifications like:
#
# + Update an anonymous twitter account with your ZFS status (https://twitter.com/zfsmonitor)
# + Playing a sound file or beep the internal speaker
# + Update Nagios, Cacti, Zabbix, Munin or even BigBrother
if [ "$problems" -ne 0 ]; then
echo -e "$emailSubject \n\n\n `/sbin/zpool list` \n\n\n `/sbin/zpool status`" | mail -s "$emailSubject" root
logger $emailSubject
fi
if [ "$problems" -eq 0 ]; then
echo "ZFS Healthy"
fi
### EOF ###
