This blog continues from the previous post describing the installation of Ubuntu and a virtualized IpFire dedicated firewall :
. Here I will be quickly describing the installation of ZFS, setting up your first zpool, some network shares and simple quotas. I will also include a health script I found on
Now to make life a bit easier, ZoL allows you to define an alias that resolves to a disk by id. This is easier when creating the pool, but is also easier when reading out your zpool status. To use these aliases create a tab delimited text file /etc/zfs/vdev_id.conf:
Now I assume you already have some dedicated disks you want to use to create the zpool (it is recommended to use full disks, but you can also create a pool from a number of partitions).
With the command 'zpool list' you can get an overview of some of the properties and used size by the pool. Note that it displays the raw sie and does not correct for the double parity.
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zfspool 10.9T 6.37T 4.51T 58% 1.00x ONLINE -
Shown are the name, total size (6 * 931 GB = 10.9 TB) and allocated / used space (6.37 TB , of which 4/6 is actual data (4 data disks, 2 parity disks) so 4.24 TB). Free is 4.51 (again 4/6 so 3.0 TB), meaning that the pool is in use for 58 %. Lastly it shows how much space is gained by deduplication but I have not turned that on due to the memory requirements.
After creation of the pool, it is created with owner root (given that it is created using su), so take ownership of the pool and subfolders (in a terminal with su):
chown -R username:username /zfspool
Note that you can destroy the pool with:
zpool destroy zfspool
When using ZFS it is recommended to create file systems (datasets) which are nested under the zpool. The idea is that this allows for much better use of the space based. One can ideally combine files of the same type (e.g. dataset for music, documents, etc).
These datasets can have different options / properties as can the zpool. Among this are compression, deduplication, mixed case sensitivity, or network share. So the actuall network sharing can be a property of the data set (both samba and nfs).
Above we used the zpool command, this controls and manages zpools. For datasets we use the 'zfs' command which controls and manges datasets.
Creation of a dataset is easy:
zfs create zfspool/backup
creating a dataset named "backup" which will be automounted under the zpool directory:
/zfspool/backup
Obviously this can be more complicated. Remember we already set casesensitivity and compression as properties of the pool above, if you don't want that for this dataset, they can be turned off:
zfs create -o casesensitivity=sensitive -o compression=off zfspool/backup
You can now show the properties of the dataset with:
zfs list
root@gertdus-server:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zfspool 4.24T 2.89T 416K /zfspool
zfspool/backup 242G 108G 242G /zfspool/backup
Showing you the space used and available (I have defined a quota here) and mountpoints. Note that now the correct space used is shown (contrary to zpool list, where lost space due to parity was not considered). The total is identical though.
Setting up network sharing using zfs is extremely simple. Though on Linux it requires a configured samba / nfs system. If you haven't install samba / nfs. Make sure that the user you want to share it for is also available as samba user and that the password is correct (see part III for further info on this).
Installing samba and nfs
apt-get install nfs-kernel-server samba
If your username is steve then use:
smbpasswd steve
Setting the actual share is as simple as:
sudo zfs set sharesmb=on zfspool/backup
or
sudo zfs set sharenfs=on zfspool/backup
You can then verify with
zfs get all | grep share
root@gertdus-server:~# zfs get all | grep share
zfspool sharenfs off default
zfspool sharesmb off default
zfspool/backup sharenfs off default
zfspool/backup sharesmb on local
Make sure that steve has permission to access all the files in your shared folders. This command inside your shares should help: (Change the name, obviously.)
cd /zfspool/backup
sudo -R chown username:username *
you might wish to change the workgroup name for Samba in /etc/samba/smb.conf.
Setting a dataset quota is also very simple. Unfortunately, to my knowledge there is no support for individual users or groups, just quota's on a per dataset level. Not setting them will have the effect that each and every data set you create get's listed as havng the full zpool worth of free space (which can be messy).
Setting the quota:
zfs set quota=1862G zfspool/backup
or
zfs set quota=1T zfspool/backup
check with
zfs get all | grep quota
root@gertdus-server:~# zfs get all | grep quota
zfspool quota none default
zfspool refquota none default
zfspool/backup quota 350G local
zfspool/backup refquota none default
Compression is another property that can be set to on. In particular using Lz4 can be highly advantageous as it requires not so much CPU (a resource usually not very stressed on a file server), but translates in much more effective use of space and quicker access.
When setting compression on simple clear text files (e.g. .xml, or .txt, *.sdf) the compression can be impressive (I get 1.73, meaning the space use is 1.73 times less, or taking up 58 % of the space when it would not be compressed).
Hence I would recommend always enabling / enabling it at a pool level (as shown above). ZFS supports multiple types, Lz4, Lzjb, and gzip. When you just use 'on' this turns on lzjb (default), however Lz4 has been shown to perform better (better compression) and faster.
Again this is a property that can be set with zfs:
zfs set compression=lz4 zfspool/werk
check with (using compress gives both the property and ratio)
zfs get all | grep compress
root@gertdus-server:~# zfs get all | grep compress
zfspool compressratio 1.09x -
zfspool compression lz4 local
zfspool refcompressratio 1.00x -
zfspool/backup compressratio 1.10x -
zfspool/backup compression lz4 local
zfspool/backup refcompressratio 1.10x -
zfspool/werk compressratio 1.73x -
zfspool/werk compression lz4 local
zfspool/werk refcompressratio 1.73x -
Easy as pie...
Note that the ratio for the whole pool is 1.09 , so the data takes up 92 % of the space, saving me in total 541 GB (8 % of the total 7.26 TB)!
ZFS provides the unique ability to validate the quality of your data via data scrubbing. Herein the data blocks are validated against their checksums and can be repaired. It is recommended to do so every once in a while to check for silent corruption. It is generally accepted that the time between scrubs can be higher using enterprise grade disks than when using consumer disks. Some people claims that the stress caused on the disks by doing a scrub leads to an lower life expectancy.
However the people at Backblaze show that this life expectancy is actually pretty good (http://blog.backblaze.com/2013/11/12/how-long-do-disk-drives-last/)
So I have settled on a scrub every 30 days (as included in the ZFS health script below).
Running a scrub is easy:
zpool scrub zfspool
And there you go, the time it takes differs based on the amount of disks, controller speed, cpu etc... (I get to about 230 MB/s, so a scrub runs in around 8-9 hours)..
This trick allows the user to run zpool status , zfs get (eliminates the need to sudo for quick reference, also allows you to use this cool app )
Perform all of the following as root or using sudo
leafpad /etc/udev/rules.d/91-zfs-permissions.rules
insert the contents:
#Use this to add a group and more permissive permissions for zfs
#so that you don't always need run it as root. beware, users not root
#can do nearly EVERYTHING, including, but not limited to destroying
#volumes and deleting datasets. they CANNOT mount datasets or create new
#volumes, export datasets via NFS, or other things that require root
#permissions outside of ZFS.
ACTION=="add", KERNEL=="zfs", MODE="0660", GROUP="zfs"
Adjust "GROUP" and "MODE" to your needs
run the following commands
groupadd zfs
gpasswd -a gertdus zfs
again, adjusting "zfs" to your match group in udev file.
And a screenshot
Here I will list some problems I encountered and simple solutions to them... (makes life easier)
When datasets are not automounted:
It's a bit dirty, but works. I have had automount problems with 0.6.2, not with 0.6.1.
add to /etc/rc.local
zfs mount -a
When kernelmodules are not recompiled after kernel upgrade:
reinstall dkms
apt-get install --reinstall zfs-dkms
Alternatively:
Get the version number of the registered modules:
dkms status
(eg: 0.6.0.90 for the daily ppa.)
Try to build the modules manually:
dkms remove -m zfs -v 0.6.0.90 --all
dkms remove -m spl -v 0.6.0.90 --all
dkms add -m spl -v 0.6.0.90
dkms add -m zfs -v 0.6.0.90
dkms install -m spl -v 0.6.0.90
dkms install -m zfs -v 0.6.0.90
If you get the same error, then reinstall the headers package:
apt-get install linux-headers-`uname -r` linux-headers-generic build-essential
Useful zpool commands:
sudo zpool list
list info on zfs pools rather than datasets
sudo zpool history zpoolname
shows history for named zpool
sudo zpool clear zpoolname
removes errors, starts resilvering if needed
sudo zdb -DD zpoolname
shows deduplication histogram
sudo zpool iostat
shows IO parameters
sudo zpool scrub zpoolname
manually starts scrub job
zpool set propertyname=value zpoolname
zpool set propertyname=value datasetname
change single property: e.g. zfs set dedup=on zfstest / zfs set sharesemb=on zfstest
Useful zfs commands
zfs mount
display mounted sets
zfs get mountpoint datasetname
get mountpoint of dataset
zfs set mountpoint=/foo_mount data
set mountpoint of dataset
sudo zfs mount -a
mounts all datasets
sudo zfs mount datasetname
mounts said dataset
zfs unmount datasetname
zfs unmount -a
unmount sets similar to above
zfs get all zpoolname
zfs get all datasetname
get all information on the zpool or dataset
zfs get propertyname zpoolname
zfs get propertyname datasetname
get information on 1 property : e.g. zfs get dedup zfstest / zfs get sharesmb zfstest
ZFS health script form calomel.org
#! /bin/bash
#
# Calomel.org
# https://calomel.org/zfs_health_check_script.html
# FreeBSD 9.1 ZFS Health Check script
# zfs_health.sh @ Version 0.15
# Check health of ZFS volumes and drives. On any faults send email. In FreeBSD
# 10 there is supposed to be a ZFSd daemon to monitor the health of the ZFS
# pools. For now, in FreeBSD 9, we will make our own checks and run this script
# through cron a few times a day.
# 99 problems but ZFS aint one
problems=0
# Health - Check if all zfs volumes are in good condition. We are looking for
# any keyword signifying a degraded or broken array.
condition=$(/sbin/zpool status | egrep -i '(DEGRADED|FAULTED|OFFLINE|UNAVAIL|REMOVED|FAIL|DESTROYED|corrupt|cannot|unrecover)')
if [ "${condition}" ]; then
emailSubject="`hostname` - ZFS pool - HEALTH fault"
problems=1
fi
# Capacity - Make sure pool capacities are below 80% for best performance. The
# percentage really depends on how large your volume is. If you have a 128GB
# SSD then 80% is reasonable. If you have a 60TB raid-z2 array then you can
# probably set the warning closer to 95%.
#
# ZFS uses a copy-on-write scheme. The file system writes new data to
# sequential free blocks first and when the uberblock has been updated the new
# inode pointers become valid. This method is true only when the pool has
# enough free sequential blocks. If the pool is at capacity and space limited,
# ZFS will be have to randomly write blocks. This means ZFS can not create an
# optimal set of sequential writes and write performance is severely impacted.
maxCapacity=80
if [ ${problems} -eq 0 ]; then
capacity=$(/sbin/zpool list -H -o capacity)
for line in ${capacity//%/}
do
if [ $line -ge $maxCapacity ]; then
emailSubject="`hostname` - ZFS pool - Capacity Exceeded"
problems=1
fi
done
fi
# Errors - Check the columns for READ, WRITE and CKSUM (checksum) drive errors
# on all volumes and all drives using "zpool status". If any non-zero errors
# are reported an email will be sent out. You should then look to replace the
# faulty drive and run "zpool scrub" on the affected volume after resilvering.
if [ ${problems} -eq 0 ]; then
errors=$(/sbin/zpool status | grep ONLINE | grep -v state | awk '{print $3 $4 $5}' | grep -v 000)
if [ "${errors}" ]; then
emailSubject="`hostname` - ZFS pool - Drive Errors"
problems=1
fi
fi
# Scrub Expired - Check if all volumes have been scrubbed in at least the last
# 8 days. The general guide is to scrub volumes on desktop quality drives once
# a week and volumes on enterprise class drives once a month. You can always
# use cron to schedule "zpool scrub" in off hours. We scrub our volumes every
# Sunday morning for example.
#
# Scrubbing traverses all the data in the pool once and verifies all blocks can
# be read. Scrubbing proceeds as fast as the devices allows, though the
# priority of any I/O remains below that of normal calls. This operation might
# negatively impact performance, but the file system will remain usable and
# responsive while scrubbing occurs. To initiate an explicit scrub, use the
# "zpool scrub" command.
#
# The scrubExpire variable is in seconds. So for 8 days we calculate 8 days
# times 24 hours times 3600 seconds to equal 691200 seconds.
#30days
scrubExpire=2592000
if [ ${problems} -eq 0 ]; then
currentDate=$(date +%s)
zfsVolumes=$(/sbin/zpool list -H -o name)
for volume in ${zfsVolumes}
do
if [ $(/sbin/zpool status $volume | egrep -c "none requested") -ge 1 ]; then
echo "ERROR: You need to run \"zpool scrub $volume\" before this script can monitor the scrub expiration time."
break
fi
if [ $(/sbin/zpool status $volume | egrep -c "scrub in progress|resilver") -ge 1 ]; then
break
fi
### FreeBSD with *nix supported date format
#scrubRawDate=$(/sbin/zpool status $volume | grep scrub | awk '{print $15 $12 $13}')
#scrubDate=$(date -j -f '%Y%b%e-%H%M%S' $scrubRawDate'-000000' +%s)
### Ubuntu with GNU supported date format
scrubRawDate=$(/sbin/zpool status $volume | grep scrub | awk '{print $11" "$12" " $13" " $14" "$15}')
scrubDate=$(date -d "$scrubRawDate" +%s)
if [ $(($currentDate - $scrubDate)) -ge $scrubExpire ]; then
emailSubject="`hostname` - ZFS pool - Scrub Time Expired. Scrub Needed on $volume"
problems=1
fi
done
fi
# Notifications - On any problems send email with drive status information and
# capacities including a helpful subject line to root. Also use logger to write
# the email subject to the local logs. This is the place you may want to put
# any other notifications like:
#
# + Update an anonymous twitter account with your ZFS status (https://twitter.com/zfsmonitor)
# + Playing a sound file or beep the internal speaker
# + Update Nagios, Cacti, Zabbix, Munin or even BigBrother
if [ "$problems" -ne 0 ]; then
echo -e "$emailSubject \n\n\n `/sbin/zpool list` \n\n\n `/sbin/zpool status`" | mail -s "$emailSubject" root
logger $emailSubject
fi
if [ "$problems" -eq 0 ]; then
echo "ZFS Healthy"
fi
### EOF ###
