[conspire] external storage recommendation

[Josef Grosch]

On 9/25/21 1:40 PM, Rick Moen wrote:


[ DELETED ]

>
> I alluded earlier to ZFS (and its slightly lackluster Linux-native
> imitator, btrfs) as being in a different category relative to regular
> *ix filesystems like ext4, in ways that make it much more desirable for
> servers.  I should elaborate.
>
> ext4 is an excellent, high-performing, conservatively designed *ix
> filesystem.  There is nothing wrong with it.  Things it doesn't do, and
> doesn't aspire to do, include:
>
> o  background fsck and auto-repair / self-healing during normal system operation
> o  checksumming and vetting of every byte written, data and metadata
> o  data snaphots and replication
> o  native volume management
> o  native handling of RAID
> o  automatic rollback, in the event of detected error or inconsistency
> o  native data compression and de-duplication
> o  a lot more.  see https://en.wikipedia.org/wiki/ZFS#Summary
Another advantage of ZFS over RAID is RAID has the write hole problem. ( 
https://en.wikipedia.org/wiki/RAID#WRITE-HOLE ). Granted this is an 
uncommon problem but if one goes to the trouble of building a storage 
server why not cover all the bases.
>
> A regular *ix filesystem, even a really good one like ext4, has none of
> those things built-in.  As filesystem sizes and data collections
> (e.g., A/V files) get bigger and bigger, the risks from bitrot and
> data corruption silently occurring and accumulating over time become
> more worrisome, and also the risk of having multiple days of system
> downtime just because of need to correct filesystem errors during a
> reboot-driven fsck -- because you had 10 terabytes of files.
>    
> The whole idea of a file _server_ on your network is supposed to be for
> it to be a _reliable_ place to house your files, which is where the
> first two ZFS advantages listed really shine.  On the minus side, ZFS is
> fairly RAM-thirsty.  So, arguably worth the RAM overhead.
True. ZFS requires lots of memory but RAM has gotten very inexpensive in 
the last few years.The OpenZFS project and FreeNAS recommends a server 
with no less than 8 GB of RAM. The rule of thumb I have seen is 4 GB of 
RAM per core. When I built my current storage server I maxed out the RAM 
to the limit the motherboard supported, 64 GB.
>
> ZFS was one of several ground-breaking Sun Microsystems projects that
> gave Solaris and several compatibly licensed BSDs a functionality edge
> over Linux.  The others were dtrace (which is kernel-level debugging),
> Solaris Containers aka Zones (which are a better implementation of
> chroot jails), a Kernel-based Virtual Machine, and OpenSolaris Network
> Virtualization and Resource Control aka Crossbow (a set of features that
> provides an internal network virtualization and quality of service).
> All of those are C-coded in the Solaris kernel under CDDL terms, which
> clashes with Linux's GPLv2 terms, hence the code, even if ported to the
> Linux kernel, cannot be lawfully distributed in binary form, as that
> would be copyright violation.[1]
>
> btrfs is an independent attempt to implement most of ZFS's features in
> Linux.  However, even now, 14 years after its debut, it still has some
> problems.  Here, this runthrough will tell you about that, better than I
> could:
> https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/
>
ZFS is really the way to go if one wants/needs a mass storage server for 
backups or a media server. The two options for an open source solution 
is either FreeNAS ( http://truenas.com ) or install OpenZFS on your 
Linux server. My current server was initially Debian 10 recently 
upgraded to Debian 11. Installing ZFS was as easy as "sudo apt install 
zfs". I also installed build-essential and linux-headers-`uname -r`.

My current ZFS server that I built last year has the following specs:

* AMD Ryzen 5 2600X Six-Core Processor

* 64 gig of ram

* 6 - 6TB drives

* ASRock X470 Taichi motherboard

My previous server used an Intel CPU and I've found that AMD CPU seem to 
be faster. The one thing to note about ZFS is that it normally spawns 
off tons of threads so it is possible to run a ZFS system on a 2 core 
CPU but the performance is going to be dog slow. The 6 core AMD Ryzen 
with hyperthreading yields 12 cores.

The real issue that one has to be aware of is choosing the right discs. 
ZFS, like RAID, wants the discs to be all the same size. Mixing disc 
sizes causes real problems. For example if one had three 4 GB discs and 
one 1 GB disc the system would present 4 GB raw space. The other thing 
to be aware about is for ZFS ONLY use CMR / PMR (Perpendicular Magnetic 
Recording) discs. DO NOT use SMR (Shingled Magnetic Recording) discs, 
the performance hit is enormous. This YouTube video explains what is 
going on.

https://www.youtube.com/watch?v=aztTf2gI55k

A couple of other things to keep in mind.

* Do not have a ZFS space with more than 80% capacity, performance is 
very slow

* I run a nightly cron job to do a smarctl on all the discs in the ZFS 
pool and send myself and email with the results.

* I also run a weekly cron job to do a scrub on the ZFS pool. This 
catches any errors that might creep in

* Currently there is no official way to grow a ZFS volume in a pool. 
Once the volume is created that is it. I have seen posts that claim to 
be able to do this but every time I try that method it ends up a 
miserable failure. The OpenZFS project with the guys at Lawrence 
Livermore National Laboratory say that they are working on a method to 
allow one to add a disc to an existing pool and then grow the volume. 
Should be interesting.


Josef

-- 
Josef Grosch            | Another day closer |
jgrosch at MooseRiver.com  | to Redwood Heaven  | Berkeley, Ca.