[conspire] external storage recommendation

Fri Sep 24 15:04:26 PDT 2021

Paul, I was replying to you quite late, and didn't have the neural
firing power to say more than I did, but am now circling back.

To recap, your digital camera has a half-terabyte memory card of some
description (maybe an SD card).  By your description, it seems to
generate both video files and still digital photos.  Probably, the
memory card has the xfat filesystem.  And you want to be able to
periodically back up stuff off it to non-laptop-internal ancillary
storage.  Am guessing the intial step is to extract the memory stick
from your camera and mount it in a slot on the laptop -- and now you 
seek some flavour of external device for backed-up files to land,
without the process taking excessive time.

This page may be useful:
https://www.speedguide.net/faq/what-is-the-typical-usb-20-external-hard-drive-403

  Typical hi-speed USB 2.0 hard drives can be written to at rates around
  25-30 MB/s, and read at 30-40 MB/s. This represents ~70% of the total
  available bandwidth.

  According to a USB-IF chairman, "at least 10 to 15 percent of the stated
  peak 60 MB/s (480 Mbit/s) of Hi-Speed USB goes to overhead - the
  communication protocol between the card and the peripheral.

  Typical SuperSpeed USB 3.0 external hard drives can transfer large files
  at ~70 MB/s

  The USB standard supports the following speeds:

  USB 1.0 - 1.5 Mbit/s (~183kB/s) - intended primarily to save cost in
  low-bandwidth human interface devices (HID) such as keyboards, mice, and
  joysticks.

  USB 1.1 - 12 Mbit/s (~1.43 MB/s) - all USB 1.1 hubs support
  full-bandwidth.

  USB 2.0 (High-Speed, a.k.a. Hi-Speed) - 480 Mbit/s (~57 MB/s) - all
  devices are backward compatible with USB 1.1. Connectors are identical
  for USB 2.0 and USB 1.x.

  USB 3.0 (SuperSpeed) - 4800 Mbit/s, 4.8Gbits/s (~572 MB/s) - connectors
  are generally backwards compatible, but include new wiring and full
  duplex operation.

Worst-case for USB2-connected HD, then:  The half-terabyte memory stick
is almost full and you are performing initial backup (not backfilling
an existing backup using, e.g., rsync).  Shirtsleeve calculation:

500000 MB of storage, divided by 28 MB / sec = 17857 sec, which is
five hours of transfer time.  If this worst-case scenario is acceptable
to you, then just get an external USB2-connectable hard drive having a
couple'a terabytes of space, and you're done.  In fact, get two:
They're pretty cheap.

Typical backup will, of course _not_ be anything like worst-case,
because, being wise in such things, you will use an efficient tool like
rsync that copies only what's new to the target.

Compare NAS.  I'll tell you now, getting close to the _theoretical_ 
gigabit-ethernet saturation speed of 125 MB/sec takes a lot of attention
to detail, starting with making sure everything on the network segment
is using jumbo ethernet frames.  And cruddy ethernet NICs can be a
significant limiting factor, and ditto cruddy ethernet switches.  And
the CPU throughput on the NAS can also be a limiting factor.  In
consequence, Internet SWAGs (silly wild-ass guesses) about what
real-world throughput you'll get are all over the map.  The optimists
boast of getting 112 MB/sec.  The pessimists say 15-30 MB / sec, which
you'll note is the same as typical high-speed USB 2.0 hard disk transfer
rates.

So, bottom line, a NAS _might_ double or triple your speed over a
USB2-connected external HD, but then again it might not.

You be the judge.  You might want to start simple by buying a pair of
4GB USB2-connected external HDs, get out your labeller (you do have a
labeller, right?) and tag one as "Tweedledum Backup" and the other as
"Tweedledee Backup".  Then rsync the memory stick contents to them
alternately.

Of course, you might have other reasons for wanting a NAS, or just want
to play with NAS gear and software.  

I haven't played with NAS in a long time.  For OS/distro, you might look
at:

1.  OpenMediaVault.  https://www.openmediavault.org/
Debian-ish.  systemd.  Probably offers btrfs, by now, I hope? (if so,
better stick with RAID1).  Defaults, however, to ext4, which, SRSLY?
For a NAS?  2015 review: https://lwn.net/Articles/628284/ 

2.  TrueNAS Core (previously named FreeNAS),
https://www.truenas.com/truenas-core/  FreeBSD w/ZFS.

3.  XigmaNAS (previously named NAS4Free), https://xigmanas.com/xnaswp/
FreeBSD w/ZFS.

4.  Rockstor 4.  https://rockstor.com/  Now built on OpenSUSE, while
Rockstor 3 was CentOS, which of course didn't end well. btrfs.  systemd.
2015 review of Rockstor 3:  https://lwn.net/Articles/628705/

5   FreeNAS.  http://freenas.org/  FreeBSD w/ZFS.
2015 review:  https://lwn.net/Articles/631216/

6.  EasyNAS.  http://www.easynas.org/  OpenSUSE w/btrfs.  systemd.
2015 review:  https://lwn.net/Articles/635222/

7.  NexentaStor Community Edition.
https://community.nexenta.com/s/community-edition .  Illumos (ex-Solaris
open source) w/ZFS.  _But_ Nexenta's own content's licensing is
proprietary.  Quoting terms in part:  "Production use is not allowed
with Community edition. It is designed and intended to help people
become familiar with the product or for personal use."

The Illumos ecosystem is very fragmented with confusing branding, that
is hard to keep up with, e.g., XstreamOS (seems to be sponsored by
Sonicle of Italy) vs. SmartOS (sponsored by Joyent, mostly seems to be
hypervisor-focussed) vs. OpenIndiana (all-rounder effort, now under
illumos Foundation, and fully an open source community).  There are a
bunch of commercial efforts that want to suck you into proprietary
supersets based atop Illumos, e.g., the various editions of the NAS
system called napp-it.  Similarly, Xstream Storage is a proprietary
superset of XstreamOS.  You know me:  I consider those a "no thank you".

DIY using OpenIndiana would be a kick.
https://web.archive.org/web/20210304145916/https://wiki.openindiana.org/oi/Building+a+small+NAS+using+OI
The current action with OpenIndiana is a new-ish maintained branch
called "OpenIndiana Hipster" that is a rolling distro.

Here's an interesting comparison, albeit it's from 2015:
https://b3n.org/freenas-vs-omnios-napp-it/

This is another interesting comparison, and is current as of 2021
(but is a bit threadbare on details):
https://www.slant.co/topics/19199/~operating-systems-for-a-nas-file-server

In case it isn't obvious, part of the attraction of a NAS distro based
on Illumos or FreeBSD is getting well-developed, native ZFS -- which,
frankly, remains preferable to btrfs -- both of which are in a different
category from ext4 as to server-role functionality.  And, as I was
saying, with those (FreeBSD/Illumos), you have to be wary of the lure of
proprietary "value-add" tchochkes, and keep a wary eye on who has a
healthy open source community outside the walled garden, and who
doesn't.  (Also, hardware support is thinner than for Linux.)

Picking and assembling the hardware would be more fun.  I'll bet you
could make a really nice one, cost-effectively, using a PC Engines APU
System Board, mini-ITX NAS enclosure (4-bay or 8-bay), PSU, and some
hard drives.  At minimum, use the PC Engines boards as a standard of
comparison, as they really set the standard for server-grade quality in
embedded sizes.

But I'm guessing all of that is a bigger short-term hobby than you're
looking for, so start with the pair of USB2-connectable external hard
drives, I'd suggest.