Development tests and tools for scubed3 ======================================= This directory contains the tests and the tools that were used while developing scubed3: torture tests for growing, shrinking, crashes and torn writes, tests with real filesystems on top, statistical tests of the write order (paranoia levels), and offline tools that read a base file without the daemon. They are not needed to use scubed3. Requirements ~~~~~~~~~~~~ - Linux with FUSE (libfuse 3; the tests unmount with fusermount3, or with fusermount if that is not there), run as root (FUSE mounts, loop devices, mount(8)) - python3, with the "cryptography" module for s3read.py and s3diff.py (and the tests that use s3read.py: corrupttest.py, ioerrtest.py, fsrepair.py) - e2fsprogs (mkfs.ext2/3/4, e2fsck, resize2fs), util-linux (losetup, fstrim, mount, runuser, script), coreutils (timeout) - for sockettest.sh: the user "nobody" - optional: xfsprogs, see discardtest.py - the mountpoint /mnt/s3 (torture.py creates it, the other tests expect it), the tests create their own mountpoints /mnt/... for the filesystems - NO OTHER scubed3 daemon may be running: the tests use the default control socket /run/scubed3/control and the mountpoint /mnt/s3 The tests create their base files (and logs) in the current directory, or in $S3BUILD if it is set; most of them are 64 MiB to 1.6 GiB. Building ~~~~~~~~ sh dev/build.sh [OUTDIR] builds, from ../src, in OUTDIR (default: the current directory): scubed3 the daemon scubed3-crash the daemon with -DSCUBED3_CRASH_TEST, it has test hooks (see below), several tests need it scubed3ctl the control program jstat the unit test of juggler.c (jstat.c) configure must have been run in the main tree (config.h). EXTRA_CFLAGS and EXTRA_LDFLAGS are added to the compiler and linker flags (for example for libraries in a non-standard place). The tests take the NAME of the daemon binary, they look for it in the current directory (or $S3BUILD). Builds with sanitizers (every test runs with them, in their own OUTDIR; -O1 for usable stack traces): EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -O1 -Wno-format-truncation" \ EXTRA_LDFLAGS="-fsanitize=address,undefined" sh dev/build.sh asan EXTRA_CFLAGS="-fsanitize=thread -O1 -Wno-format-truncation" \ EXTRA_LDFLAGS="-fsanitize=thread" sh dev/build.sh tsan AddressSanitizer and UndefinedBehaviorSanitizer find invalid memory accesses, leaks (at exit), integer overflow and invalid shifts; ThreadSanitizer finds data races (the threaded tests: level3test.py, waitreadtest.py, paranoiatest.py, killtest.py, churntest.py, fstest.py, discardtest.py, openertest.py, the shell tests). Set ASAN_OPTIONS=log_path=DIR/asan (UBSAN_OPTIONS, TSAN_OPTIONS likewise) so the reports of the daemon, which runs in the background, are kept in files; a test passes with a sanitizer only when DIR stays empty. With a libfuse without TSan instrumentation, races inside libfuse are not seen (see valgrindtest.sh, TOOL=helgrind, for those). Static analysis: gcc -fanalyzer, and clang-tidy (the clang-analyzer-* and bugprone-* checks) on src/*.c. Test hooks of scubed3-crash (environment variables of the daemon): SCUBED3_CRASH_AFTER=N exit (as if the power fails) before write N+1 of a macroblock SCUBED3_TEAR=1 with SCUBED3_CRASH_AFTER: a random subset of the 4 KiB pieces of that write reaches the disk (torn write) SCUBED3_BAD_READ=list reads that include one of the byte offsets in list fail with EIO (a bad sector), until it is written SCUBED3_WRITE_LOG=file the offset of every macroblock write is appended to file (used for the statistical tests) SCUBED3_WRITE_LOG_TIME=1 with SCUBED3_WRITE_LOG: the time of the write (CLOCK_MONOTONIC) is logged too SCUBED3_IV_LOG=file every block write of a partition is appended to file before the write: partition, block, seqno, nonce (used by ivtest.py) Running everything ~~~~~~~~~~~~~~~~~~ mkdir /mnt/s3; cd OUTDIR; sh /path/to/dev/batch.sh runs all tests below, one after the other (this takes a few hours); every test writes NAME.out, the last line of every test is collected in batch.summary (a test that passes ends with "... OK"; the torture runs end with "FINAL OK"). MKFS_XFS=path/to/mkfs.xfs adds XFS to discardtest.py. Torture tests ~~~~~~~~~~~~~ torture.py DAEMON STEPS SEED [crash|tear] The main test: random steps on a small store (40 macroblocks of 4 MiB): writes, fsync, discards (punched holes and zeroed ranges), grow, shrink (also to the minimum of 2 blocks), restarts, and (with crash or tear, needs scubed3-crash) crashes and torn writes at random moments, also during a resize and while removals are pending. An oracle knows what every mesoblock may contain: after a clean restart exactly what was written last, after a crash what was written up to the last fsync or something newer, discarded data never comes back, a grown part reads as zeroes. At the end it checks that the tombstones of the discards die out (not in level 0, see the main README). PARANOIA=0, 2 or "3 KIBPS" runs it in that paranoia level (in level 2 and 3 the free space is overwritten too; in level 3 the daemon also writes while nothing is written, so it can crash at any moment). CRASH_MAX (default 12) is the maximum number of writes before a crash. Ends with "FINAL OK: ..." and statistics of what happened. runall.sh, runall0.sh, runall2.sh, runall3.sh 8 torture runs in level 1 (normal, crash and tear), 5 in level 0, 5 in level 2 and 5 shorter ones in level 3 (with a rate of 400 MiB/s: as fast as the disk can), a summary in runall.out, runall0.out, runall2.out, runall3.out. Filesystems on top ~~~~~~~~~~~~~~~~~~ fstest.py DAEMON CYCLES SEED ext2 through grow/shrink cycles: every cycle shrinks the filesystem (resize2fs) and the partition, restarts the daemon while the shrink is not finished, writes files, grows the partition and the filesystem; after every step e2fsck -fn and the checksums of all files. killtest.py DAEMON ROUNDS SEED ext4: write files, sync, write more, kill -9 the daemon; after the restart the files written before the sync must be intact and e2fsck must find the filesystem clean. DISCARD=1 mounts with -o discard and runs fstrim before the kill. discardtest.py DAEMON SEED Discard (fallocate PUNCH_HOLE and ZERO_RANGE, fstrim): a byte exact model with unaligned punch/zero/write and restarts; completely full partitions; tombstones die out; blocks that come back with a grow with old indices of the partition on them; ext2 (with the ext4 driver), ext3, ext4 with -o discard, and XFS if MKFS_XFS is set (xfs_repair is taken from $XFS_REPAIR, from the xfsprogs build tree of mkfs.xfs, or from $PATH): fstrim frees mesoblocks, nothing changes at a restart, fsck is clean, the write cost with and without fstrim. PARTS=ABCD selects parts of the test. sizetest.py DAEMON Non-default mesoblock and macroblock sizes (-m 12..16, -M 20..24): ext4 with files, fstrim, restart, e2fsck, contents. fsrepair.py ext4 on top (uses scubed3 from the current directory): corrupt a 4 KiB block of a file on disk, reading the file gives EIO, rewriting the file in place repairs it, e2fsck stays clean. Integrity, reads and writes, openers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ corrupttest.py DAEMON ROUNDS SEED Flips bits in mesoblocks on disk: reading the corrupt 4 KiB parts gives EIO (and only those), check-data-integrity reports exactly them, also after garbage collection and restarts; a partial write into a corrupt part fails, writing a whole part repairs it. ioerrtest.py DAEMON-crash The return values of reads and writes: at the end of the partition (short reads and writes, 0 at the end, ENOSPC for writes at the end), also with -o direct_io; read errors of the base device (SCUBED3_BAD_READ): the daemon keeps running, the mesoblock gives EIO, check-data-integrity reports it, rewriting repairs it; an unreadable index of an unallocated block is harmless, of a block of the partition: the partition is not opened (the daemon keeps running). openertest.py DAEMON SCUBED3CTL ROUNDS Several openers of a partition and no EBUSY races: two file descriptors at once, mkfs/resize2fs/e2fsck directly on the partition, close and resize right after umount or close() without retries, close of a partition that is really in use, close-on-release with two openers, scubed3ctl refuses to mount twice. truncatetest.py DAEMON Truncate behaves as on a block device: dd without conv=notrunc, "> partition", truncate(1) and ftruncate() succeed, but neither the size nor the contents change. scubed3ctl and the control socket ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ configtest.sh DAEMON SCUBED3CTL KDF parameters from the configuration file: a partition created with the salt from the file is found with the key PBKDF2-SHA256(passphrase, salt, iterations), -C and /etc/scubed3.conf, -s/-i override the file, errors in the file are fatal, also an invalid paranoia= (uses script(1) to type the passphrase). opentime.py DAEMON [BLOCKS] How long it takes to open a partition on a large base device (BLOCKS macroblocks of 32 KiB, default 29000, three small partitions): with the page cache dropped, every partition is opened twice, and a passphrase without a partition; the first open reads the start of the index of every block, the later ones only decrypt it (scubed3 keeps it in memory); the partitions must contain what was written. rootswitchtest.py DAEMON SEED scubed3 keeps working when its root directory is emptied after its start (root on scubed3: run-init deletes the files of the initramfs): the daemon is started in a chroot with its libraries, /dev/fuse, /dev/urandom and the base file, then everything in the chroot is deleted; partitions are created, written, closed, opened again and read, also in paranoia level 3; afterwards a normal daemon finds the data in the base file (a hard link outside the chroot). initramfstest.sh DAEMON SCUBED3CTL The local-top script of contrib/initramfs-tools in a simulated initramfs (its paths point into a temporary directory, stand-ins for /scripts/functions and askpass, fake-cryptsetup). A key store with two views: A with root (LUKS, the root file system) and home (LUKS), H with hidden (LUKS); the passphrases at boot: a wrong one, H, A. The console shows exactly what cryptroot shows: three times "Please unlock disk base_crypt: ", twice the message of a wrong passphrase, "set up successfully", nothing else; ROOT in param.conf is /dev/mapper/myhost-root (luks_prefix=myhost-) with the file system, the daemon runs as @scubed3 (with oom_score_adj -1000, where that can be set), root, home and hidden are open, the LUKS volumes of root and home, not of hidden, level 2, the copy of the key store is gone. A second boot with CRYPTTAB_NAME: three wrong passphrases, the message of cryptroot (maximum number of tries exceeded), no ROOT, nothing open. KEEP=1 keeps the directory (its run/initramfs/scubed3.log). Uses /dev/scubed3fs and /run/scubed3/control (like a boot), no other daemon may run. installertest.sh DAEMON SCUBED3CTL partman-scubed3 (contrib/debian-installer) in a simulated debian-installer: its scripts run with their paths in a temporary directory, with stand-ins for debconf (the answers come from a list; as with cdebconf every command sets RET), the library of partman (partition lists are files), apt-install, in-target, log-output and fake-cryptsetup; the daemon and scubed3ctl are real, the base device is a loop device with the partition "old" of an earlier installation (its own passphrase and LUKS). Without cryptsetup Configure scubed3 runs anna-install cryptsetup-udeb, then an error; without the device mapper target crypt (dmsetup targets) it runs modprobe dm-crypt and depmod, then an error. Configure scubed3: the existing store (its salt), a key store with the boot passphrase; old is opened first (its passphrase, a wrong and the right LUKS passphrase: its key and its volume key go into the view of the boot passphrase), root (60 MB, LUKS), hidden (20 MB, a hidden passphrase: an empty one and two different ones are refused) and home (20 MB, LUKS) are created (a note about a backup of the key store), the layout is shown (the base device, the partitions with their sizes, views and LUKS, the unclaimed blocks, the slots of the views), level 2, the unclaimed blocks are lent to root (the size of its file does not change); the KDF of a passphrase of the key store is computed once (keystorectl derive, all other runs of keystorectl and scubed3ctl with the key store have -k; keystorectl adds the partitions and saves, scubed3ctl creates them with one view unlocked), a message is shown while the KDF and LUKS run; the views of the key store; Configure scubed3 again: the passphrases are asked again (a wrong one again, derive three times), no view is lost; root gets a passphrase of its own, at finish its LUKS volume one of 200 characters (at the end: root opens with its passphrase without the key store, the one keyslot of its LUKS volume has the passphrase, old keeps its LUKS passphrase), hidden and home random keys; init.d makes /dev/mapper/testhost-root and -home (the hostname is testhost) disks in partman (format, root at /); check.d wants /boot (not on scubed3), / on root, and / on scubed3 for /home on scubed3 (then a note); 80 MB is "installed", home is mounted at /home; finish-install copies the programs (apt-install scubed3 fails), writes /etc/scubed3.conf, /etc/scubed3.keys, /etc/scubed3/initramfs.conf, CRYPTSETUP=y, no /etc/crypttab, runs update-initramfs in the target and gives the lent blocks back (pending removals, no trim, while root is in use; making it smaller is refused while it is in use); then the local-top script of the installed initramfs boots it with the hidden and the boot passphrase: the root file system is there, home and old (LUKS) and hidden are open, writes drain the pending removals. Uses /dev/scubed3fs and /run/scubed3/control. cdebconftest.sh CDEBCONF-PREFIX CDEBCONF-SRC The passphrase questions of partman-scubed3 (s3_ask_passphrase) with the real cdebconf (its text frontend on a pty, answers typed by a script): two equal passphrases are taken, two different ones and an empty one get their error (nothing is generated), a short one the question about a weak passphrase. With cdebconf every command sets RET (db_set: "value set"): this caught two equal passphrases that were reported as different. Needs cdebconf built from its source with the text frontend (see the script; not in batch.sh); for a new partition or LUKS volume both fields may be empty (a random key), one empty field is a mismatch; writes /etc/cdebconf.conf (an existing one is put back). valgrindtest.sh DAEMON SCUBED3CTL The daemon and scubed3ctl under valgrind (memcheck): no memory errors and no leaks (definitely, indirectly or possibly lost). The workload: create, resize, write and read; ext4 on a partition with discard (fstrim); paranoia level 2 and 3; more reserved blocks and back (resize-internal), a shrink, a removal (wipe) in level 1; close, open, a wrong passphrase; the key store (create, passphrases, add, import, unlock-all, open-all, lock, remove, change); then the daemon again on the same base device (replay), a removal with wipe in level 2, check-data-integrity. Slow (the cipher runs under valgrind); the binaries have -g, EXTRA_CFLAGS=-O1 gives better stack traces. SKIPPED without valgrind. With TOOL=helgrind (or TOOL=drd) the daemon runs under that thread error detector (data races, lock order, misuse of pthreads), scubed3ctl without valgrind; in level 3 a writer that waits for its turn, reads of another partition and control commands run at the same time. valgrind.supp suppresses the known races inside libfuse 3 itself, by function name (se->exited at unmount, se->conn.proto_minor and se->bufsize at INIT; checked with libfuse 3.14 built with debug information: nothing else races). That needs the debug information of libfuse (Debian: libfuse3-*-dbgsym, or LD_LIBRARY_PATH to a debug build); without it valgrind-nodebug.supp suppresses every race reported inside libfuse (the test says so). indexfuzz.py DAEMON ITERATIONS [SEED] Structure-aware fuzzing of the indices on the base device, as an adversary with the key of a partition and write access to the base device could write them: the index of a block is decrypted, changed (the number of blocks, reserved blocks, pending removals, trims and the trim floor, seqno, next block, the indices, the bitmap, the hashes, the tombstones in the data, an index copied from another block), its SHA256 is computed again and it is encrypted again. Then the daemon (built with ASan and UBSan, see above) opens the partition, reads all of it, writes, fsyncs, runs check-data-integrity, closes and stops. A finding is a crash, a sanitizer report, an assertion, a hang or a FATAL (one bad block would stop the whole daemon, with all partitions); refusing to open the partition is the correct answer. The findings (base file, log, the mutations) go to indexfuzz-findings/. Uses its own mountpoint /mnt/s3fuzz and control socket /run/scubed3/fuzz (it can run next to the other tests); needs the "cryptography" module. lukstest.sh DAEMON SCUBED3CTL LUKS on top of a partition with scubed3ctl, and the warning below the recommended paranoia level: in level 1 mount and luksformat warn, an answer other than yes cancels; in level 2 luksformat, mke2fs and mount work on the LUKS volume, umount closes it, a file survives umount and mount, its plaintext is not in the partition. Needs cryptsetup and the device mapper (it says SKIPPED without cryptsetup); with LUKSTEST_FAKE=1 the cryptsetup in PATH may be a stand-in without encryption, like fake-cryptsetup (the plaintext check is skipped). keystoretest.sh DAEMON SCUBED3CTL The key store, keystorectl (next to SCUBED3CTL) and scubed3ctl: two passphrases (A and B) with views, root (A), hid (B), shared (A and B); keystorectl changes the key store in memory, save writes it (without save nothing is written: at the prompt a question, with -c an error); scubed3ctl has one view at a time (unlock locks the other), the partitions are created and opened with the keys from the store (no passphrases); every save changes the whole file (no 16 byte block stays the same), its size is always the same; A shows and opens only root and shared; a wrong passphrase has no view; an existing partition is imported; luksformat and luksopen with the key from the store (another key does not open it); a passphrase is changed; a save with only A unlocked loses the view of B, as the warning says; setup (guided) with two passphrases, a new, an imported and a shared partition; two partitions with the same name in two views (keystorectl warns until one view is locked, scubed3ctl opens the one of the view it unlocked); copy (the same partition under another name), unlock-until (passphrases from a pipe, one answer each, the partitions of the other views are opened; at the end of the input the next command is not done), luksopen-all, import of an open partition with LUKS (with its LUKS passphrase its volume key goes into the key store, a wrong one is refused, the volume keeps one keyslot), an empty new passphrase is refused (key store passphrases are never generated); 300 new key stores with one view: its slot is random (at least 50 of the 64 slots, at most 18 times the same), its default name is the slot (iNN); derive gives the key of a passphrase, -k unlocks and adds views with it (not with a passphrase or a wrong key); two keystorectl with one key store: a save after the other saved it is refused, create does not replace an existing key store, scubed3ctl does not change the key store; luksstatus (the LUKS header, the key in the key store, the loop device, /dev/mapper, the mount point, a partition that is not open); a scubed3ctl that waits for the daemon says so; killed with SIGTERM at a passphrase prompt on a terminal (python pty): echo is on again; a temporary file of a gone process is removed by the next write, import leaves no key file; add-passphrase (the partition opens with its passphrase without the key store; empty: a random key; two different ones are refused), luksformat (one keyslot with the passphrase, 300 characters, or a random key; the volume key in the key store opens it); the names of views (asked at the keys> prompt, only digits refused, name renames, add and lock take the name, quit asks when the changes are not saved, with -c the default iNN and numbers). With CRYPTSETUP=$(command -v cryptsetup) it uses the real cryptsetup, the device mapper and dm-crypt (the partitions with LUKS are large enough for a LUKS2 header). Uses fake-cryptsetup (CRYPTSETUP=path for another cryptsetup). entropytest.sh DAEMON SCUBED3CTL The random secrets have at least 256 bits (the passphrases of the key store are not: they are typed, never generated): the LUKS keys in the key store (keystorectl add) that scubed3ctl luksformat gives cryptsetup have 64 bytes (512 bits) and differ, the keys of the partitions 32 bytes (256 bits), both from getrandom(2). fake-cryptsetup A stand-in for cryptsetup for tests without the device mapper (a container), NO ENCRYPTION: luksFormat writes a LUKS signature and the SHA-256 of its keys, open checks the key and maps the payload with a loop device as /dev/mapper/NAME, luksDump (the keyslots, the volume key with --dump-volume-key), luksAddKey, close, isLuks; keys from --key-file (- is stdin) or typed; used by keystoretest.sh and initramfstest.sh. sockettest.sh DAEMON SCUBED3CTL The control socket: private directory (0700) and socket (0600), unsafe directories refused, no takeover of a running daemon, takeover of a stale socket, the peer check (as user nobody), the socket is removed at exit; an unmount while a command runs (check-data-integrity): the command is finished first, the daemon exits cleanly (the control thread was cancelled halfway through the command, with its locks held, and the daemon hung or stopped with FATAL at exit); numbers that are not parsed get one answer (ERR) and change nothing, several commands sent at once are all done; the daemon in the background (it forks) is locked into memory (VmLck). Paranoia levels and the write order ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ivtest.py DAEMON-crash SEED [OLD-DAEMON] The IVs of the data mesoblocks (ESSIV(seqno^nonce, slot, block)) do not repeat when the seqnos repeat: a backup of the base device is restored and used again (after every fsync the indices are read with the key), and interrupted writes (kill -9, torn) are redone (the IV of every block write is logged, SCUBED3_IV_LOG: the interrupted write is not seen in the indices, its data is under the old index); the test fails if one (block, seqno^nonce) belongs to two different writes (it also checks that the seqnos did repeat); with OLD-DAEMON (a scubed3 of version 0.4) a base device written by it is opened, read and written, both versions of blocks are read. wipetest.py DAEMON-crash SEED Removing a partition wipes every block that has an index of it: in level 1 at once (also the blocks it gave away in an earlier shrink); in level 3 in random order, while it cannot be closed or resized and its key cannot be used; kill -9 (also with a torn write) while it is removed or wiped never brings its data back, opening it again resumes the removal or the wipe (in level 1 at once, in level 3 in random order) and afterwards no block has an index of it; a partition whose old index uses blocks of another open partition is not opened (and the daemon keeps running); a partition of 2 blocks (the minimum) is removed in level 3. waitreadtest.py DAEMON SECONDS SEED LEVEL [KIBPS] [MAXREAD] A writer that waits for its turn (paranoia level 2 or 3; level 3 with KIBPS, default 640 KiB/s: 10 writes per second) does not block the reads of its partition: one thread writes partition a as fast as possible, other threads read a and b and measure how long a read takes (median, 99%, max; with MAXREAD in seconds the test fails if a read of a takes longer); every mesoblock contains its index and a version, a read must return a complete version that is not older than the last one written; at the end and after a restart a and b contain exactly what was written. churntest.py DAEMON ROUNDS SEED LEVEL [KIBPS] Partitions are created, enlarged, shrunk to 0 blocks (removed) and closed, over and over, while two other partitions are written as fast as possible, in paranoia level LEVEL (0 to 3; level 3 with KIBPS, default 6400 KiB/s: 100 writes per second); the new partition must contain what was written to it, the level must stay the same, the block counts must add up, the other partitions must contain exactly what was written to them (also after a restart), and in the end no block may have an index of a removed partition (in level 2 and 3 the test waits until it is wiped before it closes it). paranoiatest.py DAEMON-crash SCUBED3CTL SEED Level 2 with three partitions and free space (small blocks: -m 12 -M 16): the written blocks are uniform over the whole base device and consecutive writes independent (chi-square, writing to one partition only), level 1 for comparison; data correct with writes, discards, fsync, restarts and kill -9; three writers and a control prober at the same time (no deadlock); scubed3ctl paranoia 2; closing a partition switches back to level 1; a partition that is not open is overwritten (as documented). resizestat.py DAEMON-crash ROUNDS SEED Level 2 while a partition is grown and shrunk again many times: the blocks added by a grow are hit as often as any other block in the writes right after the grow, the oldest blocks of the partition (most of them are removed) as often as any other block during the shrink, all writes uniform and independent. Only things an observer could know beforehand are used to choose the blocks that are counted, and the first writes of a shrink are skipped (the comment in the script explains why: the time between writes depends on the live data of the next block). Every z must be small (|z| < 4); 1500 rounds take about ten minutes. memoryless.py DAEMON-crash WRITES [FSYNC-PROBABILITY] Level 1, one partition, no resizes: the block that is written does not depend on how long ago it was written (age quartiles uniform, chi2 with 3 degrees of freedom). level0test.py DAEMON-crash SEED Level 0: fewer block writes than level 1 for the same data, the next block is the block with the least live data, data correct after a restart. level3test.py DAEMON-crash SCUBED3CTL SEED Level 3 (10 writes per second, 64 KiB blocks): idle, the writes come at the rate and the times between them are exponentially distributed; while a partition is written as fast as possible, the same rate and distribution and the written blocks uniform over the base device, the throughput of the partition at most its share of the rate; data correct with fsync, a clean restart and kill -9; the paranoia command answers at once while a writer waits for its turn, and switching to level 1 lets the writer finish; scubed3ctl paranoia 3 KIBPS; closing a partition switches back to level 1 and the writes stop. jstat STEPS (jstat.c, built by build.sh) Unit test of juggler.c, without the daemon: every draw is uniform over the pool at that moment, also while blocks are added and removed (chi-square), self-draws and run lengths as expected for i.i.d. draws. drawstat.py LOG... The rate at which a block is drawn as its own next block in daemon logs (the daemon logs every draw), compared with the i.i.d. expectation. Offline tools ~~~~~~~~~~~~~ s3read.py [-s SALT] BASE PASSPHRASE KDF_ITERATIONS [OUT_IMAGE] An independent reader of the on-disk format (SSS3v0.5 and v0.4, written from the description in the main README, it shares no code with scubed3): lists the macroblocks of the partition of the passphrase, ignores a torn most recent block, and with OUT_IMAGE writes the reconstructed partition (corrupt 4 KiB parts as zeroes, they are reported). Some tests use it to find mesoblocks on disk. Default salt "scubed3_prod", PBKDF2-SHA256, 4 MiB macroblocks, 16 KiB mesoblocks. s3diff.py [-s SALT] BEFORE AFTER PASSPHRASE KDF_ITERATIONS The view of an adversary who has ONE passphrase and two snapshots of the base file: which macroblocks changed, and do they all belong to the partition of that passphrase? In paranoia level 1 a changed block that is not ours shows that another partition was written; in level 2 the free space is overwritten too, so it shows nothing.