bvt's backtrace

Gales Linux is an distribution started around 2017 by JWRD Consulting (jfw and dorion) for x86_64 architecture. There is and overview of guiding principles and software selection in the introductory article on jfw's blog. I gave a try installing Gales and looking at its structure.

Bootstrap & Installation

Typically, distribution provides a ready-to-use installer, which either installs binary packages, or automates the building of those during the installation procedure. With Gales, the user is responsible for building the binary distribution components from source (bootstrapping), and later installing them manually (copying or unarchiving) into the target partition.

Bootstrap procedure happened on the Cuntoo machine. The process went through exactly as shown in build document, modulo two-three extra patches:

• First, GCC build process failed due to newer version of makeinfo (6.3) used to prepare the documentation rejected syntax of one of the .info files. The upstream fix is available, I have introduced the corresponding changes manually.
• Second, if bootstrapping on GCC 5¹, there is another error, with upstream fix here. Not sure whether you want this fix in the tree, but at the very least it should not harm.

Because the Cuntoo and Gales machines have similar hardware, I have reused the Cuntoo kernel config, which worked without any problems.

When the ./mkinstaller stage was reached, it turned out that the installer is a initramfs image, which would not be a problem in itself, if the installation target wasn't accessible to me only remotely, through a SystemResqueCD LiveUSB. So I looked at the list of files necessary for the installation (./installer/initramfs.list[.sh]), and scp'ed them manually.

The output of the bootstrap process are a kernel image, an archive with base root system (base-$date), and optional archives with compilers (c-$date) and ports (gports-$date).

Install happened on the machine similar to the Cuntoo one, but a bit newer, with only M.2 storage on board. This turned out to be a source of problems, because lilo as present there won't install on the NVMe devices, suggesting to use grub2 instead². Luckily, Slackware did contain a patch for it, which I reproduce below, in the form that applied and built on top of Lilo version 24.0 of Gales (Slackware uses 24.2):

diff -Naru lilo-24.2/src/common.h lilo-24.2.new/src/common.h
--- lilo-24.2/src/common.h	2015-11-21 23:50:23.000000000 +0000
+++ lilo-24.2.new/src/common.h	2018-02-15 15:13:17.411968439 +0000
@@ -386,7 +386,7 @@
 extern FILE *errstd;
 extern FILE *pp_fd;
 extern char *identify;	/* in identify.c */
-extern int dm_major_list[16];
+extern int dm_major_list[32];
 extern int dm_major_nr;

 #define crc(a,b) (~crc32((a),(b),CRC_POLY1))
diff -Naru lilo-24.2/src/geometry.c lilo-24.2.new/src/geometry.c
--- lilo-24.2/src/geometry.c	2015-11-21 23:50:18.000000000 +0000
+++ lilo-24.2.new/src/geometry.c	2018-02-15 16:10:25.844149725 +0000p
@@ -84,8 +84,9 @@
 int dm_version_nr = 0;
 #endif

-int dm_major_list[16];
+int dm_major_list[32]; /* increased from 16 to allow for nvme disks  */
 int dm_major_nr;
+int nvme_pr = 0; /* set to none zero after geo_init if nvme disk present */

 #ifdef LCF_LVM
 struct lv_bmap {
@@ -200,6 +201,9 @@

     while(fgets(line, (sizeof line)-1, file)) {
 	if (sscanf(line, "%d %31s\n", &major, major_name) != 2) continue;
+	if (strcmp(major_name, "nvme") !=0) {  /* set if nvme drive is present */
+		nvme_pr=-1;
+	}
 	if (strcmp(major_name, "device-mapper") != 0) continue;
 	dm_major_list[dm_major_nr] = major;
 	if (verbose >= 3) {
@@ -708,6 +712,22 @@
 	    geo->start = hdprm.start;
 	    break;
 	case MAJOR_SATA1:
+	    /* check for nvme device and assume boot/this device is nvme if present */
+	    if (nvme_pr != 0) {
+	    geo->device = 0x80 + last_dev(MAJOR_HD,64) + (MINOR(device) >> 4);
+	    if (!get_all) break;
+	    if (ioctl(fd,HDIO_GETGEO,&hdprm) < 0)
+		die("geo_query_dev HDIO_GETGEO (dev 0x%04x): %s",device,
+		  strerror(errno));
+	    if (all && !hdprm.sectors)
+		die("HDIO_REQ not supported for your NVME controller. Please "
+		  "use a DISK section");
+	    geo->heads = hdprm.heads;
+	    geo->cylinders = hdprm.cylinders;
+	    geo->sectors = hdprm.sectors;
+	    geo->start = hdprm.start;
+	    break;
+    	}
 	case MAJOR_SATA2:
 		printf("WARNING: SATA partition in the high region (>15):\n");
 		printf("LILO needs the kernel in one of the first 15 SATA partitions. If \n");

Note that NVMe support here depends on the procfs being mounted. With this patch applied, lilo would happily proceeded to detect kernel and install the bootloader. In the process of building Lilo I first-handed witnessed that that the build process is indeed reproducible³-nice job!

Other than that, everything worked out according to the INSTALL instructions.

After installation and initial configuration (still inside chroot), I proceeded to install sshd - while the system's original use-case is airgapping, it was out of question in this case. The installation process was straightforward, but one wart is that the built packages do need to be signed - and I would not mind doing that - but apparently this feature is not finished yet. gpkg-install complained loudly about lack of signatures, which I had to disable with -f flag, as outlined in the PORTS document. Also, for sshd to work, /dev/pts must be mounted⁴.

To the build process, I have some comments: the authors enable by default a GCC flag to warn about excessive stack usage in the applications, with the goal of eliminating it somehow (switching to malloc where other approaches fail). While I did not detect anything objectable in the patches I had a look at, these patches must be double- and triple-checked, because malloc is a mechanism with non-obvious failure modes - it introduces locking and may interact in non-obvious ways with library constructors. Also, the O0-O1-O2 optimization level selection in both bootstrap and gports looks a bit arbitrary: it would be good to have a look at what -f[optimization] flags are enabled at each level, and hand-pick those that cannot cause any harm⁵.

Bootup and Usage

The boot time of the system is really impressive - I was greeted with a login prompt a few seconds earlier than the kernel finished initializing its subsystems.

Gales is a statically statically linked distribution, but its storage footprint is very low - 8 Mb for base filesystem, 76 Mb for tree with compilers. Syncing the gports immediately downloads ~560 Mb of sources, though.

Contrasting it with Cuntoo, the whole system is certainly much more intellectually approachable and simpler. The feel of the system is very different - daemontools instead of SYSV init, minimalistic toolset. Different file system hierarchy: after installation, you would immediately see a few more directories in the file system root⁶:

• /commands - binaries a moved here according to not-really clear criterion (daemontools components only so far).
• /gales - package and distribution service scripts. In more details:
• /gales/command - symlinks to package-installed binaries (i.e. foo -> /gales/pkg/$pkg-$major.$minor-$heathen_version/bin/foo). Both packages from base archive and ports end up here, so with a stretch it can be said that Gales follows Linux way, not BSD way here. A stretch, because the package manager is so simple, that it's hard to say that it "manages" anything.
• /gales/pkg - contains packages installed in the hierarchy of one directory per package.
• /packages - sources of daemontools only so far? If this directory is for source-built components, why not have init(1) source there?
• /services - services to be launched by daemontools
• /usr - a symlink to /

A feature that I liked a lot is that shell is the only scripting language in the default install of the distribution. Typically perl and python get pulled in unconditionally as a build dependency of a runtime dependency of some rarely-used default-installed utility, or are directly used to implement package manager, etc. With Gales, a decision about what scripting language to use can be made without constraints created by ready availability of python or others.

The lack of man pager in the default installation⁷ is a strange decision. While mandoc is certainly nicer than mandb, I would definitely see fixing it as one of priorities - using 'man -l full-path-to-man' is just WTF - in this case could just convert all docs to text, and use 'more' to read them.

The decision to install configuration files in /etc/examples is good one, the problems it can create should only manifest on the distributions with extensive software selection and regular unconditional functionality-breaking upgrades.

In general, I can well see where this system is targeted, and why it must have worked out nicely so far: with minimal amount of software installed, whole system can stay comprehensible, suitable for security-sensitive computing. OTOH, I wonder if things like Apache or imagemagick get installed, how will the package management system work out, and how comprehensible will system stay? The ports documents already point out the installation order for some of the libraries and applications, and with larger software, this may become inconvenient.

1. I run a half of bootstrapping procedure on GCC 5 Debian 9 machine, at the point of discovering this issue I applied the fix, but restarted everything on the Cuntoo machine. [↩]
2. Even given that the BIOS does support "legacy" booting off M.2 drives! [↩]
3. There was some confusion about which version of lilo got moved where, which got resolved by purging all lilo's with fire and rebuilding it, giving as a result a file that matched one of the earlier hashes. [↩]
4. Or /dev nodes manually created? There is both /dev/ptmx and /dev/pts/ptmx, which hints that this mountpoint may not be necessary. [↩]
5. This will likely require a very deep dive into GCC, so may be a not very practical option, given GCC's size. [↩]
6. I think writing something similar to hier(7) would be beneficial [↩]
7. There is mandoc in gports tree, but as the README helpfully informs, it is not working nicely with Gales directory hierarchy. [↩]