To start off, the previous item on my schedule was speeding up multiplication routines of FFA by providing their assembly versions. So far I consider this piece of work done. If someone reviews the code and discovers an opportunity for optimization or any other improvement - I'll be more than willing to incorporate the feedback into the vpatch. Otherwise, it'll stay as is, and I hope that one day it'll make it into FFA vtree.
The next item scheduled for implementation was ripping out Linux kernel RNG and replacing it with a driver that could be directly fed with random data provided from userspace (i.e. from FG). To provide a more specific work plan:
- There is a blog post in progress, of the "let's read together" sort, about the current Linux RNG code, and what expectations of other parts of kernel and userspace it should satisfy. This work should be finished by the end of next week (18.08.2019)1.
- I'll do a few experiments with the source code, using a prototype implementation of the driver. The main question is whether the system would be able to boot to userspace without any random numbers available2. This should take 2-3 week to finish (~08.09.2019).
- The clean implementation of the modified RNG driver as outlined above. The time for doing this coincides with my certain plans in meatworld, so I'll have to dedicate a month of work for it (until middle of October).
One of the decisions that I'll have to make along the way is to pick a version for kernel genesis: very quick test shows that vdiff indeed can handle a project of Linux's size, however there are some binary files there that would be lost (some images and pdf documentation). The bigger issue is that vpatch chokes on dotfiles, which are present in the Linux tree as well:
$ cat ../test.vdiff diff -uNr a/.test b/.test --- a/.test false +++ b/.test 237333992aa0d343b4f59b2c2a31cd8c3f34762be261b4bb223e4de887d46e7be33538d9ad052a1fe04f0d2ba954e6e14191d1d31a8b99af0640895de984bd89 @@ -0,0 +1 @@ +test $ vpatch < ../test.vdiff creating .test raised ADA.IO_EXCEPTIONS.USE_ERROR : new name ".test" designates a file that already exists $ ls -a1 ./ ../ .test/ vpatch-1a7d54090ee183e939ad4ba8ec610bbbfd2d46458e3b8800c16a10bdbfa347e8bfc41a036e22987dc7d63f57eb9f8f61eff3f7920b71c0ef2e42adaeca0085a9
That is, in case of dotfile, a directory is created instead for some reason. I'll have to produce a fix for this as well3... Anyhow, so far I plan to genesis a kernel currently used in Cuntoo, unless there are objections to it: unless something like kernel 2.4 is used, all the differences between 2.6.X-3.Y-4.Z kernels are in architecture support; as far as x86 is concerned, different sets of bugs&fixes to drivers would be the whole difference.
I'd like to hear your comments on these things.
- Reflecting on the proper time accounting practice, one of the mistakes that I have done several times already is not giving myself proper amount of time for write-up. Yes, work can take a week; but the write-up can take a week as well. [↩]
- There is an interface for "reading/generating" random values for in-kernel consumption, and its interaction with the boot process is not clear to me yet. Just reading the source to fish out this information seems to involve too many components, so I would like to take an experimental approach here. [↩]
- While creating the reproducer vpatch shown above, I discovered that vpatch does not handle empty vpatches too. Funny, how these things accumulate. (For this, I already have a fix). [↩]
> all the differences between 2.6.X-3.Y-4.Z kernels are in architecture support
Not quite. Some time in 3-4 they introduced "device tree" item (which is unsupported in e.g. my 'M', and will remain unsupported unless someone makes persuasive argument). And along with this, various internal API changes, which I have not had chance to fully dig out.
Device tree seems to be ~hard requirement for ARM64, you cannot drop this feature without that whole architecture. Re API changes - they happened all along 2.6-5.* releases, and I don't think there is a "correct" or "bestest" API that we would like to stabilize on, so yes, arbitrary pick is OK wrt. this issue.
Re: FG, the principal difficulty is that most users attach theirs via USB-to-RS232 boxes, which in turn rely on yet-other kernel driver (which will vary by chipset of the particular adapter, and there are several dozen common ones.)
This I would leave to users -- the plan is to have a userspace proggy feeding the bytes into kernel. So that it would be possible to feed e.g. a file, or digits of Pi, or whatever, as entropy, if required. I will provide a program for reading from /dev/ttyX, if someone wants a more complex setup -- it should be possible, but I won't go that deep into user-specific use-cases.
IMHO the ideal way to feed FG would be to let user designate any tty , which then 1) gets fed into kernel rng 2) becomes unavailable to userland until machine shuts down 3) reported in kernel log in re failures, quality (periodically, as measured by perhaps 'ent').
And not one FG, ideally, but potentially several, currently there is no simple recipe for making practical use of more than one FG in a system.
Point 2 is interesting and I can investigate whether there is an API for that, or how to add it. Point 3: I can add an ioctl for logging events like low RNG quality.
But as I mentioned before, I would strongly prefer to keep the fundamental functionality in userspace: 1. users with specific needs (multiple FGs) could do the setup the way they like it -- in kernel, this would lead to a whole new subsystem. 2. likewise, I did not consider quality measurement before you mentioned it -- with code in userspace, it's much easier add the functionality you need, also I doubt that you want ent in your kernel.
I am somewhat confused by the q. of what is the rationale for handling FG in kernelspace, if it is at the same time feedable (and uncontrolledly consumable) by the entirety of userspace.
To expand on this, near as I can tell, the only logical rationale for FG support in kernelspace could be to ensure that 1) all bits disgorged by /dev/random and supplied to userland processes in fact originate from FG(s), and are not dependent on any userland goings-on 2) no two userland processes ever receive the identical sequence emitted by the FG(s) at particular stretch of time.
Would seem to me that a "any user process can feed in bits" would violate both?
No, I don't plan to change anything in FG-device->userspace path. I plan is to change /dev/random, /dev/urandom, and internal kernel RNG "source" implementations to read bytes from FG-filled buffer.
Unless I misread, it would seem that it won't be a *guaranteed FG-only-filled buffer*, but a "fillable by any userland process" buffer, though? What useful guarantees does this provide to the operator (vs. simply reading FG directly from user application, as e.g. Eulora and Peh presently do) ?
It's up to users to keep their userspace under control, using permissions and other tools. Under targeted attack, kernel integrity would also evaporate rather soon, so I don't see how keeping the whole scheme in kernel improves the situation. Limiting the /dev/*random to a single writer seem to be enough to prevent shooting yourself in the foot.
Also, I don't see how "dependent on any userland goings-on" is a problem in itself. The problem is that those "userland going-ons" (and kernel going-ons as well!) are not under your control, and chosen based on the assumption that no-one has a high-quality HWRNG. I don't see problem in the "userland" at all. As I already mentioned, if you don't control your userspace, having direct fg->kernel feeding does not help as well (~nobody cares about local privilege escalation on Linux).
With a correct implementation of ring buffer in kernel, this should not be tricky at all.
1. User is in control -- maybe you want to add data from two FGs into the kernel buffer? Control RNG quality without keeping ent in the kernel? I see this as a big improvement from the cleanness of the whole approach point of view, even if practical improvement would be not that big. I guess it ties into the "general-purpose OS" discussion in the logs.
2. Kernel seems to use random number as well, so there is an improvement on this front.
3. There are still application that use /dev/*random, I don't think they will be gone anytime soon. Yes, if we had stdrng the same way we have stdout there wouldn't be need for all of this. It's not there, unfortunately.
> Under targeted attack, kernel integrity would also evaporate rather soon, so I don't see how keeping the whole scheme in kernel improves the situation... ...~nobody cares about local privilege escalation on Linux
Hence invites the question of why keeping *any* of it in the kernel, improves situation. Esp. if no attempt is made to keep arbitrary userland proggies from shitting arbitrary bytes into the buffer. This is really the reason why I did not bother to attempt a kernel FG handler in '16, when released device. I saw the entire concept of /dev/random centralized device as unhygienic.
The entire premise of the Unix 'userland-kernelspace' separation is that it is possible to meaningfully isolate (user proggies from kernel, and from one another.)
Granted this *is* a fiction, esp. on x86. But if one were to let go of the fiction entirely, the rationale for keeping such a thing as k-u separation itself, with the attendant expensive (see 'M' series on my www!) TLB, the context switches... falls away. And the logical conclusion is, then, would be... MP's proverbial 'MSDOS on x64'.
> Kernel seems to use random number as well...
I'm not aware of any serious application where crypto happens in kernelspace... where did you see this?
> 3. There are still application that use /dev/*random
This is the one use case I can think of immediately. But still seems to me that it would be ultimately cleaner to patch said programs to eat from user-configurable tty instead of hard-coded dev. (Can you even think immediately of any such program? other than sshd ?)
Nothing too significant: ASLR, stack protector, freelist randomization, certain debug paths and self-tests, you can get an overview here (can do the same for get_random_u32, get_random_u64, get_random_int, get_random_long). They use ~'urandom'-quality data there.
GPG symmetric encryption key generation? Other than that, only various TLSisms/VPNs remain. Patching those application in the cuntoo is an option, however in the light of last discussions about cuntoo's purpose I don't think this should happen.
BTW, the vpatch bug is not the handling of dotfiles, but in the handling of files in the root of press directory, a/test triggers the error same way as a/.test.
#1 : The absence of comment numbering makes this more difficulty than it has to be ;
#2 : I'm with Stan ; anything other than that 1-2 arrangement is ~wasted time, neither particularly useful nor something I'd deem a solution. User doesn't even have to "designate", simply use last (as opposite of first) one.
I'm not so sure "quality" should be measured ; this means sharing the secret bits with the measurer, which meh. Let the user measure his own units in his own way. How's entropy supposed to be automatically measured anyways ?
#3 : In any case, "keeping the functionality in userspace" is sheer nonsense. The whole fucking point of the exercise is to make it part of kernel, specifically and deliberately -- and welded shut and poured concrete over. At issue here isn't "shooting self in the foot" ; at issue here is making empire-style faux osen impossible as a going concern, that's the proximate cause of the activity, making expensive sanity the default.
Ps. #4 : I find it mindblowing that this thing will have to take ~3 months, but whatever.
PPS. Your blog ate my link on "Stan" above, so now we don't even know which comment I was referencing that way.
Thanks for the feedback, I will reconsider my plan then. This may be even easier to implement than what I had in mind originally.
I will fix the comments on my blog in meantime.
@bvt :
> ASLR, stack protector, freelist randomization, certain debug paths and self-tests
If you block on FG (~7kB/s) for these, your machine will run like 386, I suspect.
@Mircea Popescu:
> How's entropy supposed to be automatically measured anyways
Pretty sure we went over this at some point in '16 -- the utility of simple "smoke" tests like "ent" in case like this is to detect catastrophic failures (e.g. "oops, that ain't a FG, but is my Hayes 56K") rather than to make fine distinctions. Whether this belongs in kernel space is separate q, but IMHO any time FG output is being pooled, XOR'd, whichever manipulations, there ought to be a simple test that at least wakes up operator if somehow result is e.g. stream of 0.
> The whole fucking point of the exercise is to make it part of kernel
IMHO the ultimate q is, what part of FG eating, if any, actually wins from happening in kernel.
Re: tests -- at the very least you want a machine-wide alarm that tells you if the thing went unplugged, or somehow fell substantially below spec output rate, etc.
Also @Mircea Popescu --
> this means sharing the secret bits with the measurer, which meh
I thought this was obv., but why would you ever test *and give out to eating process* a particular string of bytes? You test periodic selections, like in most industrial process, these then get discarded.
You already have a led for catastrophic failures neh ?
And if you want to ~waste~ some bits to spuriously test, all your other arguments suddenly suffer. There's a blocking and a non-blocking kernel pool. Let them stand as such ; and yes if this means proc is running as 386 let it run as 386 ; and no don't make it spuriously shittier by "testing" unasked.
@Mircea Popescu :
> You already have a led for catastrophic failures neh ?
With naked eye, can see the one on desk, but not e.g. the ones at piz...
> There's a blocking and a non-blocking kernel pool
I actually liked your earlier idea , where entirely abolish '/dev/urandom' nonblocking pool.
> no don't make it spuriously shittier by "testing" unasked
Seems like cogent argument for leaving the matter entirely out of the kernel. Let user test when he wants, how he wants, decide how many bits he can afford to burn away.
I can see an argument for 'kernel cuts FG bitstream among N users' but that's about it.
> entirely abolish '/dev/urandom' nonblocking pool.
Yes, but that was for TMSR-os, not for confiscated-Linus item.
Review what that blond fuck did when he ruined the kernel for you, and do the same thing to him. FG in kernel so all the jwzs use it by default. And in general -- republic by default is the crowd control strategy.
@Stanislav, #18,
I'm not so much concerned about this: if it's too slow, the conclusion would be "Linux does not work nicely with the HWRNG"; and anyway I don't think Linux will use more than 2kB/s or RNG data for it's internal use.
#19
This I can see.
@Mircea Popescu,
Point taken.
Meanwhile, comments here should be fixed.
[...] his vpatch and article to Cement Keccak Hashing into Kernel RNG, continuing his prior work of ripping out Linux kernel RNG and replacing it with a driver that could be directly fed with random data provided from userspace [...]