| |
Subscribe / Log in / New account

LWN.net Weekly Edition for August 7, 2025

Welcome to the LWN.net Weekly Edition for August 7, 2025

This edition contains the following feature content:

This week's edition also includes these inner pages:

  • Brief items: Brief news items from throughout the community.
  • Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Don't fear the TPM

By Joe Brockmeier
August 6, 2025
DebConf

There is a great deal of misunderstanding, and some misinformation, about the Trusted Platform Module (TPM); to combat this, Debian developer Jonathan McDowell would like to clear the air and help users understand what it is good for, as well as what it's not. At DebConf25 in Brest, France, he delivered a talk about TPMs that explained what they are, why people might be interested in using them, and how users might do so on a Debian system.

[Jonathan McDowell]

McDowell started with a disclaimer; he was giving the talk in his personal capacity, not on behalf of his employer. He wanted to talk about "something that is useful to Debian and folks within Debian", rather than the use of TPMs in a corporate environment.

McDowell has been a Debian developer for quite some time—more than 24 years, in fact. Professionally, he has done a lot of work with infrastructure; he has written networking software, high-end storage systems, and software-defined networking. He has also run an ISP. To him, TPMs are simply "another piece of infrastructure and how we secure things".

Unfortunately, there is a lot of FUD around TPMs, he said, especially now that Microsoft is pushing TPM devices as part of the baseline requirement for Windows 11. That has been part of the baseline since it was introduced, of course, but with the end-of-life approaching for Windows 10 people are starting to take more notice.

Many people are responding to TPMs by "throwing up their hands and going, 'this is terrible'"; but they are actually really useful devices. One of the reasons that they are useful is that they are so common. If you buy a new PC, "it is incredibly likely that you have some TPM capability on it". Unless it's an Apple system—they have Secure Enclave instead, "which does a whole bunch of different things that have some overlap".

What is a TPM?

So, he asked rhetorically, "what is a TPM?" He displayed a slide with Wikipedia's definition of a TPM, which says that a TPM is "is a secure cryptoprocessor that implements the ISO/IEC 11889 standard". McDowell said he did not recognize that definition, despite having worked with TPMs for several years. He repeated the definition and said, "that doesn't mean much to me, and it's also not entirely true", because there are multiple TPM implementations that are not secure cryptoprocessors.

There are three variants of TPM that McDowell said he was familiar with: discrete, integral, and firmware. A discrete TPM is a separate chip that lives on the motherboard. Historically, the discrete TPM has been connected over the low pin count (LPC) bus, but modern systems mostly use the serial peripheral interface (SPI) bus. Then there is the integral TPM, which sits on the same die as the CPU, but as a separate processor. Examples of integral TPM include Intel's Management Engine and AMD's Secure Technology (formerly called "Platform Security Processor"). These are logically separate from the CPU that applications run on, which gives some extra security, "but not a full discrete chip".

Finally, there are firmware TPMs, such as the Arm TrustZone technology. In that case, McDowell said, the TPM is actually running on the application processor in a more secure context, but firmware TPMs can be vulnerable to speculative side-channel attacks. The idea is that the TPM is a small, specialized device that "concentrates on cryptographic operations and is in some way more secure than doing it on your main processor".

McDowell digressed a bit to talk about TPM 1.2 devices. "I hate TPM 1.2 devices. I still have to deal with a bunch of them in life. They are ancient." TPM 2.0, which is the baseline that Windows 11 expects, launched in 2014. He would like TPM 1.2 devices to all go away and said that he would not be discussing them further.

Not for DRM

One of the things that TPMs can do is state attestation. The idea is that the TPM can attest to the software that is running on the machine:

And if all of the stars align and you get everything right, you can actually build a full chain from the first piece of code, the firmware that the CPU runs, all the way up to the application layer and say, I am running this stack of software and I will provide you a signed cryptographic proof.

However, he assured the audience, TPMs are not a realistic way of doing digital rights management (DRM), McDowell said—no matter how much Microsoft or Netflix might want to use them in that way. "They could not build a database of all these values for all the legitimate machines in the world." Trying to do so would result in "support calls coming out of their ears". It is absolutely possible to constrain things so that the TPM can provide a level of security for embedded systems and appliances, he said. "In particular, you can potentially use it for some level of knowing that someone hasn't tampered with your firmware image". But full DRM on general-purpose PCs is not going to happen.

A standard TPM for a PC has 24 platform-configuration registers (PCRs), McDowell said. PCR 0 through PCR 7 belong to the firmware and are used by UEFI to measure the bootloader and "base bits" of what the operating system runs. PCR 8 through PCR 15 are "under the control of the bootloader and the OS", and PCR 16 through PCR 23 are "something different, and we'll not talk about those at all".

PCRs are SHA hash registers; at boot time the TPM sets the values for the registers to zero. Then the hash values for various objects are measured into the registers; For example when GRUB boots, it logs its activity into the TPM event log, and performs a cryptographic hash operation to extend the value of the PCR, explained in more detail in this coverage of a talk by Matthew Garrett. McDowell displayed a slide that showed a command to read the TPM's event log:

    # tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements

Each entry in the log shows something that has been measured into the registers. Details about Secure Boot, for example, are put into PCR 7, which provides an attestation that "this machine has used Secure Boot, and these are the keys it has used to do Secure Boot". All of that is machine-state attestation, he said, "which is the thing that people get worried about" being used to enforce DRM.

Key storage

The much more interesting thing from a Debian point of view, he said, is key storage. While TPMs are small and incredibly slow devices, it is possible to securely generate asymmetric keys, such as SSH keys or signing keys, on the device that cannot be exfiltrated:

You can say "make me a key", and it will make you a key, and that private part of the key can only be exported from the device in a way that only the device itself can read.

Obviously an attacker could use the TPM while they are connected to the machine. But if the user kicks them out or fixes whatever has happened, the attacker would not be able to export any keys stored in the TPM to another machine. That, McDowell said, is incredibly useful. He reiterated that TPMs are slow; they are not full-blown high-performance hardware security module devices. But they are almost everywhere, "and that's why they're interesting, right?" They are a standard piece of hardware that most PCs will have if they're not too old.

If one wants to get more into the corporate side of things, he said, some hardware vendors will provide a certificate that ties the TPM's unique endorsement key to the serial number of the laptop. "So I can do a very strong statement of 'this is the machine I think it should be'." But that, he reiterated, involves a slightly complicated procedure. For single-machine use cases, "you don't have to worry about this bit too much".

This also allows the TPM to do attestation for the key. That is more complicated, McDowell said, but "you can do an attestation where the TPM goes, 'that key was definitely generated in me'". That might be desirable, for example, in terms of a certificate authority or when signing packages. If a key is hardware backed, its use demonstrates that a user has access to a specific piece of hardware—such as a company-issued laptop.

He elaborated later that using it for attestation involved an "annoyingly interactive challenge and response dance"; it was not possible to have the TPM simply generate an attestation statement that can be validated and trusted. However, if one does the full attestation dance, "I can guarantee [the key is] hardware-backed and I can guarantee it's hardware-backed by a particular vendor of TPM".

Another neat thing that users can do is to bind a key that can only be used if the PCRs are in a particular state. That means it's possible to ensure that someone hasn't messed with the firmware, to guard against an "evil maid" attack. If the machine is still running the image that the user expected to be running, then they could use their key. "If someone has subverted that with a dodgy firmware or a dodgy kernel, then I will not be able to use my key".

TPMs can also generate random numbers, he said, though that is not necessarily particularly interesting. The TPM needs random numbers for many of its operations, and it exposes that interface "so you can ask the TPM for random numbers". There are faster sources of random numbers, such as the CPU's instruction set and USB-attached random-number generators, but TPMs are still useful largely because they are present in a lot of machines.

Crypto types

McDowell had said he was not going to talk about TPM 1.2 devices, but he mentioned them again to say they did not do cryptography right. The 1.2 specification only allowed for the use of 1024-bit RSA keys, and the SHA1 algorithm. The 2.0 specification added "this thing called crypto agility", and extended the baseline support to 2048-bit RSA keys, SHA256, and NIST P-256 elliptic-curve cryptography.

Post-quantum cryptography is not there yet but it is being actively worked on upstream. Because of the crypto-agility standard, none of the interfaces used to talk to the TPM will change much—it will just be a different key type. All of the TPM vendors are ready, McDowell said; it is just a matter of waiting for the details to settle. "This will come before we need it, which is good".

Using the TPM

Next, he demonstrated how to check to see if the random-number generator was enabled, but did not go into detail on how to use the feature. McDowell cautioned that AMD's integral TPM has some problems with the random-number generator, possibly to do with locking and conflicts over accessing the device over the SPI bus.

The TPM can also be used to produce trust paths in software using the kernel's integrity-measurement architecture (IMA). For example, if a developer was building an appliance, it would be possible to use the kernel's IMA to create a list of the privileged code, with its hashes. To test this out without messing with the system TPM he recommended the swtpm package, which provides a TPM emulator.

What's more interesting about swtpm, McDowell said, is that it can be used in conjunction with QEMU to provide a TPM to a virtual machine. "I suspect a bunch of people are doing this to boot Windows 11" in virtual machines. It is a fully-featured TPM 2.0 implementation, and it was what he had used for the examples in his presentation. He also recommended the tpm2-tools package, which he called a kind of Swiss Army knife for working with TPMs. He put up a slide showing the tpm2_pcrread command being used to read PCRs 0-7 from the TPM:

    $ tpm2_pcrread sha256:0,1,2,3,4,5,6,7

The version of GNU Privacy Guard (GnuPG) in Debian 13 (trixie) includes a feature that allows users to generate a key and store it in the TPM. "That means you've got a hardware-backed key, no need for the Yubikey plugged into your machine". Even if an attacker has access to the machine they cannot copy the key from it. "That, to me, is amazing." That feature is not available in the GnuPG version in Debian 12 ("bookworm").

I asked how users could back up their key if the machine with the TPM died and was unusable. He said there were two options: generate the key in the CPU and store it in the TPM, with an offline backup on a USB key, or use a GPG subkey. "Then you have the ability to put another subkey on your laptop because the primary key is not the one stored in the TPM." His approach was to use an offline primary key, stored in a hardware token, and then to use subkeys extensively for different machines.

McDowell also showed examples of using the TPM to store a PKCS#11 token for use with SSH, which he said was "a bit annoying" because the process was convoluted. There was another method, using an SSH agent for TPM written in Go, which he described as "cheating" because it was not yet packaged for Debian. He lamented the fact that he was speaking at the same time as the Go team BoF, so he was unable to get help figuring out Debian's Go ecosystem.

Every now and again he thinks about "jumping through all those hoops" to be able to sign his own operating system images to use with Secure Boot. If he did that he could use the OpenSSL TPM 2.0 provider as a certificate authority with a secure backend stored in the TPM. But, he reminded the audience, TPMs are slow. "If you can get 10 signing operations a second out of your TPM, you're doing exceptionally well." It would never be possible to back a TLS web server with a TPM. It was much better for one-offs, such as certificate-authority operations, where a system is not being used to issue a lot of certificates.

A really interesting use of the TPM, McDowell said, was to automatically unlock a LUKS-encrypted drive. A user could set things up to automatically unlock the drive if the firmware, bootloader, and so forth are unchanged and avoid having to enter a passphrase just to decrypt the disk. He noted that users would still need to have a recovery password for LUKS, because if anything were to change about the machine—including rebuilding the initrd—then a user would have to have a passphrase to decrypt the disk. He showed a slide with an example using systemd-cryptsetup and dracut to enable this feature and said, "this is my first time playing with dracut; I didn't like it." He also noted he could not fit the entire example on the slide, but he included a link to a blog post about using TPM for disk decryption.

An audience member asked how much of a pain it would be to "magically incorporate" the proper values when the kernel is updated so that the next time the system is booted it expects the new kernel. McDowell said that systemd does have tooling that will "attempt to do the calculations for what the PCR values will end up as"; he had not looked at that tooling extensively, however. There was still more pain than there should be in automating this, which is "one of the reasons that the systemd folks are pushing unified-kernel images" (UKIs). That would allow distributions to provide the initrd as part of the whole package and provide the PCR value along with it. In the current model, where everyone builds their own initrd, "we have no way of distributing those values as a project".

In general, he said, the systemd folks have been really good about trying to drive the use of TPMs forward. LWN covered some of this work in December. McDowell also gave a call out to James Bottomley for doing a lot of work on the kernel side of things "in terms of just generally improving the infrastructure" around TPMs.

One audience member wanted to know if he had seen any work that would allow programs like Firefox to have passkeys in the TPM. He was not aware of any implementations of passkeys in the TPM; the problem with the passkey approach and TPMs, he said, is that a passkey "normally wants some proof of user presence", such as a button press on a Yubikey. There is no equivalent of user presence with a TPM that couldn't be faked programmatically.

The slides for McDowell's talk are online now, and videos from DebConf25 should be published soon.

[Thanks to the Linux Foundation, LWN's travel sponsor, for funding my travel to Brest for DebConf25.]

Comments (65 posted)

Python performance myths and fairy tales

By Jake Edge
August 5, 2025
EuroPython

Antonio Cuni, who is a longtime Python performance engineer and PyPy developer, gave a presentation at EuroPython 2025 about "Myths and fairy tales around Python performance" on the first day of the conference in Prague. As might be guessed from the title, he thinks that much of the conventional wisdom about Python performance is misleading at best. With lots of examples, he showed where the real problems that he sees lie. He has come to the conclusion that memory management will ultimately limit what can be done about Python performance, but he has an early-stage project called SPy that might be a way toward a super-fast Python.

He started by asking the audience to raise their hands if they thought "Python is slow or not fast enough"; lots of hands went up, which was rather different than when he gave the presentation at PyCon Italy, where almost no one raised their hand. "Very different audience", he said with a smile. He has been working on Python performance for many years, has talked with many Python developers, and heard some persistent myths, which he would like to try to dispel.

Myths

The first is that "Python is not slow"; based on the raised hands, though, he thought that most attendees already knew that was a myth. These days, he hears developers say that Python speed doesn't really matter, because it is a glue language; "nowadays only the GPU matters", so Python is fast enough. Python is fast enough for some tasks, he said, which is why there are so many people using it and attending conferences like EuroPython.

There is a set of programs where Python is fast enough, but that set does not hold all of the Python programs in use—it is only a subset. The programs that need more Python performance are what is driving all of the different efforts to optimize the interpreter, but are also causing developers to constantly work to improve the performance of their programs, often by using Cython, Numba, and the like.

[Antonio Cuni]

In his slides, he represented the two sets as circles, with "programs where Python is fast enough" fully inside "Python programs"; he then added the set of "all possible programs" fully encompassing the other two. In his ideal world, all possible programs would be able to be written with Python; currently, programs that need all of the performance of the processor cannot use Python. He would like to see the inner circles grow so that Python can be used in more programs.

The corollary of the "it's just a glue language" statement is that you "just need to rewrite the hot parts in C/C++", though that is a little out of date; "nowadays they say that we should rewrite it in Rust". That is "not completely false", it is a good technique to speed up your code, but soon enough it will "hit a wall". The Pareto principle—described with a slide created by ChatGPT for unclear reasons—says that 80% of the time will be spent in 20% of the code. So optimizing that 20% will help.

But the program will then run into Amdahl's law, which says that the improvement for optimizing one part of the code is limited by the time spent in the now-optimized code; "what was the hot part now is very very fast and then you need to optimize everything else". He showed a diagram where some inner() function was taking 80% of the time; if that gets reduced to, say, 10% of what it was, the rest of the program now dominates the run time.

Another "myth" is that Python is slow because it is interpreted; again, there is some truth to that, but interpretation is only a small part of what makes Python slow. He gave the example of a simple Python expression: 

    p.x * 2
A compiler for C/C++/Rust could turn that kind of expression into three operations: load the value of x, multiply it by two, and then store the result. In Python, however, there is a long list of operations that have to be performed, starting with finding the type of p, calling its __getattribute__() method, through unboxing p.x and 2, to finally boxing the result, which requires memory allocation. None of that is dependent on whether Python is interpreted or not, those steps are required based on the language semantics.

Static types

Now people are using static types in Python, so he hears people say that compilers for the language can now skip past all of those steps and simply do the operation directly. He put up an example: 

    def add(x: int, y: int) -> int:
        return x + y

    print(add(2, 3))
But static typing is not enforced at run time, so there are various ways to call add() with non-integers, for example: 
    print(add('hello ', 'world')) # type: ignore
That is perfectly valid code and the type-checker is happy because of the comment, but string addition is not the same as for integers. The static types "are completely useless from the point of view of optimization and performance". Beyond that, the following is legal Python too: 
    class MyClass:
        def __add__(self, other):
            ...

    def foo(x: MyClass, y: MyClass) -> MyClass:
        return x + y

    del MyClass.__add__
"Static compilation of Python is problematic because everything can change", he said.

So, maybe, "a JIT compiler can solve all of your problems"; they can go a long way toward making Python, or any dynamic language, faster, Cuni said. But that leads to "a more subtle problem". He put up a slide with a trilemma triangle: a dynamic language, speed, or a simple implementation. You can have two of those, but not all three.

Python has historically favored a dynamic, simply implemented language, but it is moving toward a dynamic, fast language with projects like the CPython JIT compiler. That loses the simple implementation, but he does not have to care "because there are people in the front row doing it for me", he said with a grin.

In practice, though, it becomes hard to predict performance with a JIT. Based on his experience with PyPy, and as a consultant improving Python performance for customers, it is necessary to think about what the JIT will do in order to get the best performance. That is a complex and error-prone process; he found situations where he was "unable to trigger optimizations in PyPy's compiler because the code was too complicated".

All of this leads to what he calls "optimization chasing". It starts with a slow program that gets its fast path optimized, which results in a faster program and everyone is happy. Then they start to rely on that extra speed, which can suddenly disappear with a seemingly unrelated change somewhere in the program. His favorite example is a program that was running on PyPy (using Python 2) and suddenly got 10x slower; it turned out that a Unicode key was being used in a dictionary of strings that led the JIT to de-optimize the code so that everything got much slower.

Dynamic

He put up some code that did not really do anything exciting or useful, he said, but did demonstrate some of the problems that Python compilers encounter: 

    import numpy as np

    N = 10

    def calc(v: np.ndarray[float], k: float) -> float:
        return (v * k).sum() + N
The compiler really can assume nothing from that code. Seemingly, it imports NumPy in the usual way, the calc() function multiplies each element of the v array by k, adds them all up with sum() and then adds the constant N to that. First off, the import may not bring in NumPy at all; there could be some import hook somewhere that does something completely unexpected. N cannot be assumed to be ten, because that could be changed elsewhere in the code; as with the earlier add() function, the type declarations on calc() are not ironclad either.

But, in almost all cases, that code would do exactly what it looks like it does. Developers rarely do these kinds of things that the language would allow, but the gap between the way programmers normally write Python and the definition of the language is what "makes life complicated for the interpreter". In practice, a lot of what Python allows does not actually happen.

It is the extremely dynamic nature of the language that makes it slow, "but at the same time it's what makes Python very nice". The dynamic features are not needed 99% of the time, Cuni said, but "in that 1% are what you need to make Python awesome". Libraries often use patterns that rely on the dynamic nature of the language in order to make APIs "that end users can use nicely" so those features cannot simply be removed.

Game

The "compiler game" was up next; he progressively showed some code snippets to point out how little a compiler can actually "know" about the code. This code might seem like it should give an error of some sort: 

    class Point:
        def __init__(self, x, y):
            self.x = x
            self.y = y

    def foo(p: Point):
        assert isinstance(p, Point)
        print(p.name) # ???
Inside foo(), the compiler knows that p is a Point, which has no name attribute. But, of course, Python is a dynamic language: 
    def bar():
        p = Point(1, 2)
        p.name = 'P0'
        foo(p)
Meanwhile, here is an example where the compiler cannot even assume that the method exists: 
    import random

    class Evil:
        if random.random() > 0.5:
            def hello(self):
                print('hello world')

    Evil().hello() # 🤷🏻‍♂️
Legal Python, but "this is not something to define in production, I hope", he said with a laugh. "Half of the time it still works, half of the time it raises an exception. Good luck compiling it."

In another example, he showed a function: 

    def foo():
        p = Person('Alice', 16)
        print(p.name, p.age)
        assert isinstance(p, Person) # <<<
The Person class was not shown (yet), but there was an empty class (just "pass") called Student. In this case, the assert will fail, because of the definition of Person: 
    class Person:
        def __new__(cls, name, age):
            if age < 18:
                p = object.__new__(Student)
            else:
                p = object.__new__(Person)
            p.name = name
            p.age = age
            return p
"You can have a class with a dunder-new [i.e. __new__()], which returns something which is unrelated and is not an instance of the class. Good luck optimizing that."

The final entrant in the game was the following: 

    N = 10

    @magic
    def foo():
       return N
He "de-sugared" the @magic decorator and added some assertions: 
    def foo():
       return N

    bar = magic(foo)

    assert foo.__code__ == bar.__code__
    assert bar.__module__ == '__main__'
    assert bar.__closure__ is None

    assert foo() == 10
    assert bar() == 20 # 🤯😱
The code object for foo() and bar() are the same, but they give different results. As might be guessed, the value of N has been changed by magic(); the code is as follows: 
    def rebind_globals(func, newglobals):
        newfunc = types.FunctionType(
            func.__code__,
            newglobals,
            func.__name__,
            func.__defaults__,
            func.__closure__)
        newfunc.__module__ = func.__module__
        return newfunc

    def magic(fn):
        return rebind_globals(fn, {'N': 20})
That returns a version of the function (foo() was passed) that has a different view of the values of the global variables. That may seem like a far-fetched example, but he wrote code much like that for the pdb++ Python debugger many years ago. "I claim I had good reason to do that", he said with a chuckle.

Abstraction

So there are parts of the language that need to be accounted for, as he showed in the game, but there is a more fundamental problem: "in Python, abstractions are not free". When code is written, developers want performance, but they also want the code to be understandable and maintainable. That comes at a cost. He started with a simple function: 

    def algo(points: list[tuple[float, float]]):
        res = 0
        for x, y in points:
            res += x**2 * y + 10
        return
It takes a list of points, each represented as a tuple of floating-point numbers, and performs a calculation using them. Then he factored out the calculation into its own function: 
    def fn(x, y):
        return x**2 * y + 10
That is already slower than the original, because there is overhead for calling a function: the function has to be looked up, a frame object has to be created, and so on. A JIT compiler can help, but it will still have more overhead. He took things one step further by switching to a Point data class: 
    @dataclass
    class Point:
        x: float
        y: float

    def fn(p):
        return p.x**2 * p.y + 10

    def algo(items: list[Point]):
        res = 0
        for p in items:
            res += fn(p)
        return
That, of course, slows it down even further. This is a contrived example, Cuni said, but the idea is that every abstraction has a cost, "and then you end up with a program that is very slow". It was an example of what he calls "Python to Python" abstraction, where the code is being refactored strictly within the language.

A "Python to C" abstraction, where the hot parts of the code are factored out into C or some other compiled language, also suffers from added costs. One could imagine that Python implementations get more and more optimizations such that the list of Point objects is represented in a simple linear array of floating-point numbers, without boxing, but if fn() is written for Python's C API, those numbers will need to be boxed and unboxed (in both directions), which is completely wasted work. It is "unavoidable with the current C API". One of the ways to speed up programs that were running under PyPy was to remove the C code and perform the calculations directly in Python, which PyPy could optimize well.

An elephant

There is an elephant in the room, however, with regard to Python performance, though it is one he rarely hears about: memory management. In today's hardware, "computation is very cheap", but memory is the bottleneck. If the data is in a cache at any level, accessing it is inexpensive, but RAM accesses are quite slow. "Generally speaking, if you want to have very very good performance, we should avoid cache misses as much as possible."

But Python is prone to having a memory layout that is cache-unfriendly. He showed a simple example: 

    class Person:
        def __init__(self, name, age):
            self.name = name
            self.age = age

    p = [Person('Alice', 16), Person('Bob', 21)]
Each Person has two fields, which ideally would be placed together in memory, and the two objects in the list would also be placed together, for a cache-friendly layout. In practice, though, those objects are all scattered throughout memory; he showed a visualization from Python Tutor. Each arrow represented a pointer that needed to be followed, thus a potential cache miss; there were nearly a dozen arrows for this simple data structure.

"This is something you cannot just solve with a JIT compiler; it's impossible to solve it without changing semantics." Python is inherently cache-unfriendly, he said, "and I honestly don't know how to solve this problem". His "sad truth" conclusion is that "Python cannot be super-fast" without breaking compatibility. Some of the dynamic features ("let's call it craziness") he had described in the talk will eventually hamper performance improvements. "If we want to keep this craziness, well, we have to leave some performance on the table."

His next slide was "The end", complete with emojis of sadness ("😢💔🥹"), which is where he ended the talk when he gave it at PyCon Italy a year earlier. This time, though, he wanted to "give a little hope" so he added a question mark, then reiterated that without breaking compatibility Python could not get super-fast.

He has a proposal for the community if it decides that Python should try to reach top-level performance, which he hopes the community does, but "it's fine to say 'no'". He suggests tweaking the language semantics by keeping the dynamic features where they are actually useful, perhaps by limiting the kinds of dynamic changes that can be made to specific points in time, so that compilers can depend on certain behavior and structure. "Not to allow the world to change at any point in time as it is now."

Meanwhile, the type system should be revamped with an eye on performance. Currently, the types are optional and not enforced, so they cannot be used for optimizations. The intent would be that performance-oriented code could be written in Python, not in some other language called from Python. But, for cases where calling another language is still desirable, the extra cost (e.g. boxing) of doing so should be removed. "Most importantly, we want something which stays Pythonic, because we like this language or we wouldn't be here."

Cuni said that he has a potential solution, "which is not to make Python faster", because he claims that is not possible. SPy, which stands for "Static Python", is a project he started a few years ago to address the performance problems. All of the standard disclaimers apply to SPy, it is "a work in progress, research and development, [and] we don't know where it will go". The best information can be found on the GitHub page linked above or in his talk on SPy at PyCon Italy in late May.

He showed a quick demo of doing realtime edge detection from a camera; it ran in the browser using PyScript. The demo shows the raw camera feed on the left and, at first, edge detection being run in NumPy on the right; NumPy achieves fewer than two frames per second (fps). Switching to a SPy-based edge-detection algorithm makes the right-hand image keep up with the camera, running at around 60fps. The code for the demo is available on GitHub as well.

He recommended the SPy repository and its issue tracker in particular for interested attendees; some issues have been tagged as "good first issue" and "help wanted". There is also a Discord server for chatting about the project. Before too long, a video of the talk should appear on the EuroPython YouTube channel.

[I would like to thank the Linux Foundation, LWN's travel sponsor, for travel assistance to Prague for EuroPython.]

Comments (16 posted)

Debian grapples with offensive packages, again

By Joe Brockmeier
August 4, 2025

A pair of packages containing fortune "cookies" that were deemed offensive have been removed from the upcoming Debian 13 ("trixie") release. This has, of course, led to a lengthy discussion and debate about what does, or does not, belong in the distribution. It may also lead to a general resolution (GR) to decide whether Debian's code of conduct (CoC) applies to the contents of packages.

The fortune program prints out a random quote or other piece of text from one or more topic databases. On Debian, for example, the topic categories shipped with the fortunes package include art, ascii-art, food, love, Linux, pets, Star Trek, and more. Those are installed by default when one installs fortune-mod, the package that contains the fortune program.

The packages that are being removed from trixie—fortunes-it-off and fortunes-scn-off—contain the Italian and Sicilian (respectively) offensive fortunes. "Offensive" here is not an editorial judgment call—the packages are officially designated as offensive in the package names and are maintained separately to ensure that users do not install them accidentally. These packages are not pulled in by default when a user installs fortune-mod; a person would have to install them separately. Even if the -off packages are installed, fortune only chooses from the database of offensive fortunes if one uses the -o option. Debian's man page for fortune includes this guidance for that option:

Please, please, please request a potentially offensive fortune if and only if you believe, deep in your heart, that you are willing to be offended. (And that you'll just quit using -o rather than give us grief about it, okay?)

Debian split the offensive fortunes in English off into a separate fortunes-off package in 1997. That package was dropped entirely ahead of the Debian 12 ("bookworm") release, following a a lengthy discussion on the debian-project mailing list. That discussion was started by Debian community team member Andrew M.A. Cater after the team had received a query about the fortunes-off package. Someone had asked if the package contained any Nazi quotes, which "might make it illegal to host the content on mirrors in at least Germany or Austria". He said that he thought any such quotes had been purged from the original BSD package that served as a basis for Debian's fortune packages.

However, he raised the question of whether Debian should remove fortunes-off since it contains quotes that "probably don't fit in with our Debian values or general societal values 25 years on". Ultimately, after much debate, it was removed.

But that was only the English version of the package. The fortunes-it-off package was still included in bookworm when it shipped in June, 2023. Salvo Tomaselli took over as maintainer of fortunes-it and fortunes-it-off in November 2023, and introduced the fortunes-scn and fortunes-scn-off packages that month as well. Note that the scn binary packages are built from the same source package (fortunes-it). The scn binary package was added on November 30, 2023.

"No place in Debian"

On July 12, Cater filed bugs against fortunes-scn-off and fortunes-it-off. Cater's bug reports simply said that the English fortunes had been removed, and the subjects for the bugs stated that the packages had "no place in Debian". He did not elaborate on any specific text in either package that he found offensive or explain why he chose to file the bugs when he did. Debian is currently in freeze and preparing for the trixie release, so a removal would have the effect of dropping the packages from the upcoming release entirely, though they would continue to be available in bookworm.

Tomaselli responded a day later; he said that the packages in question are not installed automatically and that no one had actually complained that they were offended by them. "I think it's fine." He added he was happy to accept pull requests, and had been doing maintenance on the packages after it had been abandoned for many years: "and actually removed a number of fascist, racist and sexist quotes that were not in the offensive section". With that, he closed the bugs. On July 16, Paul Gevers of the release team reopened the bugs and said that he was "reinforcing the decision of the community team"; he asked Tomaselli to please drop the offensive packages.

Cater is still a member of Debian's community team, but in a conversation at DebConf25 he said that he had filed the bugs as an individual, not as a member of the community team. While Cater did not identify himself as a member of the community team, it's not difficult to understand why someone might assume he was acting in that capacity given the history. It's possible, if Cater had provided a longer bug report with examples, and if he had stated that the report was not an official community team action, that the ensuing debate could have been avoided—or shortened. He did allow, in retrospect, that he was not good at filing bugs and that the bug report could have been better.

Can they do that?

Tomaselli sent a message to debian-devel on July 16 about the removal, and asked whether he could be forced to remove a package by the community team "even though I did not violate the COC and they did not receive any complaints?" The packages have existed since 2003, he said, "and there have been 0 complaints from offended people in these 22 years". He was annoyed that, after working on the package for two years, he was suddenly being asked to remove it while in freeze for the trixie release. "I do not appreciate wasting my time, I do not appreciate having to write this email and I do not appreciate the collective time wasted on this."

Debian does not need to ship offensive fortunes, Charles Plessy argued, "people who want to install [them] still have plenty of easy ways to do so". Maybe, he suggested, there are better battles to fight. Hanno "Rince" Wagner disagreed; as long as someone wants to spend their time curating a package, he did not think that Debian should remove it.

Didier "OdyX" Raboud observed that if the project had reached consensus that the fortunes-off package in English should be removed, "it also follows that they ought to be removed in other languages".

Debian contributor "NoisyCoil" said that they had wanted to argue in favor of keeping the packages, but after looking at the content they had decided against it:

I went peeking at the package and, unless I'm completely missing something, the second offensive Italian fortune says that women's "no"s should be interpreted as "yes", while the third one explicitly calls for violence on women [1]. Like, it literally says women should be beaten on a regular basis. I'm afraid I can't help you here, sorry.

Tomaselli said that the reason that offensive file contains calls to beat women is because he had moved it out of the "normal" fortunes. "Without me it could have remained in the normal section another 200 years for what anyone here cares about."

The debate featured many other responses both for and against removing the package, though the bulk of responses seem to favor removal. Contributors shared opinions on what is or isn't offensive and questions about other content—such as the Bible—that might offend others. If this seems familiar, it may be because similar points had been raised during the prior discussion about removing the English-language package. Long-time Debian users may recall the project grappling with the same issues more than 20 years ago when Thibaut Varene proposed packaging "hot-babe", a graphical utility that promised to display "system activity in a very special way"; specifically, it would display a graphic of a woman undressing as system temperature increased.

Getting consensus

After a few days of debate, Wouter Verhelst sent an email to debian-vote and said that "it's clear by now that we need a project-wide consensus on what policies apply to the contents of packages". When he wrote Debian's code of conduct, which was ratified by the project in 2014, he had not made it explicit that it was not meant to apply to the content of packages, though that was his intent. (Updated to add not in the previous sentence.)

Since the discussion keeps coming up, he said, the project should probably vote on a GR about the subject. There were four options he thought would be appropriate: the first two were that the CoC applies unmodified to all source code in a package or that it does not apply to the contents of packages, and that no alternative CoC is needed for packages. A third option, he said, would be that the CoC applies to "all program messages or documentation" that could be seen by a user, with exceptions for "historic texts that are widely disseminated outside of Debian". If that option had been Debian policy, the guidance would have clearly applied to this situation and made it an easy call to remove the packages.

The fourth option he thought likely was that the current code of conduct does not apply to packages, but a "code of contents" should be written that would apply to packages. Creating such a thing, however, would be a lot of work that he was unwilling to do. Anyone who wanted to propose such an option should have the text ready, "otherwise we're discussing hypotheticals rather than solutions". He said that he would make a formal GR proposal with the third option "a few weeks from now", unless the thread was still ongoing and productive.

Tomaselli said that he had also been thinking of starting a thread about a GR, but Verhelst had been quicker. Tomaselli thought there should be an "entirely new and different policy" about what to accept and not accept, because "there will inevitably be a lot of mismatches if we just apply the code of conduct to all the code we have". Verhelst replied that sounded like his option four and reminded Tomaselli that he had plenty of time to draft an option like that.

Is it really a problem?

Iustin Pop thought that package content had generally not been a significant problem in Debian, with the exception of the fortune packages. He wondered whether Debian really wanted to have a CoC for enforcing morality standards "which can change over time", or if the project wanted to be neutral and ship software as-is. Verhelst replied that he wanted an answer to the question of whether the CoC applies to Debian packages. He pointed to other discussions, such as hot-babe and the sudo insults feature, "which used to be enabled by default, but was disabled after a bug report with complaints". He said it was not absolutely necessary to have a CoC for packages, but he thought it was a good idea.

Former Debian Project Leader Bdale Garbee thought a code for acceptable content was "an exceptionally bad idea". He said that the only policy that should govern package content should be compliance with the Debian Free Software Guidelines (DFSG):

If external forces, like laws, force us to elide some content or come up with additional complications in our distribution mechanisms as US law on crypto export once did, then fine, we'll deal with those when we must. But trying to apply some sort of moral code to package content, or offering to avoid offending individuals or groups with the software we distribute, feels likely to be directly in conflict with DFSG 5, "No Discrimination Against Persons or Groups", and/or DFSG 6, "No Discrimination Against Fields of Endeavor".

At best, Garbee said, it would be a slippery slope given the differences of opinion that exist within the Debian project. Tiago Bortoletto Vaz felt that a GR was unnecessary. He said it was not a recurrent problem for the project, "so perhaps it doesn't really need new rules":

Honest question: in ~30 years, how many packages have been removed from our archive due to offensive content? 4? 5? How many of the removal requests turned into big drama?

"AFAIK, all of them", replied Tomaselli. This was the first time he had encountered package removal as a Debian developer, but not his first time "having packages disappear because someone else decided they were immoral". Having a firm rule would be less controversial, he said.

Russ Allbery noted that, since there is no content policy for packages, every time there's a question about content it gets argued "to death" on the mailing lists:

A careful debian-devel observer could have listed most of the people who would respond to the thread and written a pretty good paraphrase of what they would say the moment they saw the first few messages in the thread.

Verhelst said, that it would also make life easier for Debian's release team to have a concrete policy. The team had decided that the package should be removed, based on Debian's code of conduct, but Verhelst said it was unclear whether the CoC even applies to packages. He also questioned whether the release team is even the right team to make the decision, though he was quick to add that was not meant as a criticism: "They do a hard job under difficult circumstances, and that is appreciated".

Clearly, he said, some people in the project believe that Debian should have a content policy that should be imposed. That being the case, the project should make a decision:

And honestly, if we think about this and decide as a project that "anything is allowed except things that are obviously illegal", then that's fine with me too. I just want us to think about this and make the call, rather than leaving this to a team that really have a different responsibility and will take the flak for doing something that shouldn't even be their job but nobody else is doing it.

Later he added that a policy is already being imposed with the removal of the offensive fortune packages:

This means that we already do have an effective code of acceptable conduct, decided by the release team and not the project at large, and I think that is wrong.

For now, the conversation has largely died down; many of the participants are no doubt busy getting trixie ready for its release on August 9 without the offensive fortune files. Sometime after the release, Debian may finally come to a firm decision on whether it wants to moderate the content of packages, and where it draws the line if it chooses to impose one.

Comments (129 posted)

The NNCPNET email network

By Daroc Alden
August 1, 2025

Running a modern mail server is a complicated business. In part, this complication is caused by the series of incrementally developed practices designed to combat the huge flood of spam that dominates modern email communication. An unfortunate side effect is that it prevents people from running their own mail servers, concentrating people on a few big providers. NNCPNET is a suite of software written by John Goerzen based on the node-to-node copy (NNCP) protocol that aims to make running one's own mail servers as easy as it once was. While the default configurations communicates only with other NNCPNET servers, there is a public relay that connects the system to the broader internet mail ecosystem.

NNCP

NNCP is a spiritual successor to the unix-to-unix copy protocol (UUCP) in the same way that SSH is a spiritual successor to rsh. Sergey Matveev created the project to be an easier to use, more secure replacement for UUCP. Like its predecessor, NNCP is both a (relatively simple) protocol, and a suite of reusable single-purpose command-line tools. The project's code is licensed under the GPLv3.

As with early computer networks, routing in NNCP is manual. In order to contact a host, the user needs to explicitly tell NNCP what routes are available to connect to it. In modern parlance, that makes it a friend-to-friend network, where each computer connects only to computers that the owner knows about and trusts. This makes it impossible to tell exactly how many people are using NNCP, because there's no network-wide consensus on membership. Routes may also be indirect: if two servers aren't directly connected, but they both know how to reach some third server, they can be configured to talk to each other via the third server. This avoids the need for explicit bang paths in each email, but is still a form of manual routing. In practice, many users likely have a connection to the quux.org server operated by Goerzen for this purpose.

That server publishes a list of all of the nodes that it knows about (by making the file available via NNCP file requests). Being added to the list is a manual process that requires sending Goerzen an email. This is reflective of the back-to-basics ethos of the NNCP network: the tools focus only on the core job of sending data where they're told, leaving decisions about the network's structure and policies up to the human administrators.

Messages in NNCP take the form of encrypted packets that can be exchanged via the internet, shared storage, or pretty much any other kind of connection. The tools have built-in support for exchanging data at specified times, so that bulk traffic can be sent when a network is not otherwise being used. NNCP is a store-and-forward network, so it directly supports nodes with intermittent connections, such as an air-gapped computer that uses a removable storage device to send messages to other computers. When a message needs to be sent via another node as a relay, the payload is wrapped in layered encryption, so that each node along the configured route removes a layer. This makes it harder for an outside observer to tell the difference between a message destined for a specific server, and one that is merely being relayed through it.

An example

Suppose that node A wishes to send a message to node C, relayed via node B. All three of these nodes know each other's public keys, because the humans configured them out of band. If we use "E(msg, key)" to mean the encryption of the message using some public key and "||" to mean concatenation, A computes "E(relay header || E(msg, C-pub), B-pub)", and sends that to B (possibly over a link secured with A and B's transport keys, possibly via a sneakernet). B receives the message, decrypts it, and sees that it is a request to relay a message to C. B sends "E(msg, C-pub)" to C, which decrypts the message.

In addition to the at-rest encryption used on messages awaiting delivery, connections between nodes that operate over the internet are also encrypted, to further disguise message metadata. Two separate keypairs, one for the at-rest encryption and one for the in-transit encryption, are generated when the node is initialized using the nncp-cfgnew command. The public part of these keypairs needs to be communicated to every other server that it will communicate with. Keys aren't expired or rotated automatically — users wishing to rotate their keys will need to do so by hand.

NNCP's flexibility is both a blessing and a curse; on the one hand, it follows the Unix philosophy of letting the user compose their own special-purpose tools to do things that the developers didn't anticipate. On the other hand, it makes setting up a node to participate in the network a manual (and potentially error-prone) process. Goerzen wrote NNCPNET as an automated way to set up an NNCP node for the common case of handling email, and as a way to introduce people to NNCP.

NNCPNET

When I asked Goerzen about his motivation for working on NNCPNET, he said: "I want people today to be able to enjoy networking and the Internet the way I did in 1998". Goerzen started running his own mail server around that time:

For 24 years since 1995, I ran my own mail server for complete.org, on various hosted VPSs and even once a dedicated box at the house of a friend with fast Internet. In 2019, I gave it up, because it was just too difficult. SPF, DKIM, DMARC, TLS, getting blacklisted by email providers due to issues outside my control, etc.

When he stopped, he needed some way for his various servers to send him basic cron emails — he couldn't even use SMTP because of provider restrictions. Luckily, email was once transported via UUCP, and support for that mode of operation still exists in a lot of mail servers. So, he realized that all that was really needed was to write some supporting software to let mail servers connect via NNCP as well; that effort became NNCPNET.

NNCPNET takes the form of a docker container with NNCP, Exim, Dovecot, and a handful of other utilities installed. Setup is mostly automated, with a few things left for the user to do that are described in the documentation. The container has a cron job that is responsible for periodically downloading the list of nodes that quux.org is aware of and adding NNCP routes for them that go via quux, for example, but getting registered on the list is still manual. Configuring everything without docker is definitely possible, since all the individual pieces of software are packaged separately, but Goerzen has not written instructions for that.

Once set up and running, NNCPNET acts as a normal email server for normal email clients. Unlike email running over the internet, however, NNCPNET uses the fact that NNCP messages are end-to-end encrypted and signed in order to validate the sender of each message. That, plus the friend-to-friend nature of the network, is supposed to strongly discourage spam. There is also a public relay that can optionally be used to connect NNCPNET addresses to the open internet, which Goerzen politely asks people not to abuse.

Emails within NNCPNET all end in ".nncpnet.org"; for example, the test robot that can be used to validate one's email setup is "testrobot@mail.quux.nncpnet.org". Generally, the local part of the email ("testrobot") is completely up to the user; the domain part will start with their chosen NNCP node name, which is ultimately limited by whatever they can get other people to put in their configuration files. Sending an email from one NNCPNET address to another NNCPNET address works like this:

  1. The user's mail client submits mail to Exim running in the NNCPNET container (locally or via SMTP).
  2. An Exim script extracts the NNCP name of the destination host (e.g. "mail.quux") and passes the email off to the appropriate NNCP command.
  3. NNCP prepares a message with the email in it, possibly with several layers of encryption of the packet needs to be relayed through multiple hops.
  4. That message sits in the NNCP spool directory until something sends it to the next hop in some way — via scheduled connection, sneakernet, or otherwise.
  5. Any intermediate hops receive the message, strip a layer of encryption off of it, and sort it into the appropriate outgoing spool.
  6. Eventually, the destination NNCPNET server receives the message, decrypts it, and checks that the sending address has the same host as the NNCP signature.
  7. Then it ingests the email as normal, and the recipient eventually fetches it with their mail client.

Sending email between the open internet and NNCPNET is fairly similar, except that the destination host on the NNCP side is Goerzen's bridge. He does have plans to remove that single point of failure: "NNCP itself is decentralized and I would like NNCPNET to be as well. I have put some thought to that already." The bottleneck is "more interested contributors". The project doesn't take much day-to-day attention, Goerzen said, but his time is still limited.

Overall, NNCPNET is not really in a state for non-technical users, but typical LWN readers should not have a problem setting it up, perhaps as a weekend project. The documentation is fairly thorough, but I did run into two problems in getting my setup working: a problem with systemd inside the docker container expecting to have access to the host system's control-group hierarchy, and needing to fiddle with the NNCP configuration in order to do local testing with multiple nodes. That said, setting up an email server with NNCPNET is definitely possible. I asked Goerzen what he wanted people to take away from trying out the project. He replied:

You can run your own mail server again. A real one, not a toy. You can build email networks. They can be limited to NNCP only, or full participants on the Internet. Your phone can be a mail server and run a mailing list. You don't need an email provider for everything: just run it yourself. Your scanner with its "scan to email" function can work and do just that, without having to have a complex workaround because many modern email accounts require OAUTH and such.

NNCPNET is, at the moment, fairly small. The quux nodelist contains 16 entries — two of which are for quux itself, one of which is for Goerzen's personal email host, and one of which I set up for testing. So the number of people actively using the network is likely to be somewhere around a dozen. Whether NNCPNET eventually picks up more users, or remains a niche alternative remains to be seen.

Comments (34 posted)

6.17 Merge window, part 1

By Jonathan Corbet
July 31, 2025
As of this writing, just over 4,000 non-merge changesets have been pulled into the mainline repository during the 6.17 merge window. When he announced the merge-window opening, Linus Torvalds let it be known that, due to a busy personal schedule, he was likely to pull changes more quickly than usual this time around; that has been borne out to some extent. Changes merged so far are focused on core-kernel and filesystem work; read on for the details.

The most significant changes merged so far include:

Architecture-specific

Core kernel

  • The core-dump socket functionality added in 6.16 has been enhanced with a protocol that allows the core-dump server to control how dumps are handled for each task. This commit describes how the protocol works and the options that are available. There is also an experimental server implementation available that exercises this interface.
  • There have been a number of changes to the pidfd functionality. The associated kernel-internal information created with a pidfd is now tied to the process itself rather than the pidfd, allowing it to continue to exist between opens of same process. It is now possible for user space to attach extended attributes to a pidfd; those, too, will exist for the life of the process. File handles for pidfds can now be opened (using open_by_handle_at()) without needing a file descriptor for the containing filesystem. This merge message has more information about these changes.
  • The new bpf_cgroup_read_xattr() kfunc allows BPF programs to read extended attributes from the control-group filesystem.
  • The kernel's timekeeping system has grown the notion of "auxiliary clocks" that are not tied in any way to the normal system clocks. Until now, all clocks advanced at the same rate and differed only in their offsets; auxiliary clocks march to their own drummer. Documentation is scarce, but there is some information in this merge message.
  • Support for uniprocessor configurations has been removed from the scheduler; even single-processor machines will run a kernel built for SMP systems.
  • Initial support for proxy execution, which is meant to mitigate priority-inversion problems, has been merged. Proxy execution allows a task waiting for a lock to donate its execution context to the lock holder, speeding the release of that lock; in 6.17, it only works if both tasks are running on the same CPU.

Filesystems and block I/O

  • Btrfs has gained large-folio support, though it is still deemed experimental for this release.
  • Defragmentation of Btrfs filesystems now offers more control over the use of compression in the defragmented extents.
  • The EROFS filesystem now supports metadata compression.
  • The NFS server can offer write delegations to clients that open files in write-only mode. From the merge message: "We're expecting this to accelerate a few interesting corner cases".
  • The ext4 filesystem has gained support for buffered I/O with the RWF_DONTCACHE (originally proposed as RWF_UNCACHED) flag, which will cause the data to be dropped from the page cache once the operation is complete.
  • The inode number for the root of the /proc filesystem is now an acknowledged part of the kernel ABI; it is called PROCFS_ROOT_INO. User-space can check this number to ensure that /proc files have not been replaced, using a bind mount, by an attacker; see this merge message for more information.
  • The fallocate() system call has a new option, FALLOC_FL_WRITE_ZEROES, that causes the allocated range to be initialized to zeroes in the most efficient way possible. With recent solid-state devices, this initialization can be done efficiently within the device, with no I/O required. See this changelog for a bit more information; the ext4 filesystem now supports this operation.
  • Two new system calls, file_getattr() and file_setattr(), support the manipulation of a file's inode attributes. See this commit for an overview and basic manual page. Documentation of the actual operations supported is scarce, but this XFS manual page gives a good overview.
  • The "pktcdvd" packet-writing optical-media driver has been marked as deprecated since 2016. There was an attempt to remove it in 2023 that was reverted. In 6.17 it has been removed again, and will likely stay out this time.
  • Thus far, the bcachefs pull request has not been acted upon.

Hardware support

  • Clock: Raspberry Pi RP1-based clocks.
  • GPIO and pin control: Apple Mac SMC GPIO controllers and Raspberry Pi RP1 pin controllers.
  • Industrial I/O: Analog Devices AD4080 high speed analog-to-digital converters, Analog Device AD7405 and AD4170-4 analog-to-digital converters, ChromeOS EC activity sensors, and Nicera D3-323-AA passive infrared sensors.
  • Miscellaneous: HiSilicon uncore frequency-scaling controllers, Analog Devices ADP558x keypads, Apple silicon system management controllers, Apple SMC reset/power-off controllers, Raspberry Pi 7-inch touchscreen panel V2 regulators, Argon40 Fan HAT controllers, Lenovo GameZone WMI interfaces, Intel Platform Monitoring Technology interfaces, Qualcomm Milos interconnects, Renesas RZ/T2H serial interfaces, Canaan Kendryte K230 SoC reset controllers, NXP IMX Secure AHB to IP Slave bus (AIPSTZ) bridges, Allwinner A523 PCK-600 power domain controllers, Loongson-2K SD/SDIO/eMMC host interfaces, and AMD heterogeneous core hardware feedback interfaces.
  • Sound: Richtek RTQ9124 mono class-D amplifiers.
  • SPI: Amlogic SPISG controllers and Renesas RZ/V2H RSPI controllers.

Security-related

  • The new FS_IOC_GETLBMD_CAP ioctl() command allows user space to retrieve information about the integrity protections that apply to a given file. In 6.17 it is only supported by block devices, but the plan is to support it within filesystems as well. See this commit for an overview of this new API.
  • The kernel can now use the stack-depth tracking provided by Clang 21 to implement kernel-stack erasing, along the lines of what STACKLEAK provides with GCC.
  • The deprecation of the /sys/fs/selinux/user API continues; attempts to access it will now result in a five-second delay along with a logged warning.

Internal kernel changes

The 6.17 merge window can be expected to close on August 10, but Torvalds has made it clear that he wants to see all significant pull requests well ahead of that date. Given that, as of this writing, there are still over 7,500 commits sitting in linux-next, it is evident that quite a few of those requests are yet to be acted on. As always, LWN will provide a summary of the remaining changes once the merge window closes.

Comments (11 posted)

Improving control over transparent huge page use

By Jonathan Corbet
August 5, 2025
The use of huge pages can significantly increase the performance of many workloads by reducing both memory-management overhead in the kernel and pressure on the system's translation lookaside buffer (TLB). The addition of transparent huge pages (THP) for the 2.6.38 kernel release in 2011 caused the kernel to allocate huge pages automatically to make their benefits available to all workloads without any effort needed on the user-space side. But it turns out that use of huge pages can make some workloads slower as the result of internal memory fragmentation, so the THP feature is often disabled. Two patch sets aimed at better targeting the use of transparent huge pages are currently working their way through the review process.

Over the years, the kernel has evolved a number of ways to control the use of THP; they are described in Documentation/admin-guide/mm/transhuge.rst. At the global level, the /sys/kernel/mm/transparent_hugepage/enabled knob controls behavior system-wide. It can be set to "always" or "never" with obvious results. This knob also supports the "madvise" setting, which only enables THP for processes that explicitly opt in for specific memory regions with a call to madvise(). The kernel, in other words, allows for the imposition of a system-wide policy, with the possibility of restricting THP usage to places where applications have explicitly enabled it.

Tweaking prctl()

There are more control points for THP usage, though, including a whole set of knobs for the khugepaged kernel thread (which builds huge pages out of base pages in the background) and a set of kernel command-line options. There is also the PR_SET_THP_DISABLE option to prctl(), which lets a process disable the THP usage for its entire address space regardless of the overall system policy. This option is used in programs where the developer feels that THP allocations will only harm performance; for example, MariaDB uses this feature.

The implementation of PR_SET_THP_DISABLE is absolute; it will override any madvise() calls that the process may subsequently make to enable THP for an address range. Disabling THP process-wide in this way is a big hammer. There are use cases where it would be useful to be able to disable THP generally (on systems where it is enabled globally), while still being able to enable THP for specific memory regions, but the kernel does not currently offer that option. There is, in other words, no way for a process to request "do not use THP unless otherwise requested".

As described in this patch from David Hildenbrand (which has been made part of this series by Usama Arif), one could consider letting madvise() override PR_SET_THP_DISABLE but, as Hildenbrand put it, "this would change the documented semantics quite a bit". There may also be cases where the current semantics are what is actually wanted, so this does not seem like a good option.

Instead, Hildenbrand's patch adds a new option to PR_SET_THP_DISABLE, called PR_THP_DISABLE_EXCEPT_ADVISED, that provides the new semantics. When this option is provided (as the third parameter to prctl()), THP will be disabled process-wide except for regions where madvise() has been used to request an exception, addressing the above-mentioned use case. This series has been through a couple of revisions and appears to be stabilizing.

This new option may solve an immediate problem, but it is one more knob added to a complex and interacting pile of them. It also does not address another aspect of the THP problem. Once upon a time, there was only one size of THP: the "PMD size", which is 2MB on many systems. The advent of multi-size THP, which allows many different sizes of huge pages, has complicated the situation. One result of that can be seen by looking in the /sys/kernel/mm/transparent_hugepage directory, which now includes subdirectories for a range of huge-page sizes. The system administrator can use this large array of knobs to control the allocation of specific THP sizes system-wide, but there is currently no way to tune the policy with size granularity for a specific process's needs. The idea of adding more complexity to the prctl() and madvise() interfaces to provide that flexibility is looking increasingly unappetizing.

Using BPF

At the conclusion of his changelog, Hildenbrand acknowledges that the new prctl() option does not solve the whole problem:

Likely, the future will use bpf or something similar to implement better policies, in particular to also make better decisions about THP sizes to use, but this will certainly take a while as that work just started.

The current form of that work can be seen in this patch set from Yafang Shao. It introduces a new struct-ops hook with this BPF callback: 

    int (*get_suggested_order)(struct mm_struct *mm, unsigned long tva_flags,
                               int order);

If a BPF program is attached to this hook, it will be called from the memory-management subsystem when page-allocation decisions are being made. The mm argument will point to the mm_struct structure describing the address space, tva_flags describes the current context (and, specifically, whether a page fault is being handled), and order is the largest size that is being considered for this allocation. The function should return an order number indicating the largest allocation that should actually be performed; returning zero will disable huge pages entirely in this case.

To make this decision, the BPF program will likely want to know a bit about the context in which the allocation is being made. The patch series adds a couple of new kfuncs, bpf_mm_get_mem_cgroup() and bpf_mm_get_task(), to give that program access to the relevant control-group and task information. Shao notes in the series that there may be a need for other helpers in the future to provide information about the memory pressure that the system is currently experiencing.

The patch series, currently in its fourth revision, has not yet reached the point where it is ready to go upstream. One significant change was requested by Zi Yan, who would like to give the callback access to the relevant virtual memory area (VMA) as well. Shao answered that "mm is sufficient for our use cases", but Hildenbrand said that an approach based on virtual memory areas would be required.

A separate concern, raised in regard to the previous revision, was that any BPF callbacks could quickly find themselves cast in stone as part of the kernel's ABI. That is especially undesirable given that the problem of how to best control the use of THP is clearly not fully understood at this point. As Hildenbrand put it:

Every MM person I talked to about this was like "as soon as it's actively used out there (e.g., a distro supports it), there is no way you can easily change these callbacks ever again - it will just silently become stable."

That is actually the biggest concern from the MM side: being stuck with an interface that was promised to be "unstable" but suddenly it's not-so-unstable anymore, and we have to support something that is very likely to be changed in the future.

As a result of these concerns, the current patch set includes warnings that the interface could change or even be removed in future kernel versions. That matches the general understanding around BPF programs; they are not normally deemed to be a part of the kernel's stable ABI. That said, if a given BPF interface becomes sufficiently deeply integrated into how systems are managed, it will become increasingly hard to change or remove. Memory-management developers could end up having to support it for years, even if it gets in the way of needed changes.

So the addition of BPF control for core memory-management decisions is likely to be approached with a great deal of caution, even by developers who feel that it is the most logical way to solve the problems around control of THP usage. There will almost certainly be a way to control THP allocation with BPF in a future kernel, but that may not happen as quickly as some might like.

Comments (5 posted)

A look at the SilverBullet note-taking application

By Daroc Alden
July 31, 2025

SilverBullet is a MIT-licensed note-taking application, designed to run as a self-hosted web server. Started in 2022, the project is approaching its 2.0 release, making this a good time to explore the features it offers. SilverBullet stores notes as plain Markdown files, and provides a Lua scripting API to customize the application's appearance and behavior.

Architecture

SilverBullet features an unusual design; despite being a single-user application, it is written as a self-hosted web app. The project is distributed as a standalone server executable. The web-based interface that it provides is a progressive web app (PWA), which lets devices that support PWAs, such as mobile phones, optionally "install" the application as though it were a native executable. For all other devices, it can be accessed through the browser like a normal website. The server is responsible for reading and writing files from disk, and synchronizing changes between multiple clients. When a new client connects (by opening the web interface), it furnishes the client with a full copy of the user's notes.

That copy is kept in the client's local storage; while browsers do limit this space, it's still more than large enough for a set of plain-text files. The rendering, editing, indexing, and even Lua scripting of the files is performed by the client code running in the browser. This lets the user view and edit their files even when offline; when they reconnect, the changes are automatically pushed back to the server. During my testing, I did notice that the synchronization protocol is fairly chatty. Rather than using push notifications or WebSockets connections, SilverBullet directly polls the server every few seconds. This is not a problem, per se, but some users may find it inelegant.

While SilverBullet does allow specifying a username and password that must be given to access the application, the intended deployment scenario seems to be putting it on one's local computer, a Raspberry Pi in one's home, or on an existing private network, so that only trusted devices can connect to it. SilverBullet does not support multiple users or editing permissions, but it does support running in a read-only mode, to allow publishing static versions of a set of notes.

Both SilverBullet's client and server are written in TypeScript, although there is also a substantial Lua library that implements SilverBullet's database and styling features. The server program is a self-contained bundle of all the components, so users who don't wish to compile the project from scratch can just download the file from the project's GitHub repository and run it. Running the project for the first time creates an index.md file with some tips to get started:

[A screenshot of SilverBullet's welcome page]

Editing

The core of any note-taking application is the editing experience. SilverBullet's editing is an interesting hybrid between "what you see is what you get" (WYSIWYG) and editing plain Markdown. Specifically, individual Markdown files are rendered according to the CommonMark standard, a unified version of Markdown supported by several tools. The rendered form appears on the page with its full rich formatting. In fact, on first looking at my new installation, I did not realize that the text (pictured above) was actually directly editable. Placing the cursor anywhere on the page reveals the truth: the underlying document can be edited right there, with no separate editing mode. Moving the cursor inside a Markdown element (such as a **bold** phrase) reveals the markup used to create it, so that one can edit it. The usual Control-b (bold) and Control-i (italic) shortcuts are available as well.

[A screenshot of SilverBullet's editing interface]

Vim users will find that SilverBullet supports optional Vim-style keybindings. Although I myself am an unrepentant Emacs user, the Vim keybindings seemed accurate enough to me that I believe muscle memory should transfer over.

Another feature borrowed from existing editors is the ability to run commands, which can be defined in Lua, bound to keys, or invoked by name from a searchable list. SilverBullet takes an interesting approach to configuring and customizing the application; Lua code can be directly embedded in a file as a Markdown code block, and either referenced elsewhere, or run directly and spliced into the rendered page. This lets the user create interactive elements in their notes. The application's configuration lives in CONFIG.md; any Lua code in that file is run when the SilverBullet client starts, to set everything up. Anything that can't be handled using the Lua API can be done with the TypeScript plug-in API, which unfortunately doesn't work in the same manner, requiring one to write separate TypeScript files.

Tags and searching

SilverBullet supports tagging many parts of Markdown files, such as tasks, code blocks, bullet-list items, and so on. Writing #tagname in a document will attach the tag to whatever Markdown element it appears after or within. Parts of files are also automatically marked with certain default tags, such as what type of Markdown element they are and what page they appear on. This is, so far, not much different from any note-taking application. Where SilverBullet differs is that it makes a searchable index of these tags available not only to the user, but also to the Lua code fragments embedded in their documents.

The application provides a custom SQL-like query language for interacting with the tag database. SilverBullet's documentation shares this example of how that can be used to create interactive, always up-to-date tables of tasks, pages, or other items meeting a given criterion: 

    ${template.each(query[[
      from index.tag "page"
      order by _.lastModified desc
      limit 5
    ]], templates.pageItem)}

Readers familiar with Lua will recognize the [[...]] multi-line string syntax, here being used to give a string to the query() function. This snippet searches for the five most recently modified pages, renders them using the templates.pageItem template (which renders an appropriately titled link to each page), and then embeds the result in the current page using the special ${...} syntax.

The tag database also powers several of SilverBullet's internal features. For example, the default style for a page includes a snippet at the bottom that finds all of the places in the user's documents that link to that page, and displays a back-link with a little bit of context. The project's community forums also contain a discussion board for sharing useful bits of Lua code.

Future prospects

SilverBullet has, for the past several months, been undergoing a rewrite from version 1 to version 2. At the time of writing, the project has published four pre-releases for 2.0.0, and seems likely to stabilize things quite soon. Some features from version 1, such as "online mode" and non-Lua-based interactive queries, were removed because they made it difficult to extend and support the application. Zef Hemel, the founder and primary developer of the project, explained the move like this:

SilverBullet has grown into a large project over the last three years. About half a year ago I started working on Lua integration, and this [made] me discover a significantly more elegant way to implement a bunch of features in SilverBullet than all the custom features I had built before.

Initially, I implemented this "alternative universe" style, but lately this has become a significant burden for me to mentally handle and maintain both these universes long term. Therefore, I contemplated taking a more radical approach and introducing a "v2" where I would do some serious rapture-style removal of features to simplify the product, but also keep it more sustainable by — well — primarily me (although I do get more and more outside contributions, which is great).

Outside contributions to the project are welcome but currently somewhat infrequent. Documents from version 1 are viewable and editable in version 2, since they're both just Markdown. But the documentation does have a guide on converting old queries and interactive elements to use the new Lua API.

Conclusion

While it will be hard for any note-taking program to tempt me away from Emacs's org-mode, I do think SilverBullet is a compelling alternative. Its design as an offline-first web app backed by plain files promises to offer the best of both worlds. Plain files can be imported, exported, and backed up trivially; but making the primary interface web-based allows for easy styling and synchronization across devices. The integration of Lua scripts directly into the user's documents is a flexible platform on which to customize a potentially personal platform.

While the 2.0.0 release is not quite ready, the pre-release that I tested seemed stable. People who are happy with their current note-taking solution are unlikely to want to switch, but those looking to try a simple, flexible alternative may well want to give SilverBullet a look.

Comments (10 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

  • Briefs: AUR malware; Secure boot; kbuild and kconfig maintenance; GPU drivers; NVIDIA on AlmaLinux; Proxmox 9.0; Quotes; ...
  • Announcements: Newsletters, conferences, security updates, patches, and more.
Next page: Brief items>>

Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds