AccessKit

New website

2024-08-25T00:00:00+00:00

We are proud to unveil this new website with better colors and up-to-date content. More accessible, lighter, easier to maintain. This is exactly what we wanted for a while now.

We hope to post here more frequently, so stay tuned!

If you encounter an issue with it though, feel free to report it</a>.

AccessKit featured at RustConf

2023-10-04T15:20:33+00:00

I gave a presentation about AccessKit at RustConf 2023. The presentation featured demonstrations of what currently works and discussed how the community can get involved.

Here's the video: </iframe>

Talon Voice Control Application Implements AccessKit in Latest Beta

2023-03-16T15:24:48+00:00

It is with great pleasure that we recognize Talon</a>, a cross-platform voice-control application, as the first end user application to implement AccessKit! Current Windows and Mac beta builds of Talon have had their app core ported from QT to Egui, thereby adopting AccessKit. Linux support is coming soon pending a bug fix.

We would like to extend our sincere thanks to the Talon developers for integrating AccessKit into their project.

Users can currently obtain an accessible beta build of Talon via their Patreon</a>.

AccessKit integration makes Bevy the first general-purpose game engine with built-in accessibility support

2023-03-06T18:57:39+00:00

This is just the start of the journey toward enabling fully accessible games with Bevy; you can read more about what still needs to be done in the announcement linked above. Still, we’d like to thank the Bevy core team members, as well as Nolan Darilek</a>, for working with us to reach this milestone. We look forward to continuing to work with Nolan and the Bevy core team to make first-class accessible games a reality.

Dramatically reducing AccessKit’s memory usage

2023-02-08T16:51:25+00:00

In our recent status update</a>, we called out the use of a single large data structure for all accessible UI elements as a known potential weakness in the design. After that post, feedback from potential users of AccessKit made it clear that this design flaw was a pressing problem that blocked them from using AccessKit. One of these discussions also led us to a particularly attractive technique for solving the problem. So we decided to go ahead and do this optimization work, to unblock further adoption of AccessKit and get the inevitable incompatible API changes out of the way sooner rather than later. This post summarizes the results of this optimization, explains how we did it, and looks ahead to further potential optimizations.

The numbers</h2>
To provide meaningful numbers, let’s look at a concrete example. Suppose we have a simple static text node with a bounding rectangle. Since the text string is allocated on the heap in both cases, we won’t consider it here, but we’ll look at the size of everything else. These numbers are for a typical 64-bit platform.
Before the recent optimization, each AccessKit node was a single structure with a size of 1,416 bytes, excluding heap-allocated strings and arrays.
Now, calculating the total size is more complicated, but the end result is more efficient. The short version is that the size of our example node is reduced by 5x or more.
Here’s the long version: The base node structure is only 32 bytes. Then we add 40 bytes for each dynamic property. Our example node has two properties, the text itself and the bounding rectangle, for a total of 80 bytes for the properties. Finally, each node has a class, which is a structure with a current size of 100 bytes. So the total size of our example node is 212 bytes, plus the heap-allocated text string, plus any overhead for the heap-allocated and reference-counted node class and property array. These latter overheads are partially platform-specific and more difficult to measure, but you get the idea; the new scheme is dramatically more memory-efficient.
Also note that, because the node class is reference-counted, it can be, and usually is, shared between nodes that have the same class. A node class consists of the node’s role, supported actions, and indices of set properties. So if we have multiple static text nodes that were constructed the same way, i.e. with the same properties set in the same order, these nodes will share a class. That means that the size of the shared node class is 100 bytes plus the overhead of heap allocation and reference counting, and the size of each static text node with the same class is only 112 bytes plus the overhead of heap-allocating and reference-counting the property array.
What does this mean for AccessKit users?</h2>
As mentioned above, this optimization requires a breaking change to the AccessKit API. In short, the previous node structure with public fields has been replaced with two opaque structures, Node and NodeBuilder, which have methods for getting and setting the node properties. This new design is more future-proof, allowing further optimizations, so we don’t anticipate another breaking change of this magnitude. The necessary changes in an AccessKit provider (e.g. GUI toolkit) are tedious but straightforward. egui</a>, the first GUI toolkit to integrate AccessKit, is already updated, thanks to the outstanding responsiveness of Emil Ernerfelt</a>, that project’s maintainer. Check out the recent egui commit</a> to get an idea of what changes will be required in other AccessKit integrations.
How did we get here?</h2>
In our opinion, the final, optimized design was not obvious. That’s why we started with the naive approach of using a single large structure. We knew that approach was inefficient, but we weren’t fully satisfied with other options we considered. Alternatives that we considered included:

Multiple arrays of key-value pairs, one for each type of value, e.g. one array for strings, another for integers, and so on. This is what Chromium’s accessibility implementation uses. But we weren’t satisfied with the size overhead of multiple heap-allocated, dynamic arrays (vectors).</li>
A single array of key-value pairs or a hash table, in which each value was a Rust enum (called a variant in other languages) allowing all possible types. This type of enum did end up being a key component of the final design. But we weren’t happy with the idea of doing linear search through an array of key-value pairs (a problem that would also affect the previous option), and in one of the earliest AccessKit design discussions, we were warned about the overhead of using a hash table for each node.</li>
Abandoning a one-size-fits-all node structure altogether in favor of a trait (called an interface in other languages) that could be implemented by each provider. This would be more similar to what the platform accessibility APIs themselves define. In principle, this would allow each AccessKit provider (i.e. GUI toolkit or application) to implement an optimal structure for representing each type of node that it can provide. But this approach has its own downsides. In particular, a key property of the current AccessKit design is that all nodes can be easily serialized, so an accessibility tree or incremental tree update can be pushed to another process or even another machine. We haven’t yet fully explored the possibilities enabled by this feature, and we didn’t want to compromise those possibilities. We were concerned that calling an opaque function through dynamic dispatch (also known as a virtual function call) for each node property would make future push-based implementations much less efficient than they can be when serializing static data. So the trait/interface approach wasn’t a clear win.</li> </ul>
Then, Federico Mena Quintero</a>, a founder of the GNOME desktop project who is now working on GNOME accessibility, drew our attention to an article about how he reduced memory usage in his librsvg project</a>, which is also written in Rust, and we immediately knew that this article described an approach that would also work well for AccessKit. Briefly, this approach uses two arrays: a dynamic array (vector) of property values, and a statically sized array of entries for all properties. Each entry in the second array is either an index into the first array, or a sentinel value if the property is not set. Because both librsvg and AccessKit have fewer than 256 properties, each entry in the second array can be just one byte. Thus we avoid the overhead of a hash table, but we also don’t need to do linear search whenever we fetch a property.
This insight alone would have significantly reduced our memory usage, and we did start by implementing this optimization by itself for all properties that occupied more than 1 byte in the original structure. But then we realized we could go further. First, we packed all of the flags into a single 32-bit integer. Of course, this was an obvious optimization, but we didn’t do it before because the fields of our structure were all public, we were using automatically generated serialization code, and we wanted the serialized representation of a node to be completely natural to users of higher-level languages. In the end, we were still able to meet that latter goal through a custom implementation of serialization.
Then we realized that Federico’s array of property indices is similar to the vtable design of FlatBuffers</a>. And in FlatBuffers, a single vtable can be shared by multiple tables (the primary data structure in FlatBuffers), as long as those tables are part of a single buffer. We rejected FlatBuffers itself for both legal and technical reasons. First, FlatBuffers is available solely under the Apache license, which is not compatible with GPLv2, and we wanted AccessKit to remain GPLv2-compatible. But also, a FlatBuffers vtable can only be shared across tables in a single buffer, and that wouldn’t have worked for sharing property tables across multiple accessibility tree updates. Finally, each entry in a FlatBuffers vtable is 4 bytes, as opposed to the 1-byte property indices in librsvg and AccessKit. Still, we liked the idea of a shared property table and wanted to use it somehow.
Finally, we realized that a shared property table wouldn’t require anything as esoteric as arena-allocated buffers with a custom layout, as implemented by FlatBuffers; we could do it in ordinary, safe Rust, using reference-counting and a simple shared set. Then, because the idea of dynamically constructing a property table, then determining if the table could be shared with other nodes, seemed conceptually similar to V8’s hidden classes</a>, we decided to wrap the shared property table in a construct that we call a node class, which we then extended to include the node’s role and supported actions. At this point, it also made sense to extend the set of dynamic properties to include the non-boolean properties that occupied just 1 byte in the original structure, to further reduce the base size. And with that, we arrived at the final, optimized design that yields the numbers presented above.
Can we do better?</h2>
While this design is more optimized, it’s certainly not perfect.
As mentioned above, the size of each dynamic property is 40 bytes (for a typical 64-bit platform). While we believe this is good enough for now, we’d like to see if we can do better. The current limiting factor is the size of our rectangle structure, which is 32 bytes. To get around this, we could allocate the bounding rectangle separately on the heap, but we’re reluctant to do this for such a common property. Alternatively, because the bounding rectangle property is so common, we could include it in the base structure, increasing the size of that structure to 64 bytes. That would make the optimization less impressive on paper, but may be a win overall in real-world use cases.
While this optimization is a clear win for data size, it’s a modest regression for compiled code size in some cases. The executable size of egui’s hello_world example, when compiled with egui’s release build settings on x86-64 Windows, increased by about 15 KB. However, when compiling the same example with maximum optimization for size, link-time optimization, and panic = "abort", the AccessKit node size optimization actually decreased executable size slightly. So maybe we just need to add some more inlining hints. We’d appreciate help on this from developers with more experience optimizing Rust code. In the meantime, we believe the modest executable size increase, in cases where it does happen, is a reasonable tradeoff for the dramatic reduction in memory usage.
We haven’t yet measured the effects of this optimization on speed. It stands to reason that building a property table, determining whether that table can be shared with other nodes, then looking up properties through that table, as opposed to just getting and setting fields in a simple struct, isn’t free. But the synthetic benchmarks we might devise at this point seem unlikely to reflect real-world performance. We look forward to feedback from users as they measure AccessKit’s performance in real-world applications, particularly applications with complex, data-rich user interfaces.
Finally, we’d like to explore what we might achieve by allocating an arena for each AccessKit node, or perhaps storing all nodes and their classes in a custom heap. Maybe we can reduce the overhead that comes from multiple heap allocations (and deallocations), as well as the overhead of using absolute pointers for everything, particularly on 64-bit platforms. We recently came across a compact garbage-collected heap in C++</a> that is particularly impressive. However, we don’t want to rush into adding unsafe code in the core of AccessKit, and we have higher-priority things that we need to work on first, so this potential optimization will remain a longer-term project.
Conclusion</h2>
Now that the memory optimization described in this post is complete, we’re confident that the design of AccessKit is fundamentally sound. We can always optimize more, and we may still add and remove some properties before freezing the AccessKit API, but we don’t anticipate any more large-scale breaking changes that will affect GUI toolkits and applications. That means this is also a good time to begin work on bindings for other languages; after all, our goals for AccessKit extend beyond the Rust ecosystem.
We’d like to again thank Federico Mena Quintero for the article that inspired our optimization work, as well as the FlatBuffers and V8 projects for indirectly inspiring our implementation of shared node classes. And, of course, we must thank everyone who has contributed to the Rust language, for empowering us to fearlessly optimize without compromising safety. We all build on each other’s work, and now that AccessKit’s design is maturing, we hope that many more developers will build on our own work to make their applications accessible.

Looking back; looking forward

2023-01-01T19:59:24+00:00

2022 has been a great year for AccessKit, thanks to continued funding from the Google Fonts team. As we celebrate the new year, I’d like to look back on what we’ve accomplished in 2022, and look forward to the work that still needs to be done to fulfill AccessKit’s potential.

Progress in 2022</h2>
At this time last year, AccessKit was a proof-of-concept on Windows only, with support for only a handful of user interface control types, not including text edit controls, which are the most notoriously difficult type of control to make accessible. Now, with platform adapters for both Windows and macOS, robust support for both single-line and multi-line text edit controls, and enhanced support for other control types including sliders and steppers, AccessKit is ready to start making real applications accessible.
As evidence that AccessKit is ready for production, it is now integrated into egui</a>, a popular Rust GUI toolkit. This makes egui, as far as we know, both the first pure-Rust GUI toolkit and the first immediate-mode GUI to implement platform accessibility APIs. AccessKit is enabled by default in eframe</a>, the official ready-to-use application framework on top of egui. Other egui integrations, such as bevy_egui</a>, will need to integrate AccessKit as well. Check out a video demonstration of AccessKit in an Egui application on Windows. </iframe>
I’d like to thank Emil Ernerfeldt</a>, the primary developer of egui, for enthusiastically working with me to complete this integration. Also, thanks again to the Google Fonts team, for funding not only my work on AccessKit itself, but the integration in egui.
Speaking of that Google team, now that they’ve established a clear direction for their future research by starting the xilem</a> and [glazier](https://github.com/linebender/glazier] projects, we’ve started the work on integrating AccessKit into these projects as well. AccessKit is already integrated into glazier, an abstraction over the various platform windowing APIs, similar to winit</a> but with an emphasis on deep integration with the platform for rich desktop applications. Unlike AccessKit’s winit adapter</a>, a bolted-on solution which requires some inelegant workarounds, the integration in glazier is direct and clean. We look forward to an equally clean, first-class integration of AccessKit in xilem, the new GUI toolkit being built on top of glazier.
Work on integrating AccessKit in other GUI toolkits is also underway. In particular, Vizia</a> will soon have AccessKit support. This integration is being done primarily by Vizia’s main developer, George Atkinson</a>.
We even have a proof-of-concept integration of AccessKit into a non-Rust environment, specifically, the Unity game engine. You can check out this integration in action in our modified version of an existing open-source Unity-based game</a>. I presented this demo in a talk at the NarraScope 2022 online conference</a>, my first conference talk featuring AccessKit. As a result of this talk, at least one independent game developer is now interested in making their games accessible with AccessKit. This integration was particularly challenging because we have had no cooperation, at least so far, from the Unity developers. As a result, you may notice that in the demo, the window isn’t full-screen as it was in the original game, and it flickered once on startup. However, we believe we now have a solution to this technical problem. Thanks to Arnold Loubriat</a>, my main volunteer collaborator on AccessKit for more than a year, for doing most of the hard work on this project.

Looking forward</h2>
As happy as I am with AccessKit’s progress in the past year, we’re not done yet. There’s much more work to do before AccessKit reaches its potential as universal, reusable accessibility infrastructure.
First, before it can be used to make the full range of desktop and mobile applications accessible, AccessKit needs to support more types of controls, including list views, drop-down lists, combo boxes, menus, scrollable areas, tabs, tables, and tree views. In some cases, this will be as simple (though tedious) as mapping specific properties in platform accessibility APIs to their counterparts in AccessKit. AccessKit itself already has many, if not all, of these properties, thanks to the schema that we adapted from Chromium’s cross-platform accessibility abstraction early in the project. But in some cases, we will probably need to go through some trial and error to successfully account for the different conventions around concepts such as focus, selection, and popups on different platforms.
While our support for text edit controls is robust within its current intended scope, it’s not yet comprehensive. In particular, AccessKit doesn’t yet expose formatting information, such as font name and size, style attributes such as bold, italic, and underline, colors, headings, and lists. Exposing some of these attributes will be straightforward, but we expect that the moderately more difficult part will be supporting text with a mix of different values for these attributes. We’d eventually also like to support hypertext, including embedded links and images, but this is a much more difficult project, which we will most likely postpone for a year or more.
As AccessKit is used to make more complex and data-rich user interfaces accessible, we may find it necessary to spend time on optimization. AccessKit’s current use of a single large data structure to represent all types of accessible UI elements is a known potential weakness in the design. However, we want to take our time to explore alternatives, and collect more data on the performance of AccessKit in real applications, before we commit to a different design. If and when we do make a major design change, we will do everything we can to ease the migration for existing users.
In the shorter term, we are eager to expand AccessKit to more platforms. As I write this, the platform adapter for GNOME and other Unix-based desktop environments that use the AT-SPI protocol, developed primarily by Arnold Loubriat</a>, is almost ready for production. You can check out its current state in the pull request</a>. We also plan to develop adapters for Android and iOS. Finally, and probably most difficult of all, we want to develop an adapter for the web platform, for applications and toolkits such as egui that use the HTML canvas element. The prioritization of these platforms is entirely dependent on future funding and volunteer contributions.
We are also excited about the potential of AccessKit to be reusable across programming languages, beyond Rust. We are already working on polishing our proof-of-concept Unity integration into a truly ready-to-use plugin. We also plan to integrate AccessKit into at least one project based on the Java Virtual Machine, which we’ll announce when it’s ready. Support for other languages is, again, dependent on future funding and volunteer contributions.
As AccessKit becomes more widely used, thorough and approachable documentation will become more important. We realize that most GUI developers don’t deeply understand accessibility as we do, and we need to make that understanding more widely available. Watch this site for more documentation, including introductory documentation for beginners, in the coming months.

Conclusion</h2>
We believe AccessKit is gaining momentum, and we can’t wait to find out where it will go in the coming year. To keep this momentum going, we need funding to continue our work. If you’d like to directly fund my work on AccessKit, checkout my new GitHub Sponsors profile</a>. Of course, as an open-source project, AccessKit is also open to volunteer contributions, especially in GUI toolkit integrations, programming language support, and documentation. Let’s work together to make many more applications accessible in 2023!

Introducing AccessKit: a New Open Source Project Designed to Make more Apps Accessible

2021-09-26T17:31:46+00:00

Originally published on the Pneuma Solutions blog</a>

There’s just about nothing more frustrating for blind people than trying to use an app, only to find that the app is inaccessible. Maybe your screen reader can’t read a key part of the app, or maybe it can’t read anything at all. When apps are developed using standard buttons, edit boxes, list boxes, tables, or other user interface elements, it’s not hard to make them accessible. But some apps are developed using a non-standard user interface toolkit, and these toolkits are completely inaccessible. Implementing accessibility from scratch in a user interface toolkit is difficult and time-consuming, so it usually doesn’t get done, except in the top few user interface toolkits with heavy corporate backing. Sometimes it’s truly necessary for an app developer to use a custom user interface toolkit, and sometimes it’s not, but regardless, we end up with inaccessible apps.

This problem is especially urgent for apps that are critical to particular jobs. It’s not uncommon for a blind person to be unable to get a particular job because the job requires the use of an app that’s inaccessible. These are often niche apps for which the usual economic incentive to implement accessibility, namely selling to the government and educational sectors, doesn’t apply. So we urgently need a solution that breaks down as many of the barriers as possible to making the long tail of apps accessible.

As with the work Pneuma Solutions is doing on remediation of documents and meeting content, machine learning can help with this problem. Apple is leading the way in this area with the Screen Recognition feature built into iOS, and the results so far are promising. However, this technology isn’t yet available on all computers and devices, and the results are often not satisfactory. We can’t wait for this solution to mature and be more widely adopted; we need another approach that will be practical in the shorter term. Also, many developers of both apps and user interface toolkits are willing to make their software accessible, if only it weren’t so difficult and time-consuming. These developers are already willing to meet us halfway; now we need to do the same.

That’s where my new open-source project, AccessKit</a>, comes in. The goal of AccessKit is to provide shared infrastructure for making apps accessible, across as many platforms and programming languages as possible. With AccessKit, a developer working with multiple platforms won’t have to implement accessibility for each platform from scratch. Another goal of AccessKit is to be better documented and easier to use correctly than the existing platform-specific accessibility APIs such as UI Automation for Windows or Cocoa accessibility for Apple platforms. After all, a broken accessibility implementation can be almost as frustrating as no accessibility at all.

AccessKit is still early in its design and development, but it’s already attracting interest from other developers, including code contributions from one other developer. And now I have the great privilege of receiving funding from Google to work part-time on this project, starting with the Windows implementation. I look forward to usable results by the end of this year.

I’m sure any developers reading this will want to know more about how AccessKit will work. The short version is that AccessKit will provide a cross-platform accessibility abstraction that’s heavily inspired by the Chromium browser engine. This abstraction is based on serializable data structures, thus minimizing the overhead of interfacing between programming languages. AccessKit will be implemented primarily in the Rust programming language, which provides a unique combination of reliability and efficiency. However, AccessKit will be usable from a variety of programming languages. More details are available at the AccessKit GitHub repository</a>.

As an open-source project, AccessKit needs participation from the developer community to be successful and sustainable. In particular, I’d appreciate contributions from developers that are proficient in the Rust programming language. The overall design of AccessKit is still fluid, so it’s too early for me to delegate much implementation work. What I do need at this point is peer review, especially from developers that are more experienced with the Rust language than I am. If you’re interested, please join us on GitHub</a>.

In closing, a big reason why I left Microsoft to cofound Pneuma Solutions is that I want to have the freedom to work on accessibility-related projects that I believe will have a great positive impact for our community, beyond a single platform. I’m still enthusiastic about the work we’re doing at Pneuma Solutions, and that work will continue. With AccessKit, I now have the opportunity to solve a problem that has weighed heavily on my mind for many years. I look forward to working with the software development community to make many more apps accessible.