Update #2: The need for Wayland

We left off previously in that I was about to write code to load xkbcommon keymaps into the X server, that soon proved to be not a very good idea because the keymap type on xkbcommon is an opaque structure, this means the authors want to be able to change its organization at their will without breaking someone else’s code, so the only way to manipulate it is through their API. We could technically bypass this but I think the maintainer burden added would be too much, and it would make difficult to update to newer versions of the library. Another way to accomplish this is as a patch to the tree to libxkbcommon, which sounds fine, but as I said before I don’t want to bother other people because this is sort of an experiment, also it would add code there that will be useless once we move to Wayland, and will also make people less willing to move out from X.

After putting this to the side, it occurred to me that I could use the .vapi file I already had for libxkbcommon to manually translate keypresses into their unicode characters and then send them to the applications, I just needed to know where to “tap” the event stream going from a keypress up to when it was sent to an application. This seamed a reasonable idea because I would have to do it anyway to implement ibus (or at least it seemed like the obvious way to do so). Turns out this stream can’t be tapped because it doesn’t even go through Gala, and it also not how ibus is implemented (I guess precisely due to impossibility of intercepting keypress events). So how does this actually work? I’ll try to explain briefly next.

The first warning sign that told me this wasn’t how things worked was how ibus is implemented in Gnome’s Shell, there is a comment in their code that explains this, it says events reach the X server, then Mutter intercepts all of them, lets Clutter handle them through the clutter_x11_handle_event() function which will send them to the corresponding application (Clutter Actor), then the application receives an event that hasn’t been translated yet, notices an input method is enabled and sends it to the d-bus daemon that will push the resulting translation into a gdk event stream that the shell is listening into (through Mutter). Mutter then, assumes every event here comes from ibus, so it sends it back to Clutter so it reaches the corresponding Clutter Actor again.

At this point I decided to verify all this by myself, because if we are filtering all X events through Mutter and sending them to the corresponding Clutter Actor, why can’t we translate them in Mutter and then send them just once without this strange round tripping to the application?. I found 3 points at which I could intercept this event stream to see what was happening.

I knew there was a way of intercepting Clutter events, but nothing guaranteed that the function clutter_x1_handle_event() handled X events by translating them into Clutter events. After some reading of Clutter’s code I verified this was actually true, X events are translated and pushed into Clutter’s event queue. So I thought adding a Clutter event filter would show all keypresses before them being delivered to the application, if this were true I would just need to translate the keycode as I wanted and let the event reach is normal destination. After doing this I found out no CLUTTER_KEYPRESS events where going through (except for the tab and alt key while alt tabbing), what was going on here? at some point we were loosing events, but they were reaching the application because everything worked normally, so was there another event queue somewhere handling this?.

Then I found out another filter on ClutterX11 this one received an X event and a Clutter event, my guess was that this function was called when translating a X event into a Clutter event, so if my function filled the Clutter event appropriately and returned TRANSLATED then I would be doing the translation and continue sending the Clutter event instead of the X event. So I tried listening for X keypress events. Sadly enough, there were still no keypress events to be seen. I could only see some events of type GenericEvent, this seemed like another dead end. After some research I found out that listening for X key events was very naive, now a days to support weird input devices like Wacom tablets we use XInput2 that uses GenericEvents to send bigger events than what X supports. So I needed to decode these GenericEvents (there seemed to be a lot of these, just what I would expect if every keypress and keyrelease was being sent) to know if there were some XI_keypress events going through there. Doing this wasn’t a trivial task because XInput2 does not have much documentation and there are no Vala bindings for this. After creating a .vapi file for XI2.h I still had some trouble because I needed to cast a struct pointer to another struct pointer which is not documented on the manual vapi file tutorial (turns out to be a simple pointer cast on vala). After all this I could finally check if my keypresses were going through here, and guess what… they weren’t, only the tab and alt key I had seen before were going through here.

The third point at which I could intercept events was overriding the method Meta.Plugin.xevent_filter(), I could see exactly where it was being called from Mutter and I am pretty confident all X events that reach Mutter go through this before anything happens, in fact handling the event here would make it never to go through Clutter at all. Doing things here had a problem though, I would have to find the correct Clutter Actor that should get the event by myself, but nevertheless if keypresses were going through here it would be something I could work with. So I copied my code from the ClutterX11 filter here, and if stuff wasn’t disappointing enough already… well, no interesting keypress events were to be seen here either. So again, what’s going on here?.

My conclusion of all this is, key press events don’t ever reach Mutter and consequently neither Gala. This means the comment on Gnome Shell’s code is quite misleading or maybe I misunderstood it from the beginning. Events reach the application (X client) directly then if an input method is enabled these are sent to the window manager to be displayed on the input method window (the bubble). Keycode translation does not happen in the window manager but in the application itself, mostly hidden into Gtk so that application developers don’t have to do this explicitly.

It seems to me like i18n is a hard problem that no one can claim to have solved completely without knowing every language that a unicode string may represent. This makes people unwilling to commit to an API that will be frozen forever. Instead what I feel is that projects just pass the ball around to other projects without no one trying to actually solve it directly. Wayland right now only has basic keycode translation with libxkbcommon just like X did with xkb, and leaves everything else to the applications, but then Gtk tries to make easy to create internationalized applications by hiding all this below some API. Let’s hope wayland-im will improve things.

Where does this leave us regarding Gala?, well I’ve came to the conclusion that what I wanted to do (load arbitrary xkb files) can’t be done in a non hacky way, because currently as an X window manager we don’t have full control of the events going through, this will only be true once we move to Wayland (and even then we may have to deal with the remaining abstractions (Mutter and Clutter). Still I think users can’t wait for Wayland right now so I’ve decided to move away from my original idea and fix this temporarily for the next elementary release just the way Gnome does it. I will then start looking into  moving Gala to Wayland because I think this is what we really need to take the X server out of the middle and let us handle events by ourselves. I also think we need a better infrastructure that allows developers to test things quickly before actually committing and releasing to everyone else, without having to deal with code that is outside in abstract dependencies, so we’ll see how that turns out later.

 

Update #1: Too many layers

I’ve been digging, trying to implement the loading of xkb files on Gala, turns out that stuff has been abstracted several layers deep, which makes it difficult to experiment with this stuff without bothering people upstream and having to make changes and coordinate with several different projects (and teams). Because what I’ve been trying to do is rather experimental, I don’t want to just create patches for a lot of projects and argue they should merge my code because I think it will be good (I don’t even know that myself), instead what I’ve been trying to do is implement stuff and see how well it works out and if I really think it’s useful for others.  I’m also looking to add the least amount of code to Gala, so rewriting the whole keyboard handler seems like the most extreme solution, and I am trying not to come to this.

I will try to explain briefly all the layers involved with keyboard handling on Gala, before in place of this I had about 3 paragraphs trying to do so but they were mostly just ranting about the excessive abstractedness of the whole thing, but in the end I don’t think that’s relevant for the discussion and I don’t have a global understanding of all projects to question decisions made by others, so I will just point out important details I’ve seen I need to get things done. For this I will just show a diagram of how systems interact (it’s important to note that this relationships are only related to keyboard handling I have no idea how these systems interact for example for graphics).

keyboard_handling_deps
Dependencies of keyboard handling

There are 5 projects here, arrows represent where some project calls a library from someone else. Gala is written in Vala which means calling C code from here is not a trivial task. Mutter and Clutter are written in C but using GObject and Glib, this both ease calling it from Vala code but because they are in the end just C code they can easily call C libraries like XLib, or libxkbcommon. This makes them our most accessible interfaces to lower level interfaces from Gala. Also a sidenote about Mutter is that it aims to be both compatible with Wayland and X11 so it has two backends to support this, but it also means we can’t expect Wayland specific functionality to be provided from it.

I think Gala uses Mutter’s X11 backend but I don’t know how can I test this to be sure. The problem of this is that the API provided by Mutter to set the keyboard layout only uses the RMVLO description which seems to be a legacy interface that comes from the fact that this is what setxkbmap does and was the easiest to copy, as opposed to what xkbcomp does which implied understanding the xkb description specification and how to upload it to the X server (both approaches in the end invoke the xkb compiler every time a layout switch happens which is one of the issues I’m trying to solve). Because Mutter cares about providing functionality available only to X11 and Wayland it’s unlikely that an API that uses libxkbcommon to load xkb files will be provided.

Although Mutter with it’s native backend does use libxkbcommon to change the keyboard layouts, when trying to call the function that does this from Gala I stumbled upon several issues, the most important one seems to be that X11 grabs input devices and does not let Clutter listen to events, an assertion on the library fails saying “Clutter is not the device manager”. Also, this interface is not part of the API stability guarantees which on one hand could bring some problems in the future, but even more frustratingly: makes them completely unusable from Vala code, because as it turns out outputting #define symbols through Vala is impossible (we just need to add the line #define CLUTTER_ENABLE_COMPOSITOR_API ) and even worse do it before the #include directive for the library. So at this point using Clutter’s keyboard handling on the native backend becomes unfeasible.

So, what do we do now? Well I’ve come to the conclusion that right now the simplest approach is to make functionality to load libxkbcomon’s keymap format into X11 directly on Gala. Currently for testing what I did before I created a .vapi file by hand for libxkbcommon so I can now use it from Gala, I haven’t seen bindings for XKBLib so I may have to do this next, we’ll see how that all goes. If I’m successful with this then maybe it would help Mutter to include an API for this, so I could upstream some of it if it would actually work for someone else.

As a minor side rant, I have to say that I wished Gala would have a more monolithic design and would not impose OOP through Vala in the way it does. I mean, the most complex, big and by far successful free software project is the Linux Kernel and it’s also a huge monolithic piece of code, I think segregating functionality around several projects adds a lot of overhead for people trying to help. It’s true that the codebase would grow substantially, but I think having code that actually does something as opposed to glue code would ease fixing problems and trying out new things.