0 votesclick to vote up

Ted Gould: Searching Menus Published Jan 24 2012 via RSS

planetUbuntu

One of the neatest parts about starting to get more application data into the open is that we can start to use it in interesting ways. I'm happy to talk about a new way that we're using the menu data: the HUD. The idea behind the HUD is that you can quickly find functionality in an application without having to know the menu structure. But how does it do it? How can you make it better?

Getting the data

We're using the same Dbusmenu data that is currently exported to the global menu, just remixing it for search. We are searching through the labels in the menu items which gives us already localized data straight from the application. This means that it should work for the language that the application is in. In the future we hope to use information like accessibility data as well as any tooltips that might be attached to a menuitem (though we don't show tooltips in the global menu).

Any application that works with the window menus today should also work with HUD out of the box.

Matching the label

To match the label we're basically using an implementation of the Levenshtein distance with a few additions. What this allows us to do is rank possible solutions in a relevancy order, and present some solutions that might occur via "fat finger" or other similar type errors. But, this also means that there is some fuzzy algorithms involved in the matching which will have to be tuned.

We expect to tune them over the next few releases, and to do that we have a set of test cases that we're using for the tuning. The problem with those test cases? They're only in the languages I speak. You probably speak in more/different/better languages than I do, please feel free to propose merges that extend this test suite so that as we continue to tune the search algorithms we don't leave any language behind.

Remembering Favorites

One of the additions that we add to the distance calculation is an offset based on which entries you've used most recently. Your favorite functionality in the application. Quite simply we're storing a list of items you've used over the last thirty days and a timestamp of when you used them. This database is simple but it can be fun to look into for the curious and I wanted to talk a bit about a couple of the tools that you can use to see the data.

$ hud-list-applications

This will list all the applications that have data on them in your HUD usage database. They are identified by the path to their desktop file as determined by BAMF. You can then look at the menu items used in a specific application:

$ hud-dump-application /usr/share/applications/inkscape.desktop 

This shows the individual items that you've used, and the number of times that you've used them. If you want to inspect the exact file tracking the data it is available at:

~/.cache/indicator-appmenu/hud-usage-log.sqlite

While talking about various tools to work with HUD I thought I'd also mention that you can also, just for fun, work with HUD from the command line using the command line tool:

$ hud-cli

Application initial bias

Application designers have always had a problem figuring out how to promote specific functionality that is commonly used to the forefront, while still making the rest of the functionality easily available. The most recent ways that they've done this is with toolbars and ribbon style. You can't adjust the positioning even when you know that the particular toolbar isn't best for the user because it will mess up the user's spacial memory. HUD sidesteps this issue by providing all the options, just promoting certain ones based on usage. They're all in the same place (the HUD) but with always improving ordering.

What happens on first usage of the application? At that point we don't have any way to know what the user wants to do, we we've provide a way for the application designer to provide the most likely items for general users. Effectively, this is the HUD's version of the default toolbar setup in an application; though it automatically decays and adjusts to the user's usage pattern.

The files that control this initial bias are very simple and there is an example in the test suite. Basically they have the various menu items along with a value that describes how to preload the usage database. A '5' there would mean that 5 entries are added to the usage database for that item on the first time that application is used; one for today and each of the four days previous. In this way, as values drop off by being too old, there isn't a step function in how the item is ranked, it just slowly drops down in priority. Application designers should start to think about how they would rank the menu items in their application, and start getting these integrated into either the releases or the packages of those applications so that users have a good first experience with their application.

Development notes

The code for the HUD lives in the indicator-appmenu repository. Currently it exists on a branch that needs to be reviewed before merging, but that shouldn't be for long. I expect it to get merged to trunk in the next couple of weeks.

After that the biggest change will be integration with indicator-appmenu. It was originally implemented as it's own service to make development more agile, but it clearly shares a large amount of data with the global menu and there's no reason to have two repositories in memory of the same data. It also needs to synchronize heavily with the application menu and BAMF, which is also already in indicator-appmenu. Thanks to the magic of DBus no one should notice the change in processes as the names and objects will migrate over to the new process.

As this is more of a first prototype there are also some missing features that need to be added. The first of those is to simply improve the matching. We also need to get better descriptions from application indicators, today we're using their accessibility description (you set those, right?) but that typically has too much information. Lastly, we need to integrate better with applications that expect the about-to-show signal for their menus. This includes XUL applications and some Qt ones, so it's an important feature for making the HUD usable for everyone.

Merges and bugs should be directed towards the indicator-appmenu project and also make sure you've signed the Canonical Contributor Agreement for any code contributed.

Comments: Identi.ca | Twitter