Just Say It!

While I love typing on small on screen keyboards on my phone, it is much easier to just talk. When we did the HUD we added speech recognition there, and it processed the audio on the device giving the great experience of controlling your phone with your voice. And that worked well with the limited command set exported by the application, but to do generic voice, today, that requires more processing power than a phone can reasonably provide. Which made me pretty excited to find out about HP's IDOL on Demand service.

I made a small application for Ubuntu Phone that records the audio you speak at it, and sends it up to the HP IDOL on Demand service. The HP service then does the speech recognition on it and returns the text back to us. Once I have the text (with help from Ken VanDine) I set it up to use Content Hub to export the text to any other application that can receive it. This way you can use speech recognition to write your Telegram notes, without Telegram having to know anything about speech at all.

The application is called Just Say It! and is in the Ubuntu App Store right now. It isn't beautiful, but definitely shows what can be done with this type of technology today. I hope to make it prettier and add additional features in the future. If you'd like to see how I did it you can look at the source.

As an aside: I can't get any of the non-English languages to work. This could be because I'm not a native speaker of those languages. If people could try them I'd love to know if they're useful.


posted on Sun, 28 Jun 2015 at 23:29 | permanent link

Role change: Now snappier

Happy to announce that I'm changing roles at Canonical, moving down the stack to join the Snappy team. It is in some ways a formalization of my most recent work which has been more on application lifecycle and containment than higher level stuff like indicators and other user services. I'll be working on the core snappy team to ensuring that snappy works for a wide variety of use cases, from small sensors embedded in your world to phones to services running in the cloud. For me Snappy formalizes a lot of trends that we're seeing all over computing today so I'm excited to get more involved with it.

To kick things off I'll be working on making Snaps easier to build and maintain using the native dependency systems that exist already for most languages. The beautiful part about bundling is that we no longer have to force our dependency system on others, they can choose what works best for them. But, we still need to make integrating with it easy.

New adventures bringing new challenges are where I like to roam. I'll still be around though, and might even contribute a patch or two to some of my old haunts.

posted on Tue, 09 Jun 2015 at 23:54 | permanent link

Tracking Usage

One of the long standing goals of Unity has been to provide an application focused presentation of the desktop. Under X11 this proves tricky as anyone can connect into X and doesn't necessarily have to give information on what applications they're associated with. So we wrote BAMF, which does a pretty good job of matching windows to applications, but it could never be perfect because there simply wasn't enough information available. When we started to rethink the world assuming a non-X11 display server we knew there was one thing we really wanted, to never ever have something like BAMF again.

This meant designing, from startup to shutdown, a complete tracking of an application before it started creating windows in the display server. We then were able to use the same mechanisms to create a consistent and secure environment for the applications. This is both good for developers and users as their applications start in a predictable way each and every time it's started. And we also setup the per-application AppArmor confinement that the application lives in.

Enough backstory, what's really important to this blog post is that we also get an event when an application starts and stops which is a reliable event. So I wrote a little tool that takes those events out of the log and presents them as usage data. It is cleverly called:

$ ubuntu-app-usage

And it presents a list of all the applications that you've used on the system along with how long you've used them. How long do you spend messing around on the web? Now you know. You're welcome.

It's not perfect in that it uses all the time that you've used the device, it'd be nice to query the last week or the last year to see that data as well. Perhaps even a percentage of time. I might add those little things in the future, if you're interested you can beat me too it.

posted on Fri, 14 Nov 2014 at 13:35 | permanent link

HUD for the command line

HUD shown over terminal app with commands visible

Most expert users know how powerful the command line is on their Ubuntu system, but one of the common criticisms of it is that the commands themselves are hard to discover and remember the exact syntax for. To help a little bit with this I've created a small patch to the Ubuntu Terminal which adds entries into the HUD so that they can be searched by how people might think of the feature. Hopefully this will provide a way to introduce people to the command line, and provide experienced users with some commands that they might have not known about on their Ubuntu Phone. Let's look at one of the commands I added:

UnityActions.Action {
  text: i18n.tr("Networking Status")
  keywords: i18n.tr("Wireless;Ethernet;Access Points")
  onTriggered: ksession.sendText("\x03\nnm-tool\n")
}

This command quite simply prints out the status of the networking on the device. But some folks probably don't think of it as networking, they just want to search for the wireless status. By using the HUD keywords feature we're able to add a list of other possible search strings for the command. Now someone can type wireless status into the HUD and figure out the command that they need. This is a powerful way to discover new functionality. Plus (and this is really important) these can all be translated into their local language.

It is tradition in my family to spend this weekend looking for brightly colored eggs that have been hidden. If you update your terminal application I hope you'll be able to enjoy the same tradition this weekend.

posted on Fri, 18 Apr 2014 at 12:46 | permanent link

Measuring Upstart App Launch using LTTng

One of the goals of this cycle is to decrease application startup times on the Ubuntu phone images. Part of my work there was to look at the time taken by Upstart App Launch in initializing the environment for the application. One of the tricky parts of measuring the performance of initialization is that it contains several small utilities and scripts that span multiple Upstart jobs. It's hard to get a probe inside the system to determine what is going on without seriously disrupting it.

For measuring everything together I decided to use LTTng which loads a small kernel module that records tracepoints submitted by userspace programs using the userspace tracer library. This works really well for Upstart App Launch because we can add tracepoints to each tool, and see the aggregate results.

Adding the tracepoints was pretty straight forward (even though it was my first time doing it). Then I used Thomas Voß's DBus to LTTng bridge, though I had to add signal support.

To setup your Ubuntu Touch device to get some results you'll need to make the image writable and add a couple of packages:

$ sudo touch /userdata/.writable_image
$ sudo reboot
# Let it reboot
$ sudo apt-get update
$ sudo apt-get install lttng-modules-dkms lttng-tools
$ sudo reboot
# Rebooting again, shouldn't need to, but eh, let's be sure

You then need to setup the Upstart App Launch environment variable to get it registering with LTTng:

$ initctl set-env --global LTTNG_UST_REGISTER_TIMEOUT=-1

Then you need to setup a LTTng session to run your test. (NOTE: this configuration allows all events through, but you can easily add event filters if that makes sense for your task)

$ lttng create browser-start
$ lttng enable-event -u -a
$ lttng start

To get the Upstart starting events from DBus into LTTng:

$ dbus-monitor --profile sender=com.ubuntu.Upstart,member=EventEmitted,arg0=starting | ./dbus_monitor_lttng_bridge 

And at last we can run our test, in this case starting the webbrowser once from not running and once to change URLs:

$ url-dispatcher http://ubuntu.com
# wait for start
$ url-dispatcher http://canonical.com

And then shut things down:

$ lttng stop
$ lttng destroy browser-start

This then creates a set of traces in your home directory. I pulled them over to my laptop to look at them, thougth you could analyze them on the device. For complex traces there are more complex tools that are available, but for what I needed babletrace was enough. All of this contributed to a set of results that we are no using to optimize upstart-app-launch to make applications start faster!

posted on Mon, 16 Dec 2013 at 13:17 | permanent link

Emergent Complexity

When looking at Upstart it comes off as very simple. Almost too much so. It's just events. And jobs that take those events. And not much else. You're so used to configuring every last detail of a system to ensure that it's tightly tuned for every possible scenario, certainly you need more tools than just events! When you let go of that preconception and really start to understand it, you realize that events are not only enough, but they're exactly what you need.

One might have thought – as at first I certainly did – that if the rules for a program were simple then this would mean that its behavior must also be correspondingly simple. For our everyday experience in building things tends to give us the intuition that creating complexity is somehow difficult, and requires rules or plans that are themselves complex. But the pivotal discovery that I made some eighteen years ago is that in the world of programs such intuition is not even close to correct. — Stephen Wolfram, A New Kind of Science.

And so we take these simple jobs that we've built and we start to build a system. They each just wait on a set of events, and some emit events as they go along, and the system starts to take form. But where are our initialization phases and guarantees and complex dependencies? (writing code to solve them is fun!) They still exist, but as an emergent behavior of the system. Let's look at Graphviz diagram of the Ubuntu Saucy system init:


(Graphviz | Full SVG)

Without zooming in and just looking at the shapes that emerge you start to see a natural grouping of the jobs. And there are stages of the boot. There are types of jobs that are gathered together. There'd be even more if initctl2dot could break down the runlevel job into its various values. What we see is a the complex boot of a modern operating system broken down into pieces for analysis. What we see is a model of the behavior of the system, but that model is the only place that the complexity actually exists, it becomes the emergent behavior of the system. And that, that is why Upstart can be so simple and yet be powerful enough to boot a modern Linux system.

posted on Sun, 04 Aug 2013 at 00:06 | permanent link

Application-focused Control

I love BAMF, may it die a peaceful death. BAMF was always stuck trying to solve an unsolvable problem, trying to recreate information and associations that had been lost through the X11 protocol and general fuzziness about what should happen. BAMF then had to handle a wide variety of corner cases and try to bring things back into a situation of sanity. And I loved BAMF because I didn't have to do that myself, BAMF did it, and all of its clients just got sanity. We knew when starting another display system we didn't want another BAMF.

With Mir there is a closer tie between applications as a whole and their windows. When an application asks for a session it specifies who it is, then Unity can make sure it understands who it is, and gets a chance to veto the connection. This means that Unity can check on the status of who the app says it is before it gets any windows and can track that directly throughout the application session. To do this we're using what we call the "Application ID", which for most apps you have today is the name of their desktop file (e.g., "inkscape", "gedit").

Let's look at an overly verbose message sequence diagram to see how this works. Note: all of the columns on here aren't separate processes, the diagram is made to explain this idea, not to represent the system architecture.


(svg | msc)

What we can see here is that because Upstart is doing process tracking, and making sure it knows the state of the application, Unity have have assurances of the application's name and existence. It then can work with Mir to block applications that do have proper configurations and can't be matched well. 100% matching, by design.

posted on Wed, 24 Jul 2013 at 11:55 | permanent link

Application Centric User Experience

One of the design goals of Unity was to have an Application-centric user experience. Components like the Launcher consolidate all of the windows into a single icon instead of a set like the GNOME 2 panel. Nothing else in Ubuntu thinks about applications in this way making it a difficult user experience to create. X11 worries about windows. DBus worries about connections. The kernel focuses on PIDs. None of these were focused on applications, just parts of applications. We created the BAMF Application Matching Framework (BAMF) to try and consolidate these, and while it has done a heroic job, its task is simply impossible. We need to push this concept lower into the stack.

First we looked at the display server and started thinking about how it could be more application centric. That effort resulted in Mir. Mir gets connections from applications and manages their associated windows together. They can have multiple windows, but they always get tracked back to the application that created them. Unity can then always associate and manage them together, as an application, without any trickery.

Application confinement also provides another piece of this puzzle. It creates a unified set of security policies for the application independent of how many submodules or processes exist for it. One cache directory, set of settings and policies follow the application around. Apparmor provides a consistent and flexible way of managing the policies along with the security that we need to keep users safe.

To start to look at the process management aspect of this I started talking to James Hunt about using Upstart, our process manager in Ubuntu. Working together we came up with a small little upstart user session job that can start and stop applications, and also track them. I've pushed the first versions of that to a test repository in Launchpad. What this script provides is the simple semantics of doing:

$ start application APP_ID=gedit
$ stop application APP_ID=gedit
to manage the application. Of course, the application lifecycle is also important, but Upstart provides us an guaranteed way of making sure the application stops at the end of the session.

Upstart can also help us to guarantee application uniqueness. If you try and start an application twice you get this:

$ start application APP_ID=inkscape
application (inkscape) start/running, process 30878
$ start application APP_ID=inkscape
start: Job is already running: application (inkscape)
This way we can ensure that a single icon on the launcher associates to a set of processes, managed by the process manager itself. In the past libraries like libunique have accomplished this using DBus name registration. Which, for the most part, works. Using DBus registration relies on well behaving applications, which basically guarantee their own uniqueness. By using Upstart we can have misbehaving applications, and still guarantee their uniqueness for the Unity to show the user.

We're just getting started on getting this setup and working. The schedule isn't yet final for vUDS next week, but I imagine we'll get a session for it. Come and join in and help us define this feature if it interests you.

posted on Tue, 07 May 2013 at 08:16 | permanent link

Bustle Boot Log Under Upstart

For a while I've had a little project for debugging the desktop. Basically it starts tracking all of the DBus events on the user session startup so that you can figure out what's going on. This is especially an issue for indicators, where they're started at login, and sometimes it can be hard to track what is happening.

Previously it was pretty hard to inject into the startup of the session. Getting in the middle of building a very long command line was risky and pretty fragile. Not proud of what I had to do. Now that I have Upstart user session running, I took the opportunity port this debugging script over to Upstart.

Now I have this one simple configuration file that can be dropped in /usr/share/upstart/sessions and gets started immediately after dbus:

description "Bustle Boot Log"
author "Ted Gould "

start on started dbus
stop on desktop-end

script
	rm -f ~/.cache/bustle-boot-log/boot-log.bustle
	mkdir -p ~/.cache/bustle-boot-log/
	timeout -s INT 30 bustle-pcap --session ~/.cache/bustle-boot-log/boot-log.bustle &
end script

The beauty of this is that I can inject this small little script in, and have Upstart figure out all the startup mess. I also have minimal impact on the natural desktop boot which is critcial for testing. Simple things to make debugging easier.

posted on Tue, 30 Apr 2013 at 14:59 | permanent link

Indicators in Upstart

I've started to prototype and lay the foundations for the indicators to use the Upstart User Sessions. It's an exciting change to our desktop foundations, and while it's still very fresh, I think it's important to start understanding what it can do for us. For right now you're going to need a patch to Unity and a patch to indicator-network to even get anything working, not recommended for trying at home.

Previously for indicators the way that they've worked is that a small loadable module was loaded by the panel service that had indicator specific UI in it. That plugin also took care of the responsibility to restart the indicator backend, respawning it if it crashed. While this works and it has created a robust desktop (most people don't notice when their indicator backends crash) it has had some downsides. For one, it makes it difficult to build and test new backends as you pretty much have to restart Unity to stop the previous service from getting respawned. Also all the debugging messages end up coming under the DBus process in ~/.xsession-errors because we were using DBus activation to start them.

With upstart user sessions we're now getting a lot more power and flexibility in managing the jobs in the user session, it makes sense that indicators would start to use it to control the backend services. This comes with a set of advantages.

The first one is that there is better developer control of the state of the process. It's really easy to start and stop the service:

$ stop indicator-network
$ start indicator-network
and the ever exciting:
$ restart indicator-network
All of these ensure that the same commands are run each time in a recreatable way. Plus give the user and/or developer full control.

Upstart also takes the output of each process and puts it into its own log file. So for our example here there is a ~/.cache/upstart/indicator-network.log that contains all of the junk that the backend spits out. While this is nice to just make xsession-errors cleaner, it also means that we can have a really nice apport-hook to pick up that file. Hopefully this will lead to easier debugging of every indicator backend bug because they'll have more focused information on what the issue is. You can also file general bugs with ubuntu-bug indicator-network and get that same information attached.

In the future we'll also be able to do fine tuned job control using external events. So we could have the indicator network backend not start if you don't have any networking devices, but startup as soon as you plug in that USB cellular modem. We're not there yet, but I'm excited that we'll be able to reduce the memory and CPU footprint on devices that don't have all the features of higher end devices, scaling as the features are required.

Those that know me know that I love diagrams and visualizations, and so I'll have to real quickly say that I'm excited about being able to map our desktop startup using intlctl2dot. This gives a Graphviz visualization of startup and how things will interact. I expect this to be a critical debugging tool in the future.

What's next? Getting all the indicators over to the branch new Saucy world. We also want to get application indicators using a similar scheme and get a fast responsive desktop. Hope to have a blog post or two on that in the near future.

posted on Thu, 25 Apr 2013 at 12:30 | permanent link