Just returned from FOSS.Aalborg '09 where I held a talk about OpenEmbedded. It's been my second time in Aalborg (first time was Mobile Developer Days '07) and it's been a nice and informative experience. Met some folks doing amazing things with embedded. Everything was organized very professional and the presentations were interesting -- particularly the talk about security which was held by two guys from the OpenBSD project. OpenBSD has never really been on my radar, although according to what I've seen in the presentations it seems to be of really high quality. I guess I'll give it a look soon.
Next week I'll be in Bern for OpenExpo '09. In May, the first FSOSHRCON and in June LinuxTag'09 are planned and perhaps FrOSCon again in August, but other than that, I pretty much try to stay at home for the remainder of this year.
Btw., there's free WiFi on Aalborg Airport -- that's the spirit!
I felt it's time to recap the stuff that kept me busy the last months and give you an overview over the achievements planned for this year -- always focusing the free software movement, of course.
Let's start with the major project I've been working on, the freesmartphone.org project, funded by Openmoko, Inc. FSO grows, and it grows in the right directions. We get more API customers -- notably the SHR project and the Paroli project -- and refine our API and the reference implementation. The 5th milestone has just been released and apart from a major foobar with read-only partitions, it's pretty good. We are going to fix this OE-inheritance and release a milestone 5.1 in a couple of days.
For some modems -- e.g. the TI Calypso (see my previous post on ogsmd and its modems) -- until now we have relied on pyneo's gsm0710muxd. Over the last weeks we found some severe problems (race conditions, buffer overflows) with this though, so I thought I have a shot at developing my own GSM 07.10 Multiplexer.
The result is called fso-abyss and is -- as with all our software -- available at git.freesmartphone.org under a free software license. The major difference to gsm0710muxd is the architecture (and maintainability). While gsm0710muxd combines talking to the serial ports, the pty's, handling dbus queries, and doing modem specific things, fso-abyss went a different route.
At the heart there is a minimal protocol engine implementing GSM 07.10. Since there was already something available in Qtopia -- even nicely seperated without any external dependencies -- I took that one and factored it out in a dedicated project called libgsm0710 (available in git as well). The idea here is that different interest groups can collaborate on getting the protocol engine right, since not everyone wants a DBus frontend such as implemented in fso-abyss. The next step was writing a VAPI file for glueing the protocol engine to Vala (more about that one in a bit), which has been used to develop the upper layers of fso-abyss.
Last but not least, there was the pty implementation, the serial port communications abstraction, and finally the dbus server. The DBus API originally designed in cooperation with pyneo has been enhanced to feature the additional features (only) present in fso-abyss. Apart from the architecture, fso-abyss also can handle virtual serial port signalling, 07.10 test commands, automatic session handling, has a wakeup service, and more. Next up is adding support for the Cinterion mc75i which has some proprietary extensions to GSM 07.10 Basic Multiplexing.
Modern DBus APIs are pretty dynamic, i.e. objects can come and go at any time. Depending on the hardware, you may find more or less objects of a certain kind. You can now add infrastructure to query the objects (essentially a duplication of what DBus should provide), or just rely on the existing DBus introspection API. Unfortunately this API is missing some critical features to make it really usable, such as querying objects that implement a certain interface.
So I took the plunge and factored this out of the freesmartphone.org frameworkd, since it has broader use. This is the API for it (as introspected by mdbus):
root@om-gta02:~# mdbus -s org.freesmartphone.DBus /org/freesmartphone/DBus [METHOD] org.freesmartphone.DBus.ListBusNames() -> ( as:result ) [METHOD] org.freesmartphone.DBus.ListObjectPaths( s:busname ) -> ( ao:result ) [METHOD] org.freesmartphone.DBus.ListObjectsByInterface( s:busname, s:iface ) -> ( ao:result )
Here are examples of how you can use it (demonstrated within a Python shell):
>>> hlid.ListBusNames() [ 'org.freedesktop.DBus', 'org.freesmartphone.omuxerd', ':1.21', 'org.bluez', 'org.tichy.launcher', ':1.13', ':1.0', 'org.freesmartphone.frameworkd', ':1.14', ':1.1', ':1.2', ':1.3', ':1.4', 'org.freesmartphone.ogsmd', ':1.6', 'org.freesmartphone.DBus']
>>> hlid.ListObjectPaths("org.freesmartphone.ogsmd") ['/org/freesmartphone/GSM/Device', '/org/freesmartphone/GSM/Server']
>>> hlid.ListObjectPaths("org.freesmartphone.odeviced") [ '/org/freesmartphone/Device/Audio', '/org/freesmartphone/Device/CPU', '/org/freesmartphone/Device/Display', '/org/freesmartphone/Device/Display/0', '/org/freesmartphone/Device/Display/gta02_bl', '/org/freesmartphone/Device/IdleNotifier/0', '/org/freesmartphone/Device/Info', '/org/freesmartphone/Device/Input', '/org/freesmartphone/Device/LED/gta02_aux_red', '/org/freesmartphone/Device/LED/gta02_power_blue', '/org/freesmartphone/Device/LED/gta02_power_orange', '/org/freesmartphone/Device/LED/neo1973_vibrator', '/org/freesmartphone/Device/PowerControl/Bluetooth', '/org/freesmartphone/Device/PowerControl/UsbHost', '/org/freesmartphone/Device/PowerControl/WiFi', '/org/freesmartphone/Device/PowerSupply/ac', '/org/freesmartphone/Device/PowerSupply/adapter', '/org/freesmartphone/Device/PowerSupply/apm', '/org/freesmartphone/Device/PowerSupply/battery', '/org/freesmartphone/Device/PowerSupply/usb', '/org/freesmartphone/Device/RealTimeClock/0', '/org/freesmartphone/Device/RealTimeClock/rtc0']
>>> hlid.ListObjectsByInterface("org.freesmartphone.odeviced", "org.freesmartphone.Device.LED") [ '/org/freesmartphone/Device/LED/gta02_aux_red', '/org/freesmartphone/Device/LED/gta02_power_blue', '/org/freesmartphone/Device/LED/gta02_power_orange', '/org/freesmartphone/Device/LED/neo1973_vibrator']
While working on implementing GSM time(zone) support for ogsmd, we found we had too few samples, especially since time(zone) information are only sent by few providers all over the world. Moreoever, we missed a generic means to record all the data the frameworkd is sending out via its signals, such as:
To support this (and more), we came up with fso-monitord, which is available from git as well. fso-monitord logs its data to a flat file format that you can send to us to improve our databases or for debugging. We also figured this would be the best place to add a generic frameworkd watchdog -- monitoring all fso components -- shutting down or restarting components as necessary and also logging incidents such as API violations.
For milestone 5.5 (due end of march), we have two major features on the roadmap, namely bluetooth networking (headset profile) and extended PIM support. Milestone 6 will then sport full-fledged networking.
Beyond milestone 6 -- apart from one major thing, which I'll cover in a second -- we only have some rough plans, such as revamping or refining the subsystems we're not perfectly happy with (oeventsd and opreferencesd come to mind). Also, alsa audio scenario handling is broken by design, but this is something we have to take up with upstream.
The freesmartphone.org reference implementation has been progressing incredibly fast. This is partly due to choosing Python as the implementation language (which has been a wise choice) of our DBus APIs. Now you all know that although I truely love Python (I even wrote a book about it) and try to use it everywhere it fits, I'm very well aware that for the future of the freesmartphone.org project, it might be important to come up with a frameworkd reimplementation in a compiled language -- to reduce the footprint and squeak every possible bit of performance out of the (embedded) system.
This is why I have decided to encourage a second reference implementation. This one will be written in Vala (I might have mentioned it before, did I?) which is an incredible combination of elegance and performance, featuring a complete lack of any runtime penalties and additional dependencies. It's simply amazing and I'm seriously thinking about writing an introductionary book about Vala later this year.
Anyways, back to the topic, the first bits of this Vala implementation has landed in the freesmartphone.org git in the form of the very successful GSoC project odeviced, written by Sudarshan S. Stay tuned for some amazing FSO runtime speedups coming in autumn and winter this year to your device.
Next to writing software for the freesmartphone.org project, I also found some time to pick up working with my favourite writing tool LyX. LyX, which could be described as a LaTeX frontend, nowadays features integration with the new LaTeX variant XeTex. In contrast to other incarnations such as pdfLaTeX, XeTeX can utilize system fonts such as AAT or OpenType, which are the latest technology in computer-assisted typesetting.
I can now use my "corporate" fonts FF Meta and FF Meta Serif from LyX -- amazing!
Although still working on cutting down my travelling, I can't miss some conferences this year. I managed to skip FOSDEM, which made me a bit sad, but I'll be compensated by attending
and possible some more... This year my main topics will be OpenEmbedded and freesmartphone.org -- both dedicated to reducing the fragmentation of Linux-based embedded systems and to ease writing software for mobile devices running free and open source software. I hope we'll bump into each other at one of these occasions.
Stay tuned!
Our Sid Player application has been on sale in the AppStore for three weeks now. We received quite a nice amount of feedback considering that this is a niche application -- thanks to all of you who downloaded and reviewed or mailed us!
Behind the curtains, we have been analysing the feedback and have just started to work on the first update which we expect to be available within the next 4 weeks.
This next release is mostly a database update, since we will add the missing songs from the HVSC directories 'GAMES' and 'DEMOS'. This adds about 2000 more classy titles that were left out due to not being present in the appropriate 'MUSICIANS' subdirectories.
Apart from that we will add an NTSC-override in the settings tab and -- if all goes well -- add some download-all-the-good-stuff buttons ;-)
Stay tuned for more exciting things coming to the Sid Player this year!
Although traditionally the Chaos Computer Congress' schedule is slightly suboptimal for me (12/26th is my birthday), I'm going to be in Berlin from 12/28th to 12/30th and will visit CCC on the 3th day (12/29th). I'm going to attend Harald's talk about GSM base stations, so if you want to talk to me, just pick me up afterwards.
You all know how much I adore the C64 sound aesthetics. This is the #1 program I was missing in my iPhone.
Recently, I have been working on adding more support for various types of modems to ogsmd. "Why's that?", I hear you say, "isn't that already done since long?". Yes and no. There are two dimensions where modems might differ:
Both dimensions come with their own set of problems:
As you might know, there are basically three different means for exporting an AT interface:
Before we go into the details, some notes about what I mean with multiplexing mode. The 3GPP standard 07.10 defines a binary multiplexing protocol that exposes a number of virtual channels over which you can talk AT as defined in v250, 05.05, 07.07, etc. The concrete number of channels modem-specific -- it usually varies between three and four.
Let's come back to the three ways of exporting an AT interface now:
A single line without multiplexing support is the easiest, however it is also the most limiting way. The reason for that being that quite a lot of commands might take a long time to complete -- during that time the single line is blocked and you can not perform additional commands nor receive the so-called unsolicited responses from the network. These messages usually inform you about changes in the environmental GSM network condition, but also occur as notifications for incoming SMS or calls. In that scenario you will not be able to handle GPRS and voice calls at the same time.
In the most simple form, a modem might support two discrete lines, e.g. one for all AT commands, the other one for GPRS connections. Some modems support three, four or even many more lines, however this usually comes with some strings attached. Multiple premultiplex lines usually are non-homogenous, that is you can not perform every AT command on any line, rather there is a line for SIM access, a line for voice calls, a line for network commands, etc.
In contrast to the premultiplxed line, these kinds of modems support an optional multiplexing mode over the single line transport (see explanation above). While this means that you need support code for the demultiplexing, a usually nice side effect is that the multiplexed lines are homogenous, e.g. the application (ogsmd in that case) can chose how to operate the lines. If you have more than two lines, you might want to dedicate one to unsolicited responses only, so you can react to network events even while dialling out. You might want to reserve one line for GPRS data connections and -- if you have a modem that does not buffer SIM access -- you might want to dedicate one line for the (potentially slow) SIM card retrieval commands.
The second dimension of differences is the set of supported AT commands. The GSM AT command set is relatively "old" (in terms of computer science) and has its root in the (even older) command set (e.g. v25ter / v.250) designed for analogue terminal adapters. The GSM standards such as 3GPP 05.05, 07.07 define commands that serve as addition to the ones defined in v.250.
Unfortunately, the lion's share of these commands has been tagged "optional" -- also some of the GSM functionality is severely underspecified. This can lead to trouble such as follows:
Briefly, some examples:
The TI Calypso is a 2.5G (no EDGE) modem from Texas Instruments, as found in the Openmoko mobile phones Neo1973 and the FreeRunner. It has a fairly comprehensive AT implementation. All the commands support the query parameter and there are few standard violations. TI invented about 50 extracommands, of which ogsmd is using about 10 -- mainly for extended call progress information and network status.
The Freescale Neptune is a 2.5G (EDGE) custom modem as appearing in the widely successful Motorola EZX Linux mobile phone family. The AT implementation is a nightmare, there are dozens of standard violations, implicit behaviour all over the place, and as if that wasn't enough, it has about 100 completely undocumented custom commands. And yes, 90% of the commands do NOT support the query format...
The Cinterion (previously SIEMENS) modem is one of the latest in a huge family of industry modems, deployed in millions of stationary and mobile devices. It will appear in the next-generation Openmoko device (codename GTA03). Its AT implementation is conforming to all standards and has been canonically enhanced to fill out the missing spots in the specifications. The documentation is a dream.
Supporting an additional modem in ogsmd can take from a day to some weeks, depending on the discussed two dimensions "way of exposing the interface" and "commandset implementation".
By the way, one particular ugly area is the call handling. As I have mentioned, the GSM standard only supports little information here (due to the compatibility with analogue modems), so most vendors have added their own unsolicited commands to take care of that. For those who have not though, we have a generic abstraction in ogsmd that uses timings and additional status requests (+CCLC to the rescue) to overcome this limitation. However, imagine a singleline modem with no vendor supported extra commands blocking during an ATD command... welcome to the dark.
So all this has motivated you to add a new modem abstraction to ogsmd? Excellent, be my guest! It now boils down to first identifying the way how to talk to it and then either chose to derive from the generic 'singleline' or the 'muxed4line' modem class. If the latter, you can adjust the number of individual channels created based on the capabilities of your modem. If your AT interface is premultiplexed, there's nothing else to do but match the channels to the right device nodes. If your AT interface supports a multiplexing mode, use the gsm0710muxd inbetween ogsmd and your modem (take a look at the TI Calypso implementation which is doing that as well).
Second, you need to find out where the standard AT implementation does not work with your modem or is not giving you all the information your modem could provide (think special commands). For such cases, in your new modem class you want to override specific mediators and add new unsolicited handlers. I'll briefly explain what these two are:
In ogsmd, a mediator is a small class that encapsulates a dbus command such as:
ogsmd's abstract mediator module (ogsmd/modems/abstract/mediator.py) contains mediators that are using standardized AT commands to perform their job. If your modem does not support these commands -- or has better commands to do that -- you should override said mediator (ogsmd/modems/mediator.py) to issue different commands in the 'trigger' method. Then, collect the results in 'responseFromChannel', and forward it to dbus -- that's all.
Unsolicited handlers specify what happens on incoming messages that originate from the GSM network, e.g. a +CRING that informs you about an incoming call. The handler then can perform additional queries or just trigger a dbus signal.
If your modem has additional (non-standard) unsolicited commands (or violates the 3GPP standards, which happens often enough) that you want to process, write a handler function in the unsolicited module (ogsmd/modems/unsolicited.py) for each of these.
I hope you have an idea what it means to add a new modem support class to ogsmd now. If you have more questions or want to help, drop a mail to smartphones-userland@linuxtogo.org.
I hope you enjoyed this months' tech talk ;-)
Thanks to my framework team buddy Daniel 'Alphaone' Willmann, I just fell in love with the german band Pornophonique. Everyone appreciating the 8-bit chipsounds we had in the old days of computing will absolutely enjoy their work. My favourite song is 'Sad Robot' from the album 8-bit-Lagerfeuer -- it sounds a bit Everlast-inspired, but hey, that's a very common 4-chord-loop.
Did I mention that their music is creative commons licensed? Hats off, guys -- you rock!
::: {.img-shadow}
{width="200"}
:::
好久不見! Three weeks passed within a blink. Last sunday, we smoothly landed in Frankfurt/Main after 14 hours of a calm flight. 4/5 of the Openmoko Framework Team (while Stefan was on vacation in .au and missed all the fun) met in Taipei to tackle some outstanding issues and synchronize with the plans for the next major Openmoko release.
For a start, please refer to the blog postings of Charlie and Daniel, who mentioned some of the things we did in detail. Let me cover the current status and what we are going to work on in the remainder of this year and then step back and talk a bit about the meaning of all this. Right now the framework offers you support for the following tasks, everything accessible via consistent and convenient DBus interfaces:
The two major things missing until we officially declare a 0.9 release are PIM and Networking. For the former, we're attempting to integrate the results of a Google Summer of Code project (opim API), for the latter, we're (still) evaluating whether NetworkManager, Moblin Connman, or Exalt can fit our usecases and needs -- plus a very limited set of convenient calls on top.
All of the above is of course complementing the freedesktop.org initiative and should serve as a natural addition in order to help standardizing important Linux-based embedded APIs, as found in Maemo, Openmoko, LiMo, Moblin, OpenEZX, etc. We are commited to cooperate with said platforms to help defragmenting the mobile device world so that application programmers have it easier to target multiple platforms.
Along this line, the recently posted weekly Openmoko engineering newsletter adressed the issue of Openmoko and its relationship to freesmartphone.org. The bottom line is that Openmoko is funding the freesmartphone.org initiative to help fighting fragmentation. Previously Openmoko's strategy was breadth-first to show the amazing versatility of open devices. Their next strike is going deep in one direction to create something that is both attracting developers (thanks to the dbus service level framework) as well as users (thanks to a pleasingly simple, extensible, phone application). We will see both of this in the forthcoming Openmoko 2009 distribution which is going to be the first FSO-compliant distribution ever -- with more (e.g. Debian with pkg-fso, SHR and Rasterman's work) being underway. Lots of them built out of OpenEmbedded, of course. (Speaking of OpenEmbedded -- the long awaited switch to git is happening this week. Open the flood gates and embrace our new revision control system :))
Last but not least some personal remarks. With me being part of the Openmoko family for more than two years now, it was time to reevaluate and redefine our relationship. I'm really glad to announce that Openmoko supports my direction of stepping a bit back from being overall Openmoko platform architect, allowing me to concentrate on the freesmartphone.org framework -- making this part of the Openmoko platform as strong as possible.
As said, the three weeks passed too quickly and we found out we need to synchronize more often -- hence I'm looking forward to increase the frequency a bit and next year stay more often at the Openmoko headquarters. 乾杯!
Although trying (really!) to cut down travelling, it's still a lot. Here's a sweeping swipe of what happened during the past couple of months and what's going to happen soon.
FrOSCon took place at its usual place on the last weekend in August and it was a very well organized conference -- even better and more streamlined than the previous year was. I had the pleasure to listen to the Minix3 talk from Prof. Tanenbaum, which was very entertaining. Unfortunately my Openmoko talk was right after his one, so I had to take people kind of down to earth ;)
Since I was feeling pretty weak at this weekend I could only attend the first day. Looking forward to next year's session.
Been travelling from Frankfurt to Braunschweig (to work with the Openmoko students on the framework) for a couple of times and I have started to actually use my Openmoko FreeRunner as a GPRS-forwarding device for my laptop. Using the freesmartphone.org framework it's a breeze to do that. I just have to issue the dbus command ActivateContext("internet.eplus.de", "", "") and wait until the context goes online. Then I use iptables to enable NAT and forwarding for the laptop on the FreeRunner and it's done.
GPRS is very solid on the FreeRunner -- it works for hours without any disconnections or other interruptions. If the data connection is not 100% loaded you even get incoming call signalling and can take phone calls in between
Talking about the Openmoko / freesmartphone.org framework... we had a successful milestone3 release of it, debuting PDU-mode for SMS and phonebook data as well as lots of bugfixes over the place. We also have some nice reference docs now -- i'm still working on introductionary type docs.
Unfortunately despite me trying to educate, lots of people still don't get the point of the framework releases -- I get frequent comments on the testing UI zhone, but rarely anyone is actually contributing to the dbus API specification and implementation discussion :( I seriously ponder whether to release console images in the future to make this 100% clear. I say it again: FSO is not about user interfaces, it's about a strong independent dbus service level framework to facilitate 3rd party development.
That said, there are four important contributions in the freesmartphone.org world: pkg-fso, fso-gpsd, frameworkd-glib, downloads.freesmartphone.org:
All software goodies live in our git repository.
Last week I had the honor to give an invited talk about OpenEmbedded and Qt-integration into OE for the Mobile Developer Days 08 conference in Berlin. This conference is pretty unique in that it adopts a platform-agnostic approach, i.e. you will find people working on Symbian, Qt, PalmOS, Windows Mobile there. I even spotted iPhone folks. In my opinion, such a holistic approach is important for the future of development on mobile devices. Congrats, folks.
Speaking about the iPhone... true readers of this column may remember that I have been a MacOS user since early this year. To add up on this, lately I acquired an iPhone to gain some experience with this exciting new development platform. I have just been looking into what this system provides. I can already say that there's a whole lot of stuff where FOSS can learn and I'm glad to be a part of both worlds, so I can try to be a catalysator.
On Sunday I'm going to fly over to Taipei. It's been a while (12 months to be exact) since I met the folks in person there and there's lots of stuff to catch up on. Now that the framework approaches its 0.9 release at the end of the year, we need to discuss the plan for the next 6 months. Can't wait to meet all the engineers again! It's going to be three interesting weeks. Stay tuned :)