Howto build a trivial Debian package with dh_make

Building a Debian package can be complicated. If you’re in a hurry to build a really simple package, you might find the available documentation (such as the Debian New Maintainers’ Guide) overwhelming. Maybe you just want to install some files with dpkg. Maybe you don’t want to deal with Makefiles. Maybe you discovered the source package for hello-debhelper (the “Hello World” of Debian packaging) uncompresses to 3.1MB. Maybe these instructions will be more helpful. Then again, maybe not.

This example will create an architecture-independent package called foo which will install an empty file at /usr/share/example/file. To install other files, just place them wherever you want them to go under the example-src directory.

mkdir foo-0.1
cd foo-0.1
dh_make --createorig --indep # this will prompt you to hit enter
mkdir essentials
mv debian/{changelog,compat,rules,control} essentials
rm -r debian
mv essentials debian
echo './example-src/* ./' > debian/foo.install
mkdir -p example-src/usr/share/example
touch example-src/usr/share/example/file # create the empty file to be installed
dpkg-buildpackage -uc -tc -rfakeroot
dpkg --contents ../foo_0.1-1_all.deb # inspect the resulting Debian package

Notes:

• In addition to these steps, you probably want to edit the debian/control and debian/changelog files.
• The version of the package is defined by the changelog entry, which must conform to the Debian changelog format. (dh_make wrote the first entry for you, with the version taken from the name of the directory you ran it in.)
• For reasons I haven’t investigated, if you try to put files under /usr/local/ this recipe won’t work. (There are probably other directories this will also not work for; /usr/share and /var are all that I’ve tried successfully.)
• These instructions were tested using debhelper 8.0.0ubuntu1 and dh-make 0.55; if you find that they do not work in some newer version please leave a note in the comments.

Notes on dual-booting Ubuntu and MeeGo on an ExoPC

I attended an Intel developer marketing event last thursday promoting MeeGo and their AppUp program, where everyone received an ExoPC tablet. Officially these are not gifts, but rather a three year loan “for engineering purposes only”: after three years we are supposed to destroy them or return them to Intel. (They said it is up to us which of those options to choose.)

I hope MeeGo is a success, and I hope to make some apps for it (although, as I am not at all interested in the AppUp DRM regime, I was glad to hear that they will allow apps to be listed in their catalog without using the AppUp apis), but first I wanted to see how a modern version of Ubuntu fares on this keyboardless tablet. The short version of this post is: it works quite well!

I was pleased to find that ubuntu-11.04-desktop-amd64.iso (copied to a USB stick with usb-creator-gtk) supports the ExoPC’s touchscreen, audio, video and wifi without any hassle. Unfortunately gparted (and thus the Ubuntu installer) does not yet support resizing btrfs, which is what MeeGo uses as its root filesystem. Luckily it wasn’t too difficult to do this manually.

I shrunk the btrfs partition down to 10GB using these commands in a root shell in the Ubuntu live environment:

mkdir /meego
mount /dev/sda2 /meego
btrfs resize 10G /meego
umount /meego

Then, using fdisk I then deleted (“d”) sda2 and recreated it (“n”) with a size of 10G, and created a new (“n”) partition sda4 in the new free space. I wrote the changes to disk (“w”), quit fdisk (“q”), and ran partprobe. Then, I ran the ubuntu installer and told it to format sda4 and use it as / and to use meego’s swap partition (sda3) for swap and to use the meego /boot (sda1) as its /boot but not to format it. Somewhat to my surprise, this all worked out fine, and I am now dual-booting MeeGo and Ubuntu from the Ubuntu grub2 menu!

I was going to create an LVM-on-LUKS crypto layer under the Ubuntu filesystem (I recently retaught myself how to do that manually from the desktop installer CD – something else I should write up a blog post about) but then realized that would mean I’d need a hardware keyboard to boot up since the passphrase is needed long before X starts. So, I held my nose and went with the “encrypt home directory” (ecryptfs) option instead.

Three caveats about dual-booting: First, interacting with grub obviously requires a usb keyboard. Second, MeeGo (aka “Unknown linux distro on sda2”) is listed twice, first with the MeeGo kernel and then with Ubuntu’s. Finally, to actually boot MeeGo it is necessary to edit the boot options to add “insmod btrfs”. I could put that line in grub.cfg so I wouldn’t need to do that every time, but it will just get overwritten on the next kernel upgrade… I haven’t yet figured out the correct way to make this type of setting stick with Ubuntu’s grub2.

I think my next step with MeeGo might be to try to boot its partition in VirtualBox using raw disk mode; if that works then I could do development on the device in Ubuntu and test on the same MeeGo system which I can also boot natively. But for now, I’m more interested in using Ubuntu. :)

I played with the new Unity desktop shell for a while, but soon switched to “Ubuntu classic” (the familiar GNOME desktop).

I installed the “Grab and Drag” firefox extension, which is nice once you get used to its physics. It should have more positions on the various sliders in its settings.

I was very pleased to discover that, when properly configured, the “onboard” software keyboard makes Ubuntu quite usable without a hardware keyboard! Unfortunately, unless I’m missing something, adding a launcher for it ironically requires another keyboard to type its name – it comes preinstalled, but is not listed in the menus or UI anywhere except on the login screen. I was especially pleased to discover the “show onboard when unlocking the screen” checkbox in its settings! Onboard also provides right-click and middle-click functionality which works in many but not all contexts.

I also installed the florence onscreen keyboard, which is nicer looking than onboard but much less usable. For instance, the firefox awesome bar steals every other keypress from florence while with onboard it works fine.

In the Appearance settings, under Fonts, I changed the dpi from the default of 96 to 135 (which is what wikipedia says the ExoPC’s screen is). This made lots of text larger, and generally made the whole UI more finger-friendly.

The ExoPC has a single button sensor at the top left of the screen. By default in Ubuntu it launches the audio player. I remapped it to the “Hide all windows and show the desktop” toggle in Keyboard Shortcuts.

I haven’t yet figured out how to get the brightness to stop changing when it is on battery power, and sometimes when I plug it back in the brightness does not return until I go to power management and use the slider. This is irritating.

I haven’t installed the broadcom CrystalHD driver yet, but I understand that doing so will improve flash video performance which is presently not great.

If you’re running Ubuntu on an ExoPC, here are some other resources with useful info:

• http://shallowsky.com/blog/linux/install/natty-on-exopc.html
• http://exocommunity.com/forum/viewtopic.php?f=111&t=2363&sid=bb78187b3579ab5defa3ae592730165c
• http://www.phlerion.com/blog/exopc/exopc-ubuntu/

thunkurry: a Python decorator for partial application

Here is a decorator which allows you to curry named and/or positional arguments as many times as you like. The underlying function is finally called when the decorated function is called as a thunk (meaning, without any arguments).

def thunkurry(f_, *t_, **d_):
    """Currying thunk decorator
    Example usage:
    >>> prod = lambda *a: reduce(operator.mul, a)
    >>> sum_ = lambda *a: sum(a)
    >>> thunkurry(prod)(2)(3)(4)() == prod(2,3,4) == 24
    True
    >>> thunkurry(lambda a,b,c:a+b+c)(c=1)(2)(c=3)(4)()
    9
    >>> a = thunkurry(sum_)
    >>> a = a(1)(2)(3); a()
    6
    >>> a = a(1)(2)(3); a()
    12
    """
    def fn(*T_, **D_):
        return f_(*t_, **d_) if not (T_ or D_) \
                else thunkurry(f_, *t_ + T_, **dict(d_.items() + D_.items()))
    fn.__doc__ = f_.__doc__
    return fn

I’ve mostly found this useful as a class decorator.

(Note: the Talk:Currying page on wikipedia has some discussion about the terms currying vs. partial application; I’m using the terms interchangeably here and am not sure if that is wrong.)

MySQL relay-log-space-limit vs. your page cache

MySQL slaves have two replication threads, the IO_THREAD and the SQL_THREAD. The first continuously fetches binary logs from the master and writes them as relay logs, creating a new file every time the current one reaches max-relay-log-size. The second continuously reads those relay logs, applies them to the database, and deletes each file when it is finished with it. So, when everything is running normally, and the SQL thread is keeping up with the master, there will not be more than two relay log files.

There are numerous reasons why  MySQL replication might lag briefly and then recover. The situation which caused me to write this post was due to a spike in writes on a database that was already close to being disk-bound under normal conditions. (Even though the master and slaves have identical hardware, and the slaves get no traffic other than replication because they’re intended only for high availability, the slaves hit the IOPS bottleneck first due to having only one SQL thread.)

When the SQL thread is not able to keep up, logs will accumulate. You can set the location of your relay logs with the relay-log option; in the default configuration (and many production configurations), the relay logs are stored on the same disk as the rest of MySQL’s data. If the total size of the relay logs ever exceeds relay-log-space-limit, the IO thread will pause until the SQL thread has moved on to its next file and deleted its previous one.

Unfortunately, relay-log-space-limit defaults to zero (unlimited). This means that when the SQL thread can’t keep up, the slave’s relay logs will continue to accumulate indefinitely (until its disk fills up). When the total size of those logs approaches the amount of vfs page cache available, the SQL thread will begin to read them from disk (instead of getting cache hits), which will exacerbate the IO shortage that caused the problem in the first place.

Setting relay-log-space-limit will cause the slave’s IO thread to pause and resume as needed, so that the slave is always reading relay logs from its page cache. You should probably do that.

This will have a side effect of making lagged slaves cause cache misses on the master instead, because the IO thread will now be reading older binary logs which have already fallen out of cache. But, depending on your application, this might be just what you want: if the extra disk contention from reading those logs on the master slows down the application’s writes, it can effectively throttle everything until the slave catches up.

If you’re like I was a few weeks ago, you’re learning this while you’ve got a lagged slave that is thrashing its disk needlessly reading relay logs, and you face a dilemma: changing relay-log-space-limit requires restarting MySQL. If you have a large innodb_buffer_pool_size that you need to keep warmed up, restarting can take a while… during which time the replication problem will continue to worsen. But, if you do nothing, the extra IO of the SQL thread reading its relay logs may prevent it from catching up for even longer. Depending on how short on IOPS you are, once you start reading relay logs from disk instead of page cache, you may be unable to catch up at all!

So, if you want to have the effect of relay-log-space-limit, but you don’t want to restart MySQL to get it, this hack I call “pseudo-relay-log-space-limit.sh” might be just what you need:

#!/bin/bash
# this is a stupid hack for situations where you have a lagged slave
# without relay-log-space-limit set which you want to be able to catch up, 
# without needing to restart and rewarm it.
# note: Seconds_Behind_Master is NULL while the IO thread is stopped.
max=250000000
while true; do
cur=$(echo 'show slave status\G'|mysql|grep Space|tee /dev/stderr|egrep -o '[^ ]+$')
if [ $cur -gt $max ]
then echo STOP SLAVE IO_THREAD  |tee /dev/stderr| mysql
else echo START SLAVE IO_THREAD |tee /dev/stderr| mysql
fi
sleep 10
done

Okuda theme for N900

The Okuda theme and an accompanying sound effects package are now available in the maemo.org extras-testing repository. Please download, test, and vote for the theme and sounds so that they will be promoted to the main repository!

How to extract text from PDF files using poppler and GOCR on Ubuntu

Poppler provides a suite of utilities for working with PDF files. If you’re using Ubuntu, you’ve already got it installed; otherwise, see if your operating system provides a poppler-utils package.

If your PDF file contains text, you can use the pdftotext command:

pdftotext foo.pdf

If the PDF contains images of the text which you wish to use OCR on, you’ll need to extract them first:

pdfimages foo.pdf foo-page

…which will give you a bunch of files called foo-page-nnn.pbm (read man pdfimages if you’d like to only extract certain pages)

If, as in my case, the resulting images are of white text on a black background, you can correct this using the convert command from ImageMagick (run sudo apt-get install imagemagick)

for i in *pbm; do convert -negate $i $i; done

(I used the shell to iterate through the files because the convert command’s batch functionality doesn’t seem to work with pbm files.)

And, finally, use GOCR to recognize the text in the images:

for i in *pbm; do echo gocr $i; gocr $i > $i.txt; done

Note: I wrote this early last year and just today noticed it still sitting in my drafts. I’m not sure why I didn’t post it before, so I will now. Hopefully these instructions are still accurate!

Microblogging is easier than blogging

This blog isn’t dead, it just smells funny. While you continue to wait (with bated breath, I’m sure) for me to put the finishing touches on some things I’ve been meaning to publish for the last decade but still haven’t gotten “done”, you can read my fascinating microblog. I also just added a widget for it on the sidebar here.

Computing permutations with a recursive generator expression

#!/usr/bin/python2.5
def permute(li):
    """Generate all permutations of a sequence
    >>> for i in permute([1,2,3]): print i
    (1, 2, 3)
    (1, 3, 2)
    (2, 1, 3)
    (2, 3, 1)
    (3, 1, 2)
    (3, 2, 1)"""
    return ((i,)+j for i in li for j in (permute([k for k in li if k is not i])
                                         if len(li) > 1 else [()] ) )

import doctest
doctest.testmod(verbose=True)

LCARS 4.2 released

Here is another photo and a screenshot of our new theme, Okuda:

 

The changelog has details on what else is new. To install or upgrade, just use this .install file. After installing, select a theme and a background and then reboot for everything to be properly reloaded.

The install process is unfortunately rather slow, usually taking between 1 and 3 minutes, due to the numerous cache files which must be updated post-install. Starting with this release, the themes’ postinst scripts now run gtk-update-icon-cache on the icon themes’ directories, and lcars-extras runs it on the hicolor icon theme in postinst and postrm. The Okuda theme includes a font, so it runs fc-cache. And, as before, all of the themes run hildon-theme-cacher (which is the slowest of the three types of cache). I’m not sure why hildon-theme-cacher needs to be run on the device; it seems like it would make more sense (and be quicker for users) to pre-generate those gtkrc cache files as part of the themes’ build process. Oh well.

Enjoy the new themes, and please let us know if you make modified versions! :)

Preview of our new Maemo theme

Ian and I have been working on a new release of our LCARS packages, featuring a number of bug fixes as well as a new theme called “Okuda”:

If all goes well, we should be ready to release this within the next 48 hours. All four of our themes now include icon templates, which we’re slicing with hildon-theme-slicer (the same way that the main theme template is processed). This is what the icon template for the Okuda theme looks like:

Anyone else wishing to create a hildon-theme-tools (and ThemeMaker) compatible layout definition might find this Python script helpful:

#!/usr/bin/python2.5
"""Generate a hildon-theme-tools layout from variously sized images.
Edit script to specify desired margin, width, and origin (dx,dy).
usage: hildon-theme-layout-grid-generator.py `find icons|sort|grep png`
"""

__author__ = "Leif Ryge <leif @synthesize.us>"
__license__ = "no rights reserved / public domain"

from sys import argv
from PIL import Image

margin, width, dx,dy = 10, 600, 0,0 # margin, output width, starting coords
sizes = {}

for f in argv[1:]: sizes.setdefault(Image.open(f).size,[]).append(f)

print "[Main]\nTemplateWidth=%s\nTemplateHeight=%s\n\n[Elements]" % (width, \
    sum((h+margin)*(len(sizes[w,h])/(width/(w+margin))+1) for w,h in sizes) )

for w,h in sorted(sizes):
    column = width / (w+margin)
    row = 1 + len(sizes[w,h]) / column
    print "# %s images are %sx%s (%sx%s grid)" % (len(sizes[w,h]),w,h,column,row)
    for i, f in enumerate(sizes[w,h]):
        print "%s=%s,%s,%s,%s" % (f, dx+(i%column)*(w+margin),
                                     dy+(i/column)*(h+margin),w,h)
    dy = dy+row*(h+margin)

The actual template can then be created from existing images using Nokia’s hildon-theme-regenerator tool.