Loar's Lair - Technical

Computer Science Programs - Joel Spolsky Saves Me Some Effort

Tue, 27 Oct 2009 23:04:43 GMT

Ever since my friend Elton made a post about what he wished he had learned in college, I have been meaning to make my own post about my experiences at UIUC, er, I mean, ILLINOIS. I don't disagree with the things he wishes CS programs would teach, though I do disagree with the value of the education I received in college. While Elton seems to have enjoyed his time at UT and done well, the three years I spent at UIUC were the most miserable of my life so far. As I often find myself explaining to people today, there is surprisingly little overlap between the skills required to be a good software engineer and the things they teach you as an undergrad in what is supposedly one of the top 5 computer science programs in the country. My classes there were far more about obtaining a pretty piece of paper to hang on my wall than preparing to enter the workforce. This was exemplified by the CS grad student I met who appeared to be incapable of writing "Hello World" in C if his life depended on it.

Anyway, today I was going through my RSS feeds and found Joel Spolsky's post on the subject, and he hits it on the head. Let's teach our students Scheme! It's so functional and elegant! Nobody in industry uses it, but that's because they're all unsophisticated heathens! One of my colleagues once remarked on my zeal for producing a quality product by saying that "If you ignore a bug long enough, Matt will fix it." Apparently if I delay a blog post long enough, Joel Spolsky will write it.

While I'm on the subject of Joel Spolsky, however, I need to take exception to something he said in his post on "Duct Tape Programmers":

Duct tape programmers have to have a lot of talent to pull off this shtick. They have to be good enough programmers to ship code, and we'll forgive them if they never write a unit test, or if they xor the "next" and "prev" pointers of their linked list into a single DWORD to save 32 bits, because they're pretty enough, and smart enough, to pull it off.

As a member of the team which recently became the not-so-proud owner of GDI and GDI+, I will say that if you do a crazy what-the-eff-is-he-doing "optimization" like munging bits in a pointer and then fail to supply any unit tests for the poor bastard who has to maintain your code after you run off to work on the Next Big Thing, then you deserve to be fired. Or shot. Maybe both.

HTC Fuze

Sun, 26 Apr 2009 00:08:32 GMT

A few weeks ago I replaced my broken Blackjack with an HTC Fuze. So far it seems to be an improvement. The HTC Fuze combines a touchscreen with important things like Send and End keys and a slide-out QWERTY keyboard. The slide-out keyboard is definitely an improvement on the Blackjack's tiny keys and allows the Fuze to be narrower than the Blackjack, if twice as thick. The Fuze is pretty heavy, and I have the feeling that it would not take well to being dropped. Another thing I like is that unlike the Blackjack, it has a standard Mini-USB port.

Software-wise the phone is marginal. It runs Windows Mobile 6.1 and comes loaded with HTC TouchFLO(tm) software which provides a shinier and more touch-friendly home screen. Of course it's not very customizable, so the first thing I did was turn it off. I like how it files SMS messages into conversations. I never used SMS on the Blackjack, so I don't know if this is a recent feature or not. When using the web browser I have run into an issue where it returns a "403 Forbidden" page for all sites. I suspect this may be an AT&T issue rather than a phone issue, as disconnecting and reconnecting to the network fixes it. This is made easier by the fact that the function performed by holding the End key is configurable, and I have configured it to toggle "airplane mode."

The voice quality on the Fuze is fine, but using it as a phone can get very irritating. The keypad is through the touchscreen. When a call is active, the display has a number of buttons for functions such as turning on the speakerphone, and you have to hit a softkey to bring up the keypad. Also, the screen turns off after a second while a call is active (presumably to avoid inadvertent touchscreen presses), so you have to press the power key to turn it back on. This makes using touchtone interfaces (like voicemail) very frustrating.

Today I woke up to find my phone frozen and had to remove the battery to restart it. I think the phone had been up for about a week. It's a sad state of affairs that I think a phone crashing after a week of uptime is marginally acceptable, but hey, it's WinMo.

afsbak 1.0

Fri, 13 Feb 2009 06:26:41 GMT

So I recently found myself wanting to share files between my desktop and my laptop. So I did what any sane, rational person would do. I set up an AFS cell.

Of course, I didn't want to put anything important in AFS until I had a way to back it up. At ACM, we had access to the campus Tivoli TSM service, so we ran an AIX machine for the sole purpose of using the old version of the TSM client that had AFS support. However, this wasn't exactly an option for me.

I already used BackupPC for backups, so I wanted to use that to back up AFS as well. But how? Someone pointed out BackupPC4AFS, but this doesn't allow for file-level backups, and the last thing I want to do is maintain a modified BackupPC.

I got the idea to take a vos dump and transform it into tar format and then feed that into BackupPC. I grabbed the restorevol utility from the OpenAFS source tree and with some work produced afsbak. It reads a vos dump on stdin and produces a tar archive, optionally producing shell scripts that when run will restore the ACLs on the containing directory.

Please let me know if you find it useful or find issues.

A Brave New World of Secure DNS

Tue, 27 Jan 2009 07:38:12 GMT

I was looking at the OpenSSH code the other day and I saw some code related to retrieving fingerprints from DNS. I had never heard of this, and I wondered how that could be secure.

Reading RFC 4255, the answer to that was simple: DNSSEC. This really makes a lot of sense: if you're trusting DNSSEC to guarantee the integrity of the value of the A record providing you the IP address to connect to, why not trust it for the encryption key to expect when you connect? This is actually quite nice, since DNSSEC protects you against DNS spoofing but doesn't protect you against IP spoofing.

Of course, there's no reason it has to stop with SSH. It turns out that there is a CERT RR defined in RFC 2238 which is designed for storing X.509 or PGP certificates in DNS. This is particularly interesting given the way that the trust chain for X.509 certificates for HTTPS currently works. The only guarantee you are connecting to the real HTTP server for www.amazon.com and not one run by an attacker is trusting that none of the Certificate Authorities that your browser trusts would issue a certificate for www.amazon.com to anyone other than Amazon.com, Inc. And for the vast majority of users, the CAs their browser trusts are the ones that their browser or operating system vendor saw fit to include, which is generally a very long list of CAs run by private companies whose rationale for why you can trust them seems to range from "we charge money" to "we charge obscene amounts of money."

However, if it were possible to securely publish SSL certificates in DNS, the need for these shady authorities would be eliminated. And in turn, not needing to shell out money for SSL certificates would hopefully lead to greater proliferation of SSL for general web browsing, not simply for things like order forms and online banking. The true beauty is that this is backward-compatible - there is no reason why you couldn't obtain a certificate from a CA and publish it in DNS. CAs might even continue to serve a purpose for those who want to go to the hassle of getting an Extended Validation certificate.

Likewise, the CERT record could be used for storing public keys for signing and encryption of email. It would be much simpler for someone wishing to send you an encrypted message to find your key if it were published in DNS, as they would not have to determine which keyserver your key was on and whether it was in fact your key. Having such a key in DNS would also make verification of signed messages easier for tools such as SpamAssassin or even MTAs. Greater proliferation of signed email could help in the neverending battle against spam.

Keys stored in such a manner could also be used to solve the longstanding problem of having to remember passwords for the myriad websites you visit. Many websites already use email addresses as usernames. In a future world, this email address might be used to look up a public key, which could be used for a challenge-response authentication. Though there's no reason sites couldn't ask for a PGP public key instead of a password when creating an account today - the main problem is that there is no protocol I know of for doing web authentication using PGP keys - instead browsers support SSL client authentication, which appears to me to be very unwieldy and I have never personally seen it used.

And I think that about sums it up - there are several separate public-key cryptography systems that serve overlapping purposes: X.509 certificates, PGP keys, and SSH keys. Each has a different trust model. But imagine if hosts and users on the internet were able to authenticate themselves to each other based on a single trust apex at the DNS root. Maybe this is all pie-in-the-sky, but I have to hope that the future will bring resolution of the awful mess we have now.

How to Waste Time

Fri, 09 Jan 2009 00:16:48 GMT

My method of choice? Rewriting managed code projects in C++.

I've been doing that for a couple of things, namely Neztu and most recently bloog. The former actually makes sense, since I have blogged before about my issues with Mono. My most significant qualm about it is its lack of support for fully precompiled sites, which means your clients get wonderful 5-10 second delays when it decides it needs to recompile something. I had been under the impression that writing a web application in C++ would be an exercise in pain and suffering, but it turns out that when you combine cgicc with the absolutely excellent pqxx, you get a (relatively) rapid development platform, and combined with the magic of FastCGI you get the performance and stability that is sorely lacking on so many sites these days. I have my new C++ Neztu in a very functional state, but I wanted to implement all of the features of my 0.1 release before I make a formal release. In the meantime, you can look at the hg repo, under the neztu-cxx branch. I have also decided to drop the liquidsoap support. It seemed cool at the time, but forcing two tracks into a "not queued but not playing" state is kind of a dealbreaker, and the methods of interacting with the rest of Neztu were kludgey at best.

My other project was pretty stupid, as other than being managed code, there wasn't anything terribly wrong with bloog. But my philosophy is that if I enjoy it, it's not really wasting time. Which is good, because thanks to wmwork I can tell you that I spent 15 hours rewriting it in C++. The two main benefits were that I discovered the original bloog had some bugs in date parsing that meant it didn't handle timezones correctly (all my latest entries are timestamped in UTC, but older ones need adjustment). Also, even my get-it-done sloppy unperformant C++ is 22% faster than the Boo version. Like the original Boo version, I don't intend to make any formal release of this code, both because it's hardly production quality and because I seriously doubt its usefulness to anyone else, but it is in Mercurial.

PuTTY and GSSAPI

Mon, 03 Nov 2008 00:10:00 GMT

So apparently when I wasn't looking, the PuTTY authors added support for GSSAPI, which is great. Unfortunately, they only support SSPI on Windows. Granted, this is the cleanest way, but depending on which version of Windows you are running, SSPI may or may not work for you. If the client and server are in the same realm or AD domain it should work, but cross-realm is dicey since support for client-side host-to-realm mappings wasn't added until Vista. Some people on the Kerberos list expressed disappointment that the PuTTY authors went this route.

So I intend to add this support back in. The PuTTY GSSAPI support has a radically different code layout from my current patch, so the first thing I have done is merge in the PuTTY upstream and streamline my diff against it. The PuTTY code delayloads SSPI, so I'm hoping it will be straightforward to delayload MIT GSSAPI instead.

At the same time, I got a feature request a few months ago for gssapi-keyex support. I haven't made much progress there, but I am determined to fully exploit my current enthusiasm for working on PuTTY - it's always nice to work on projects that are actually useful to other people :).

Flite and Clunit Voices

Mon, 22 Sep 2008 23:48:54 GMT

I have been playing around with Flite lately, which is basically a way to take a FestVox voice built for Festival and convert it to C code which can then be compiled into one monolithic binary. Flite ships with the KAL diphone voice. However, I don't particularly care for this voice, and have been trying to convert the CMU CLB ARCTIC voice for use with flite. I followed the guide, and the automated conversion went smoothly, but at runtime I got bad synthesis and errors like this:

clunits: unit type "f" not found
clunit_get_unit_index: can't find unit type f, using 0

After many hours of studying the flite code and the festival code, I finally discovered what the problem was. In festvox/cmu_us_clb_arctic_clunits.scm, there was a phone-to-clunit mapping function:

(define (cmu_us_clb_arctic::clunit_name i)
  "(cmu_us_clb_arctic::clunit_name i)
Defines the unit name for unit selection for us.  The can be modified
changes the basic classification of unit for the clustering.  By default
this we just use the phone name, but you may want to make this, phone
plus previous phone (or something else)."
  (let ((name (item.name i)))
    (cond
     ((and (not cmu_us_clb_arctic::clunits_loaded)
       (or (string-equal "h#" name) 
           (string-equal "1" (item.feat i "ignore"))
           (and (string-equal "pau" name)
            (or (string-equal "pau" (item.feat i "p.name"))
            (string-equal "h#" (item.feat i "p.name")))
            (string-equal "pau" (item.feat i "n.name")))))
      "ignore")
     ((string-matches name "[aeiou].*")
      (string-append
       name
       (item.feat i "R:SylStructure.parent.stress")))
     ((string-equal name "pau")
      name)
     (t
      (string-append
       name
       (item.feat i "seg_onsetcoda"))))))

However, the converted C code contained just:

static char *cmu_us_clb_arctic_unit_name(cst_item *s)
{
    return cst_strdup(item_name(s));
}

Not surprisingly, converting arbitrary Scheme code to C is outside the capabilities of the Flite conversion code. The code I replaced it with doesn't encompass all of the logic of the original, but appears to work fine:

static char *cmu_us_clb_arctic_unit_name(cst_item *s)
{
    const char *name = item_name(s);
    if (*name == 'a' || *name == 'e' || *name == 'i' || *name == 'o' || *name == 'u')
    {
      const char *stress = ffeature_string(s, "R:SylStructure.parent.stress");
      char *namey = cst_alloc(char, strlen(name) + strlen(stress) + 1);
      cst_sprintf(namey, "%s%s", name, stress);
      return namey;
    }
    else if (strcmp(name, "pau"))
    {
      const char *onsetcoda = ffeature_string(s, "seg_onsetcoda");
      char *namey = cst_alloc(char, strlen(name) + strlen(onsetcoda) + 1);
      cst_sprintf(namey, "%s%s", name, onsetcoda);
      return namey;
    }

    return cst_strdup(item_name(s));
}

(Before you go emailing me, cst_alloc terminates the process on allocation failure. This code uses the same patterns as elsewhere in the Flite code.)

And now I have a working Flite binary for this voice.

Mono and Monotony

Thu, 18 Sep 2008 07:43:40 GMT

Work has been pretty slow lately - not too many bugs to work on. But I have been keeping busy at night trying to figure out Mono issues. I upgraded the test webserver at ACM to lenny and set up Mono FastCGI, but they still managed to get it to randomly encounter compilation errors due to missing files.

Several nights of tedious debugging later, I believe I have discovered the problem - when a new ApplicationHost is instantiated, it deletes all of the files in the DynamicBase of the AppDomain. Once I discovered this, I remembered that the Mono people do say to only run one FastCGI server at a time. However, I'm not sure I like this approach, and so I'm planning to write a wrapper script that will ensure that different servers use different directories to avoid interference.

Also, I have made some edits to my previous post on Mono FastCGI. My Deny rule was ill-conceived, and so I have changed it to simply send all ASP.NET filetypes to the FastCGI server. And while I realize that the Mono folks recommend using paths instead of extensions, as far as I can tell this is not possible using mod_fcgid and FCGIWrapper.

This brings me to another question - what is the best way to make an edit to a published post? Should I try to preserve the original content, appending edits to the end or using strikeout? Or simply replace? I'm leaning toward the latter, though I would like some kind of audit trail. Since I have this all in a Mercurial repo anyway, I'm thinking I need to get the changelog wired into my system somehow. I tried this once before, but I couldn't tease the log out of hg in a format that cl2xhtml would take.

I suppose I should end this post before I really start to ramble.

Mono FastCGI and Apache

Fri, 05 Sep 2008 11:41:40 GMT

ACM has been using Apache with mod_mono for a while (my fault) and while it works relatively well, there are some issues. Namely, every once in a while the mono server will crash and requires manual restarting. So when I heard that Mono now had FastCGI support, I was intrigued, hoping that it might alleviate these problems.

I couldn't find any instructions on setting it up with Apache (using mod_fcgid), so I figured it out on my own. Here's how I configured it to run my development copy of Neztu on Debian lenny:

# apt-get install mono-fastcgi-server2
# mkdir /tmp/mono-fastcgi
# chown www-data.www-data /tmp/mono-fastcgi
# echo DefaultInitEnv MONO_SHARED_DIR /tmp/mono-fastcgi > /etc/apache2/conf.d/mono-fastcgi

I then put the following in my vhost configuration:

Alias /neztu/ /home/matt/devel/neztu/www/
<Directory "/home/matt/devel/neztu/www">
  DirectoryIndex index.aspx
  Options ExecCGI FollowSymLinks
  AddHandler fcgid-script .asax .ashx .asmx .aspx .axd .config .cs .dll .rem .soap

  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .asax
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .ashx
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .asmx
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .aspx
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .ascx
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .axd
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .config
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .cs
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .dll
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .rem
  FCGIWrapper "/usr/bin/fastcgi-mono-server2 /applications=/neztu:/home/matt/devel/neztu/www" .soap
</Directory>

And that's all there is to it. I will have to test it further to see if it is able to recover from its hiccups with this method.

Me and You and A Blog in Boo

Thu, 04 Sep 2008 00:01:27 GMT

You may have noticed that I have changed my blog yet again. I had switched to NanoBlogger a while back, but I wasn't quite satisfied with it. For one, it was still separate from the rest of my site. Also, while the concept of blogging software written entirely in bash is novel, it is also quite slow. A thousand invocations of sed can't be the most efficient way to do anything.

So I set about coming up with a way to migrate my blog to use spin like the rest of my site. About this time I was also exploring new languages as part of my "maybe C++ isn't so wonderful after all" epiphany. I had dabbled with Python before and found it kind of nice, so I started working in Python.

However, I soon found something about Python that irritated me. Python has automatic memory management, but for other resources, you still have to do C-style cleanup, which is ridiculous. So it uses refcounting/garbage collection to automatically free memory. Great. I still need deterministic cleanup for file handles, database connections, etc. Not only do you have to explicitly call close() on the object when you are done with it, but you need to build for exception safety through the use of try-finally*. This makes C++ with its RAII designs actually look good.

I had been using Mono and C# for other projects. At least the .NET Framework got this right - objects owning unmanaged resources implement the IDisposable interface. C# contains a using statement which not only automatically calls Dispose on the object no matter how the block is exited, but when used correctly also scopes the variable to the block.

However, I've worked with C# before, and I wanted to try a new language. That's when I discovered Boo. It's a language inspired by Python, but built as a .NET language (IronPython reimplements the Python class libraries on top of the .NET Framework with the goal of compatibility with code written for CPython; Boo doesn't aim for Python compatibility and uses the .NET class libraries). Boo is also statically typed with type inference (and has optional duck typing).

And thus I wrote Bloog. It is designed to take a NanoBlogger database and build a thread-based blog that can be converted by spin. While I could have simply used verbatim blocks and kept the HTML as is, I didn't want to. Luckily, I found html2text, a Python thing that takes raw HTML and converts it to Markdown, which was the syntax I had been using for my most recent entries anyway. After wiring Bloog up to Markdown.NET, I discovered something disturbing. Not only is it not possible to parse the Markdown syntax in a streaming fashion (it supports grouping link URLs at the end of the content), but all the parser implementations I have seen rely on generating a bunch of MD5 sums for the purposes of escaping certain characters. Not surprisingly, the Markdown conversion was very slow. So I hacked up the Markdown converter to output thread instead, and just converted all of my old blog entries to that. On top of that, I do all the timestamp-based optimizations to only generate what I need to, and the result was a compose-preview cycle that is orders of magnitude faster than NanoBlogger. My hope that got me through this project is that this will persuade me to post more often.

I would be extremely surprised if anyone else has the need to convert a NanoBlogger blog to thread, but I've made my Mercurial repository public, just in case.

So, this is my new and hopefully more permanent blog. Please let me know if you see anything out of order.

*Apparently Python 2.5 adds the with statement which works like using, but you still have to import it from __future__.