A friend on IRC was having trouble getting the glpng library to build on the Mac; it's unmaintained, but reputedly solid, and it's nice if the sole reason you're using SDL or whatever is to get at its image-handling functions...

Since I've done this stuff a lot I knew that it was basically a matter of fixing include files and such. So I did that. I did not bother to get "make install" to do anything sensible, so don't use that. My darcs repository is at http://www.accela.net/~dankna/software/darcs/glpng/ and I'd appreciate feedback.

Some people have been asking what my game project is about. For a long time I would just summarize it every time someone asked, but today I finally got around to putting together material that I'd written in different places and editing it into a semi-coherent overall introduction. I'm probably NOT going to post regular updates about the project here, but I'm posting this, so that I can have someplace to point people when they ask. I know the text is rough; and the paragraphing is weird because this originated as a chatlog - actually several chatlogs. Just felt the need to explain that so I wouldn't be embarrassed. :) And please feel free to ask any questions you want!

It's an exploration platformer, with lots of story. My mantra, that I feel I will need frequent reminders of through the project, is that it's a platformer enhanced by pokemon elements, not the reverse. Because a pokemon enhanced by platform elements won't sell; too crowded an area.

So for example I intend to put lots of loving detail into the play control of each creature type, and give them unique moves and stuff. The abilities that you level up in the skill tree won't be abstract names and spells and stuff, they'll be actual perform-during-play moves.

I have a lot of backstory for the world it's set on. I decided the human trainer in pokemon-genre games is totally superfluous and primarily exists to be your map-screen avatar and so that the NPCs have someone to talk to. So I got rid of him. Rather like Myst, you're going to spend a lot of time discovering and reading through ancient libraries.

It's not a silent protagonist though, you're a very specific character - a linguist from Earth who's been astrally projected to this world because none of the actual diplomats were willing to do it. (Someone commented that silent protagonists are on their way out; I said, nah, they'll never be totally dead - cf. the first chapter of Scott McCloud's "Understanding Comics".)

Dialogue is gonna be Sierra-like conversation trees, except it's a conversation web, not a tree. To motivate the player to talk to the same character more than once, despite having not yet advanced the overall plot since last time, you won't be able to explore the full web in any one session. For the simple and obvious reason that as much as they might like to, people have better things to do than to talk to you indefinitely.

Obviously - as everyone points out when I mention this part - there has to be a careful balance struck there because it could become frustrating. I'll find the balance through experiment.

Somebody observed: “Conversations are tricky in games. Either you have the simple model where you talk to some stranger who happens to impart secrets in two sentences, or you have the Mass Effect model where you end up talking far too much for most people to care. “Mass Effect is a great game, but most will say the dialogue is far too extensive... But it's nice when you have a choice as to how you respond. I can tell someone 'thank you' or 'shut up' in the same choices and get an actual different response.” Right. I call that Sierra style. :)

I hope that the dialogue will be a) interesting, and b) only necessary if you want the full experience like all the sidequests and stuff. Another thing I'm doing is giving you the option to just walk away at any time if you realize "oh shit I didn't mean to talk to this person and now I'm stuck in their menus".
Another another thing is I'm using the Metroid Prime scanner. Somebody in an indy-game-dev channel thought I should only do that if the character had a good reason to have one; but, as you can tell from the background, he does.

And, I'm using the scanner as a way of interacting with other characters. You can show things you've scanned to the technician on your team, or email them to your grad students back on earth, or whatever.

I have frequently given my lecture on what an awesome game mechanic the Met Prime scanner is, because it lets you absorb backstory totally at your own pace and based on what you're interested in. The way it works in Prime (mine will have minor differences), you switch on the scan visor and you can see hotspots in the world; select one to get a bunch of text that talks about what that thing is.

Oh, one thing I'm NOT doing is having a zillion fictitious types of stone that are susceptible to a zillion fictitious types of explosive, but, for whatever reason, each only to its matching one. One thing that space exploration actually has taught us — do you know what Mars is mostly made of? Basalt. So is the moon. An ordinary rock that appears here on Earth.

Which makes sense, as the periodic table is the same everywhere. Of course planets without water don't have sedimentary rocks, only igneous and metamorphic. The planet in my game does have water so I'm going to have its geology be quite similar to Earth.

Although I came up with some unique terrain types too, like a gelatin swamp. The idea (which may or may not be explained to the player, but it's so *I* know...) is that algae got sick of the fact that larger life forms eat them and they can't do anything about it. So, they evolved a minor use of magic, thickening the water to the point that nothing can swim in it. Because life is very persistent, there are still creatures that live on the bottom of the gelatin swamp, and there are still plants that take root on its surface. But nothing *swimming* which means most of the fluid is solely the domain of microorganisms, without interference from larger ones.

I was asked: So the geology is similar, what about atmosphere, gravity, and such? And I'd assume the creatures are still carbon-based?

Still carbon-based, yes. Another thing we know is that practically every solar system, even interstellar space, has the same amino acids floating around. (We know this through spectroscopy; I'm talking real-world here, for just a moment, haha.)

And for that matter, I observe that the basic arrangement of a face - two eyes or at least eye-centers, a nose, a mouth, ears - and their relative positioning… Has evolved quite a few times, independently, just on Earth. Not all animals have all those parts of course, like snakes have infrared patches instead of ears. But when the parts do exist their placement is generally the same regardless of lineage. And some animals have noteworthy exceptions, like spiders and their many eye-centers.

Gravity and atmosphere I've been thinking a LOT about.

One of my other concepts is that this world is really big, so that, even as ice ages killed off their equivalents of the dinosaurs and their equivalents of the mammals took over, there were places remote enough that the dinos were able to survive.

In fact, part of the game's exposition will be of how the present-day political conflict on the planet is just the same ancient conflict between dragons (dinosaur-analogues and bird-analogues) and wyverns (amphibian-analogues). Playing out for the hundredth time in the hundredth way.

Now as for how Earth enters the picture, I stole an idea from Asimov and one from Niven, and bounced them off each other.

There's an old Asimov book "The Gods Themselves..." It’s divided into three major sections, with the amusing titles "The Gods Themselves", "Against Stupidity", and "Contend in Vain?". It’s about how an alternate universe makes contact with Earth and proposes a solution to their mutual energy crises.
But Earth doesn't realize that it's getting the short end of the stick, and the other universe basically refuses to talk to Earth, at all, ever, because the less Earth knows, from their standpoint, the better.
In my game Earth was very insistent on that point and that's why our linguist got permission to go there. Also it's not an alternate universe in my game, just another planet.

The part from Niven is from his novel "The Magic Goes Away," which depicts, in essence, a magical energy crisis. All the flying cities fall down when the world no longer has enough magic to keep them in the air. All the grand architecture that was twisted through other dimensions and stuff stops working. "And those damn stupid barbarians, with their damn stupid swords, will win after all."
You can see how these ideas mesh, I think, grin.

I even borrowed a bit from Lensman - this part is *definitely* detail the player isn't going to uncover, it's just for me.

In Lensman there are two ancient races which were the first intelligent races to evolve. And the entire history of the galaxy is the conflict between them.

They stay hiddean from the newer races... every planet ever winds up being either like the Arisians (the good guys) or like the others, whose name I forget. Transformers borrowed heavily from Lensman, as you can see.

You might have jumped to the conclusion that in my case I'm making one race represent science and the other magic. Not at all, grin. They are both proficient users of both.

But, Earth was an experiment where the "good guys", observing that having access to magic made it far too easy for civilizations to stay locked in a perpetual medieval period — which is indeed what happened on the planet you're visiting — decided to suppress the ability to draw on the planet's magic for a few million years and see what happened.

Notice how I have even things like microorganisms using magic as part of their biology. So it's not enough to just say "humans never learned how to tap magic"; there has to be a better reason that there's no magic on Earth. And this is what it is.

It's not a problem the player is likely to even think of until he's played the entire game, so.

Musically, I'm trying something a bit daring, encouraged by the success of Knytt Stories (free Windows game - go play it immediately if you haven't). I didn't want mediocre music to cheapen the experience of exploring the world, so I decided it couldn't sound like "game music".

One of the key things that makes music sound like game music is that it loops. So compare it to, say, the Nutcracker or Swan Lake. You can't *do* that; the music can't be like a little story, because it has to repeat.

I already knew that I wanted the game to have some areas that use music and other areas that use environmental sound — water flowing, leaves crackling under your feet, …

The songs will be longer than the typical game song, probably about eight minutes. Once they're over, the game will drop into environmental-sound mode, until something cues another song. Which could be a plot event, or entry into a new area, or the start of a plot-significant battle...

I decided that most battles will not change or interrupt the current music, only important ones.
Somebody said, “Interrupting music is always tricky to do anyways unless they're designed to loop, but it'd be better to actually have music broken into sections to where there are 'dummy endings' in case an event happens sooner than expected, but otherwise can continue normally.” In fact I’ve been discussing exactly that sort of issue with the composers. They seem to really get it and are excited to try something new. So I'm pleased.

Another thing I'm thinking about doing, in plot-battles, is having dialogue cued by the music, in real time, and not by events of the battle. The dialogue will just display in some way that doesn't interfere with you conducting the actual fighting at your own pace, which is still probably turn-based.

Now as far as visuals, the DS is *very* hardware-restricted. Compared to a computer, I mean. Have a look at, say, Dragon Quest Monsters: Joker, which, like my game, is full-3D. If you are doing something less than full-3D you won't hit the limits. It has slightly less ability than the original playstation.

The polygon limit is 2048 triangles per frame, but the effective limit is even smaller. For non-graphics people who read this, that’s small. Fortunately the screen is small too.

It was remarked that Mario 64 runs on the DS, to which I said, if you put Mario 64 side by side with Mario Sunshine, 64 would look dated and lame. 64 has very few textures; most things are constructed from solid-colored regions. Anyway, so yeah, I'm falling back on tricks like that.

Some of my scenery objects will simply be solid-colored flat things in arbitrary shapes. That allows me to have far MORE scenery objects than I otherwise could, you see.

Because otherwise textures would dominate the memory usage - absolutely everything, graphics, sound, code, etc has to fit into just four megs of working ram.

Which is the same as the original playstation, by the way. The PSP, now, has 32 megs of working ram, and a hardware codec chip for streaming video and sound. And a slightly faster processor than the DS. But I evaluated the PSP and decided not to go with it.

Someone mentioned demographics of DS owners being appropriate for this idea, which I agree with. Mostly it's that I think there's a lot of room in the DS market. A lot of released games have some really shoddy things and yet are still selling really well.

A lot of games for it are deliberately very "cute", targetted at the pre-teen crowd, yet I observe that the handful of games targetted at the 18-30 crowd are among the top sellers. (Based on Amazon sales rank, because I can't afford to pay a real market-research company for historic data.)

I'm definitely going for gorgeous, not cute.

Well, the other day when I wrote about this, I did a test where I used an external tool to decode the LZW payload of a GIF and feed that into my generated PDF. That worked. Then I tried decoding with PostScript's codec by using Ghostscript interactively, and comparing the data to what I expected. That worked. Then I tried telling the PDF to decode with the same filter. That did not work.

So I slept on it.

Today I looked in on it again and the problem was immediately obvious: PostScript and PDF have different versions of the LZWDecode filter, which aren't quite identical. Specifically, their parameters have the same names, but in PDF only values between 9 and 12 are valid for the initial number of bits in each code. In PostScript, any value is valid. The stream I was testing with used 3 bits. I actually had already tried generating a stream that wouldn't have that issue, since I was confused by some wording in the PDF spec, but that one used 8 bits!

So that answers that question. Yay! Now I need to decide what to do about it! Obviously I need to decode and re-encode the thing. Since I want to link against Ghostscript anyway to handle EPS conversion, maybe I should figure out how to link against it now and then have it do the recoding. That way I won't need a Haskell implementation of LZW. Not that LZW is that hard... I've implemented it before, but that's precisely why I don't really want to do it again. Boring to keep doing the same thing. And since Ghostscript already contains a well-tested implementation, as well as impls for any other codec I might possibly want...

So, I've been working on embedding image files into a PDF. This could be done by uncompressing them and storing them in any convenient codec, but there's two good reasons and one bad one not to do that. A shame, because it would potentially be a lot less work; I could use the existing Haskell binding against the GD library to get at uncompressed pixel data.

First, it would potentially be substantially larger - ever seen a PDF that was tens of megs just for a couple pages? That's probably because somebody was careless about image encoding.

Second, when converting from a lossy compression format, recompressing will cause quality loss.

Third, and this would trump the other concerns anyhow, doing it this way gives me a chance to learn neat stuff.

You would really think there would be a nice library to do this stuff already, since Adobe specifically makes note of the possibility and all that, but there's not... I found a nice paper that actually surveys a lot of open-source and some closed-source libraries and concludes that almost all of them do it the lazy way.

JPEGs turn out to be really easy. One of the built-in codecs can deal with an embedded stream that's simply an entire JPEG file. It even extracts the image size and other parameters from that stream, instead of you having to specify them explicitly like with every other codec. So I've already got that working. Of course, if time were at all a consideration, I would just declare it done at this point and have the users convert all images to JPEG beforehand, but since it's not...

Other formats are a bit more work. It supports the basic codecs used for GIF - which is of course the famous LZW, finally out of patent as of a couple years ago - and PNG, which, to my surprise, is the old familiar Deflate codec that's used by both zip and gzip. Unfortunately, the pixel data isn't the only thing in the file; and they're both palletized formats, so you need to extract the pallete, too.

This is relatively simple stuff as it goes... I'm most of the way through doing it for GIF now. I got a major boost when I finally found a source of working implementations I could look at: A couple small utilities that come with Ghostscript, written as command-line tools and themselves implemented in Postscript (yes...). It's a little awkward reading that, since the PDF side of things kinda happens automagically, but at least I know it works, so it's useful.

One other thing that actually is kinda important: EPS. Encapsulated postscript is a vector format. You might think that since it's so closely related to PDF, you could somehow convert it directly, but you can't, because they have different built-in APIs (in particular, the PDF one has no flow-control constructs). You could rasterize the EPS but of course it would be much nicer to keep it resolution-independent, not to mention smaller.

I wasn't sure how to deal with EPS, but then I realized that Ghostscript can be used as a library, and in that mode you can run it without any support files installed anywhere. So you can just hard-link it into the distribution and be done with it. Have Ghostscript convert the EPS to a PDF, then extract the content stream of the PDF, and use it directly as a content stream in your own PDF. Postscript and PDF do have very similar rendering operations; they're just invoked diferently. So in effect this is running the program contained in the EPS, but trapping all the graphics calls it makes, and, instead of running them, outputting equivalent calls in PDF vocabulary. If the EPS contained a program to compute the entire Mandelbrot set, this would have disastrous results, but “don't do that, then.”

For the Larpbase project, I was thinking EPS files could be used for the heart and energy logos, although on close examination they appear to be high-res raster images right now. At any rate, it's an interesting enough challenge that I'm going to make it work one way or another!

So, I've been taking a break from my other projects by working on a component that I'll eventually be needing, although the system it will be part of is nowhere near ready yet. This is essentially mail-merge for PDF, although I'll need to support both text and images. I'll also need to support multiple fonts.

At first I looked at Perl libraries; Perl has several libraries that deal with PDFs. PDF::Reuse is almost exactly what I want, and nicely documented. Unfortunately, I didn't really understand how I was supposed to use it until I'd already read up, and now I've reimplemented enough of its functionality that it would no longer be a time-saver.

Mostly, when I was looking at existing work, I ran into many other Perl modules which were not PDF::Reuse, and which were all very poorly documented, and finally I had to turn to the official PDF reference to understand what they were supposed to be doing. And then I realized that the reason it's so confusing is that most of these libraries are just one piece of it - dealing only with the document's internal index, or only with generating content streams (the postscript-like portion of the file), or only with parsing the syntax.

So I have a syntax parser/lexer; I put it together in Parsec. I'm starting to really dislike Parsec, but I have to concede that I seriously doubt I could have supported the full syntax in just a couple hours without it. I also wrote Show instances for everything, so that my PDF.Syntax module can both generate and consume. I even handled all the weird little escape things in the string syntax; sweet, huh? Biggest holdup with this part was that, ouch! Different strings can be in different text encodings! You can even mix-and-match line endings within the same file! So I had to poke deeper than I wanted to into Haskell's IO stuff in order to come up with something 8-bit-clean.

Syntax is only half the issue. A PDF file, it turns out, is not a linear stream. Well, figures, right? Anyway, it has an index at the end of the file, the xref table, which gives a mapping of object IDs to byte offsets. I was thinking I would have to deal with space allocation as I deleted and replaced objects, and then regenerate the xref table from scratch, but it turns out that the design is nicer than that: You could do that, but you can also leave the existing document untouched and just append the new or replaced data, and another xref table which contains entries only for the changed things, plus a pointer back to the old one. Since all the internal data structures are built in terms of object IDs anyway, the ability to remap object IDs means you can do anything in append-only mode that you could do by writing an entire new file. Sweet, huh?

It took me a while to get that working, since it was hard to see where the problem in my generated output was when all I got was either total success or total failure when I tried to use the resulting file. So to do it in pieces, first I tried making a PDF entirely by hand. Ouch! Painful! Had to use a hex editor to find offsets into what was otherwise more or less a text file! So I wrote some code to generate a trivial PDF, and refactored that until it turned into a PDF updater. Which now works.

Right now, after basically eight hours of coding, what I have it doing is opening up a file and drawing a predefined graphic either on top of or below every page. I took the example code for the graphic from Adobe's reference.

Making it generate text will be easy; it just has to output a content stream with the appropriate command. Making it do so in any given font won't be much harder; the document contains copious metadata, and it just has to look up what fonts the document already has embedded, and then decide which to use.

Making it generate graphics will be a bit harder. Apparently PDF supports some of the same codecs that popular image formats are based on, particularly jpeg. So I have a choice of whether to slice-and-dice the input image and just embed its existing compressed chunks into the PDF, or whether to uncompress the image, convert it to a different format, and reencode it. There are existing tools for both. The former apparently has a very significant space savings, but is a lot more fiddly and there are fewer examples for me to work from. So, I'm reading up on that.

I also haven't decided yet whether I want to store image files as blobs within the database (which means I can take advantage of the version-control I'll be implementing for the rest of the database, and users won't have to explicitly think about moving them around), or as separate files (which would make the database smaller). Well, plenty of time to think about that. The PDF code will be designed so it doesn't care where its data is coming from, anyhow.

Shark, for those who don't know, is Apple's profiling tool. You don't need to run in their IDE to use it, which is good... It's impressed me in many ways with both its precision and its versatility, but what it did for me just now was particularly cool.

A website that I run was being slow. So I start up Shark and record a trace as I load the page from my test server, running on the local system. I don't want to assume without checking that the problem is in my code and not in, say, the Apache config causing things to be done in a slower way than necessary, so I tell it to record the whole system.

So it records the whole system and chews on its data and pops up a box informing me that the most-time-consuming function in the most-time-consuming process is Perl_utf8_length, with a self time of 41% of the total userspace time of that process during the sampling period. Hah! After a moment of thought I realize that this just confirms what I suspected anyway, namely that XML::SAX::PurePerl is way too damn slow, but still, it's nice to know that for sure.

But the slick part of all this is in what I didn't have to do. Notice how I didn't tell it the path to my script. In fact I just let it loose on the whole system and it realized that the program that was thinking hardest was probably the one I was most interested in hearing about, so it showed me that one first - with a drop-down menu to see all the others. It wasn't able to decipher the Perl call-stack, only the Perl runtime library, so I couldn't tell the name of the Perl function that was calling Perl_utf8_length, but that's still helpful; it's clearly something parsing-related. And notice also that I didn't have to tell it how to find the debug symbols for that systemwide library. Or even tell it to go fetch them at all; it did that on its own.

One other amusing thing I noticed, although it wasn't useful to me this time. I wondered whether IO operations might be the main expense, so I looked at the special process which is used to account for time spent in kernelspace - and it informed me that something like 90% of kernel time was spent in the library AppleIntelCPUPowerManagement. After a moment of thought, I realized this meant it was sleeping in the idle loop. (I have two CPUs, and parsing is inherently serial, so it can still be idle while working hard.) Pause for a moment to reflect on how slick this is. It wasn't able to get me the symbols from that driver, although it does get symbols from the kernel at large. But it continued its sampling uninterrupted (pardon the pun!), even in kernelspace, even going the extra mile to figure out what driver the PC was in.

Usability and stuff is all great too, but let me tell you, it's things like this that really warm my heart towards Apple.

Well, it's been overdue for quite some time! Note that we are now calling it CFFI-Net, as that is the ultimate aim of the project; the same C-Grovelling and POSIX-access features from CFFI-Unix and CFFI-Grovel are still there. Project page; tarball; darcsweb.

This includes two months worth of patches, so this description is probably missing something. C Enum support has been revamped and is stable now. There are fixes for several build problems that had arisen due to upstream cffi changes. Hopefully it depends less on the internal cffi interfaces, now, so it won't break again.

Thank you to Greg Pfeil for ECL support, fixing build problems and providing generally useful feedback. Thanks also to Roger Sen Montero for implementing cmucl support, Samium Gromoff for improving clisp support, and RP Goldman for ACL support (unfortunately not merged with R8, but in darcs).

This release brings us closer to our grand goal of supporting actual network usage. :) That part of the code is still alpha and I'll be surprised if it works for you... It has a prototype interface that provides the same minimal functionality as trivial-sockets. It's also designed to be relatively easy to port trivial-sockets code over, and we've documented how to in the manual.

The major obstacle that I'm struggling with at the moment is implementation-specific issues to integrate with the streams system so that we can present the network as an actual stream and not just a pseudo-stream. Sadly, gray streams are not acceptable for performance reasons.

Right now it half-works with openmcl; I'm studying sb-bsd-sockets to learn how to do it with sbcl. In the long run, once we have it working on a variety of implementations we want to put together a uniform interface and then get it pushed upstream into each Lisp implementation so that it won't break with future revisions.

As I say in the release notes:

This is an incremental improvement on the first release, from yesterday. It does more intelligent detection of constants that can have multiple names on different systems, almost eliminating the use of #+ feature-checks in unixint.lisp. It also has a new subsystem for figuring out how to access errno, tested on our development platforms. The main purpose of the release is for our generous testers to check that errno and path-max work for them.

Tarball here.

Lucca and I have been collaborating on a project to put together a portable sockets API, using CFFI. I spent last night writing a CFFI-Grovel, which does the same thing as sb-grovel, but portably: Takes care of the grunt work of figuring out types, constants, and structure layouts.

This is very preliminary, but we've been persuaded to put together a release so everybody can try it out. Have a look at the manual and general description, then grab the source. Right now, there is no "real" webpage, because we'd rather put our efforts into finishing up the incomplete parts of the code.

Comments are very welcome!

When I learned that CL-SQL supported SQLite, I was excited. See, sometimes you don't want to write a big client-server app; sometimes you want to save your data to a document that looks like a regular file, so that users can move and copy it by all the means they're used to.

But that was only the first half of the pipe-dream. Relational databases from an OO language are a pain; object databases have debatable scalability, but are far easier to work with. So naturally my question was, does Elephant's new SQL backend (written by Robert L. Read) work with SQLite?

And the answer was no. But a long night of debugging later, the answer is now yes! At least, for me. I've sent my patch off to the dev list, and hopefully it will be included into the final.

I've been dusting off the GLR parser again (described here). I decided it had gotten too obfuscated to maintain properly, but I didn't want to rewrite the code from scratch...

So I took to heart Joel Spolsky's old advice about refactoring versus reimplementing. And guess what? It worked. I fixed the bugs that had stumped me and I think I have a grasp of what to do about the efficiency issues. I've also been working on making the code nicer so that it can be embedded in other projects easily.

And then, of course, real life came along and forced me to put it down for a couple weeks. Oh well. The door is open to more progress now.

It's been forever since I posted. So, in keeping with Jakob Nielsen's list of blog-design mistakes, I'm going to start posting on a regular basis, every other week. I also migrated from Bosxom to Movable Type. The big benefit is that comments now work; also, it no longer looks hideous.

I have an implementation of Tomita's GLR Algorithm, which is able to parse any context-free grammar, including ambiguous ones. Earley's Algorithm seems to be getting more attention lately, and I'm not sure why, since as far as I know it's strictly less efficient. What I am attempting to do is expand my framework to be a scannerless parser, which would allow rapid development of parsers for just about anything, in native Lisp.

Tomita's Algorithm, like the traditional yacc LALR(1) algorithm and its variants, precomputes a parse table which drives the actual parse. The difference is that the parse stack isn't really a stack at all, it's a DAG, which means that when you there are shift-reduce or reduce-reduce conflicts, the parser is able to nondeterministically try both branches. A backtracking recursive-descent parser can do that, too - but here there's no exponential blow-up.

A scannerless parser essentially combines the roles of lex and yacc. Of course, the reason lexical analyzers have typically been separate is that most programming-language grammars are not context-free. Happily, GLR lends itself handily to extensions which address this. Right now, I have implemented one such extension, reject productions, and I am thinking about whether I should also implement follow restrictions. Reject productions say "don't allow anything that matches this," and follow restrictions say "don't allow matches which are followed by this character."

I'm pretty much done with adding features now, and am working on packaging and polishing. I still consider the code alpha-quality, and I'm going to need to clean it up quite a bit before I put it up...

This brings us to the subject of compile-time evaluation. Since generating the parse table does take some time, of course it needs to be pregenerated and saved for the next invocation of the parser. Traditional parser generators do this by outputting a new source file which contains the entire runtime component of the parser, as well as the parse table. For example, your input file would be grammar.g; the generator would output grammar.cpp, which you would then compile into grammar.o.

Although I haven't entirely convinced myself that there's anything wrong with doing things that way, with Lisp's introspective capabilities, it seems as though it ought to be able to do better than this. Couldn't the parse table just be computed when you compile the file, and stored into the fasl?

I'm not sure exactly how to do that, though. It looks like eval-when doesn't help with that; it can compute the parse table at compile time, but I don't see any way to save it for later loading. The best I can think of is to define a macro which does the computation at macro-expansion time. Does anybody else have suggestions?

Here are some papers which explain more about this type of parser:

"An Efficient Augmented-Context-Free Parsing Algorithm" -- Tomita 1987, 16 pages

"Scannerless Generalized-LR Parsing" -- Visser 1997, 54 pages

"Disambiguation Filters for Scannerless Generalized LR Parsers" -- Brand, Scheerder, Vinju, and Visser 2002, 15 pages

Somebody at Microsoft Research named Todd Proebsting, who doesn't seem to have his own blog, recently got a mention on Lemonodor for a talk he gave a few years ago, "Disruptive Programming Language Technologies". I went to read the Powerpoint slides because it sounded mildly interesting, and, well, it was mildly interesting. But I realized as I got further in that it says something I really enjoyed hearing:

Optimization, he says, is the least important open problem in language design. The most important is increasing programmer productivity.

I doubt any Lisp-using readers I have need to be told why that's a feel-good kind of thing. Personally, my aspirations have always been to build better metatools, but I've sometimes felt that that's just my private obsession, not something that most programmers would see the value of. Well, it may be true that most programmers won't see it as important, but it was nice to be told that it was this once!

Within the next few decades, I expect compilers and development environments to change beyond anything we would recognize. The most immediate change I predict is better support for cross-language development. With things like .NET out there, this is no longer a particularly insightful thing to say, but it's a trend worth pointing out. Notice that every single new language has to get an FFI really quickly if it's going to be widely used. That points to the importance of the problem.

Since it is important, of course, we are going to solve it. I think ultimately, we won't even think of programs as being written in any particular language, because we will be able to mix and match so freely.

Of course, that would be a maintenance nightmare without some sort of help to the poor programmer who is fluent in fewer than ten thousand programming languages; but I further predict systems to make it easier to cast code into familiar terms. Just translating the surface syntax into any chosen language is really quite easy for most languages; the only exceptions I can think of offhand are APL and Perl.

Of course, even with automated syntax translation, APIs and semantics would continue to be unfamiliar. A challenge to any readers I may have: Propose an off-the-wall idea for how to help programmers use APIs and semantics they have not previously encountered.

It's probably safe to regard semantics as a small set of options: Explicit deallocation; garbage collection; or no dynamic memory at all. Lexical variable scope; global scope; dynamic scope. Pure-functional, or not. Closures available, or not. Pointer aliasing allowed, or not. Sure, every few years something new gets invented, but for now it's still a fairly small set. Isn't it? Am I overlooking anything?

Oh yeah, here's those Powerpoint slides, for the curious.

This is a very new blog! Let me know you're reading!

I wrote up a short piece on how to configure Apache to use a Lisp webserver for specific files and directories, using mod_proxy and mod_rewrite. As I say therein:

There are several webservers implemented entirely in Lisp. You may be excited about the prospect of using them. But, if you've tried using them for part of a large site, you may have realized that Apache does certain things better, and it would be nice if you could somehow use Lisp for only part of the site. Or, you may be advocating that Lisp be used for a part of your corporate website, but the company understandably does not want to redo the entire site to use a Lisp server. Perhaps it's just easier to sell people on the idea of Lisp if it fits in with other things instead of trying to replace everything.

I would be remiss if I did not make note of mod_lisp, which has similar goals; but my approach requires no additional software, just Apache and your Lisp webserver. Also, it's questionable whether mod_lisp adds anything.

If anyone tries this, I'd love to hear from you. How did it work? What are you using it for? Anything I could have clarified and didn't?

I'm creating this blog as a place to write my ideas about computing, of which I have a plethora. I found that posting them to my personal Livejournal was a bad idea, because almost all of my readers simply skipped such posts, and it just wasn't a good way to invite discussion. Hope y'all have fun reading it!

I am planning to submit it for inclusion on the Planet Lisp blog aggregator, when I have a few more posts and a better feeling for how often I'm going to update it.

I hope you find CS Thoughts thought-provoking and entertaining, not necessarily in that order. If you do, you can return the favor with your comments!