Speculating about Future Development Environments

This series of essays, making up a sort of ode to the command line, got me thinking again about the user interface we face everyday, our software development environments. Remember when you were a kid and spent time imagining the ideal car — it should be able to fly and transform into a submarine —, or the ideal house — it should have an underwater garage to park the plane-submarine-car? No? Perhaps I was a weird kid. But that's what this post is about, conceiving what would be my ideal programming environment.

I can't get into this subject without first discussing the big rift: editors versus IDEs. I have to say I'm not a member of either faction, I love vim, I get a kick of learning new tricks (did you know that if you accidentally delete something and lose a previously yanked text, you can get it back with "0p ?), but I'm also a happy user of modern IDEs. In fact, I don't know why we have these factions. I suspect the origin is in the name IDE, Integrated Development Environment. It suggests a large piece of software integrating lots of tools to be used in the making of software, stuff like stupid code generation wizards and the always dissatisfying WYSIWYG interface designers. This perspective isn't entirely misleading, but I think it fails to capture the most important aspect of IDEs: they are editors that understand the programming language being used. They analyse your code's with regard to syntax, semantics and type structure to be able to realise feats such as jumping from a call to a definition site, changing a symbol definition and all it's uses, listing all lines of code that call a certain method/function, etc.

I see absolutely no reason why any programmer would forgo these abilities, but I see why many avoid current IDEs. The user interface is a mess of toolbars, menus, tabs, docked windows, popup windows of different sorts and gargantuan settings dialogs. Only after some time spent living in these strange environments one is expected to reap the benefits of working with a smart editor. In a way, the current crop of IDEs reminds me of the web just before Google appeared. To try to find anything with the search engines of the time was a guarantee of wasted time and probable frustration. The situation was so bleak that most of those search engines gave up on being starting points for the web and were becoming "portals", so users could stand a chance of finding what they were looking for within the sections and categories and subcategories of specifically produced pieces of information. Then came Google, with the right technology to extract relevance from the same sea of data everyone else was floundering on.

Aside from the immensely better ranking of search results, another striking difference was Google's minimalistic approach to user interfaces, showing little more than a search box and a list of results. That's where I'd start my ideal environment, a visually simple application to help the programmer focus on reading and writing code. Reading and writing code, not just text, and to be able to help with that, the application would need the code navigation and refactoring tools found in today's IDEs. The question then is, can this be done without all the cruft?

I think it can, and the way to do it is the same Google took to beat the competition: relevance and minimalism. All the housekeeping functionality we are accustomed to in current IDEs should be cast aside the in favor of a search centric interface. And there is plenty of housekeeping to get rid of: modules, windows, tabs, workspaces, perspectives, views, projects, buffers, editors, etc... As I envision it, a minimum productive environment would consist of a few tiled panes for editing code, with no tabs. The last part deserves a little explaining. The purpose of tabs is to show the currently opened files. My thinking is that whether a file is currently open or closed should be an implementation detail invisible to users. In order for this to work well, changes in files should be continuously saved and we should have access to a full version history (this is an old idea that is being picked up in Apple's latest OS).

As the ubiquitous file-tree view would be gone, all navigation would be done either directly — following a reference in a source file — or via search. The key to make this work is to go to extra lengths to perfect search relevance, perhaps taking contextual clues enriched by the abstract model of the code. For instance, a recently opened file that is close in the call graph to the file currently being edited would spring to the top of the results.

This minimalism isn't just an exercise in modernistic aesthetics; the problems caused by the litany of housekeeping features go beyond visual clutter. Each of those features generate work for the user, who has to spend time organising his environment and then remember where everything is. This is the kind of work we should delegate to computers, the user needs to deal only with the content he is currently working on and ask the environment for what he needs next. Don Norman makes a much better argument than I ever could for search based user interfaces in his column for the CACM.

Ok, so much for editing. What about the rest of the services of a modern IDE? Stuff like version-control integration, build systems, debuggers, test runners, console runners, diagram editors, kitchen-sink explorers, etc. My answer is that while many of these tools benefit from graphical user interfaces, they don't necessarily need to be in the same application as the editor. Some of them could maybe reuse some code from the IDE, but there is no need for them to share window space with the code.

So ends my wish-list for a future development environment, a fitting time to restate this is not a prediction, as I see no movement in this direction. Quite the opposite, actually, as most IDEs keep sprawling in ever larger feature matrices.

Experience report: Ruby

For a long time I've been curious about how the supposed benefits and liabilities of programming in a dynamically typed language¹ actually play out in practice. I'm now getting a chance to find it out, since I'm involved in a largish project using Ruby (lots of Rails plus some Sinatra and a couple of random daemons). My professional background has largely been in Java, but I spend a good chunk of my free time learning about different programming languages (and some PL theory), mostly on the static side of the fence. Anyway, here are some notes:

The big question is whether dynamic typing allows for more bugs to pass trough. My answer to this has to be put in context: we are a medium-sized team (between half and a full dozen of devs), all experienced in, and practicing, TDD. As with the rest of this blog post, I don't have hard data to show, but my impression is that the unit tests do indeed catch all the bugs that the Java type system would help to catch; with the caveat that it's often harder to pinpoint the source of a regression. I don't have much real experience in languages whose type systems help to enforce strong guarantees², but I would imagine they would catch a larger fraction of the bugs that are caught by the unit tests, while not really avoiding further bugs. The reason is twofold: firstly, in my experience, many of the bugs are found in the interaction of separate pieces of software (such as Javascript in the browser talking to the web server), secondly, even when the bug is located within process boundaries, it is most often related to a forgotten invariant than to breaking a established one. But that's all conjecture.

On the matter of productivity, the abstraction mechanisms offered by the Ruby language help to structure code and avoid repetition and that's certainly noticeable in comparison to Java (though, in my opinion, not in comparison to Scala, and probably also not in comparison to Smalltalk, ML, Haskell or Oz). That gain is offset to a point by the lack of refactoring tools. There are some who argue that those tools are made necessary by the verbosity of Java, and aren't needed in better languages. That's nonsense. If a language has an abstraction mechanism, that mechanism is used to define an abstraction and to use the defined abstraction elsewhere. If we then want to change the abstraction we must change code at the use sites as well, and that's where such tools can help. This is all so obvious that I find it almost silly to have to write it down.

Many of the abstraction gains in Ruby come from metaprogramming techniques. I'm not completely sold that they are necessary to achieve the level of abstraction attained, and I'm sure that they hinder readability. It's much easier to gain a footing in a library written with straighforward composition mechanisms (be it functions or classes and objects) than in a mess of Strings and calls to define_method and cousins.

While on the subject of abstraction, I have to comment on Rails. It's a mature web framework that does a masterful job of making the common cases easy. This is a much harder feat than it sounds, as we can glean from the failure of JSF, WebForms and similar unsuccessful attempts to abstract the web. The hidden cost of the bargain is when we get to the uncommon cases. It isn't so much the case that there are specific application features that are hard to code in Rails, though it happens, but that the structure of the code that Rails mandates sometimes isn't a good fit to the problem being solved.

I'm sounding a little negative, so let me balance it out by saying that I believe Ruby and Rails were as adequate choices for our project as any. The main reason is the availability of decent quality libraries and tools (refactoring and code navigation notwithstanding), specially in the often overlooked front of deployment and configuration management.

Apologies for an opinionated blog post.

1. Unittyped, for the pedantic.

2. Such as Agda or Coq, or some styles of programming in ML, Haskell or Scala.

Unsolicited and uninformed rant about Software Engineering Research

Jorge Aranda interviewed industry professionals regarding their opinions of software engineering research. Needless to say I wasn't asked, and also needless to say I'll shoot my mouth off anyway. As with most of the interviewees, my knowledge of software engineering research goes little beyond Brooks' classic papers. Despite my reduced exposure, the impression I have of S.E. research is less then stellar. I have a feeling that a lot of the research tries to be too helpful, and ends up lacking in enlightenment.

It seems the goal is often to evaluate a certain practice in order to recommend it or not to the industry. I find this goal suspect for a number of reasons: the definition of success is too context bound (e.g. some environments value fast feedback while others value more a well crafted user experience), the measures of project success may mislead, some practices can help at certain parts of a project and hinder at different parts, people respond differently to different incentives, personal preferences play a major factor, etc. Perhaps Cockburn's "Characterizing people as non-linear first-order components in software development" helps to explains my uncertainty.

I'm not writing just to complain that research I'm not familiar with goes about the wrong way. I'm also writing to offer unsolicited advice regarding what the aforementioned research should do. In a nutshell, I would like to learn more about programmers, teams, and culture. Research about programmers could investigate stuff like correlation between psychological attributes and professional attributes (e.g. are people with certain personality profiles more attracted to some kinds of software than others), differences in thinking processes between programmers doing different kinds of work, differences in the work of programmers that have been doing the same thing for an extended amount of time versus programmers with a more diverse career, etc.

Research about teams could look into communication patterns (how and when they tend to take place, not which are more "effective" than others), how people react to homogeneous versus heterogeneous skill-set compositions, how the amount of time spent as a team affects the work (not just wrt productivity), at what team sizes people tend to jell into cliques (and what other factors are in play in the process), how the physical layout affects the communication (again, not trying to find "optimal placements"), ...

The "culture" part may seem a strange focus for software engineering research, but I would enjoy reading anthropological accounts of the differences an similarities between behaviours of the different programming cultures: systems programming, the smalltalk/OOPSLA/XP culture, lispers, modern large-scale web development, game developers, etc. The very question of whether it is meaningful to apply the word "culture" here would be worthy investigating.

Anyway, the particulars aren't important, I just want to emphasize the most interesting research, from my point of view, is the one that tries to understand, not merely to optimze.


Edit: Jorge Aranda commented to point out the interviews were conducted by a team including himelf, Margaret-Anne Storey, Marian Petre, and Daniela Damian.

Book review: Growing Object Oriented Software Guided by Tests

I’m a curious kind of guy. It is with some surprise, then, that I catch myself re-reading a technical book: Nat Pryce and Steve Freeman’s Growing Object Oriented Software Guided By Tests. It’s a very practical book, stuffed with code and useful advice, but it’s also more than that.

The first section of the book gives an overview of the basic principles of object-oriented design and test-driven development; not much is new but everything is clearly explained. The third and final section is a potpourri of test-centric techniques to identify problems and improve code quality. But the meat of the book is in the second section, a long walkthrough the development of a sample program. It’s a much larger example than usually found on programming books, tackling thorny issues such as asynchornous inter-process communication and end-to-end testing of GUIs. And it reads like a real software project, there are missteps, refactorings are rolled-back, some of the work is almost clerical, but then there are the great design breakthoughs, the elegant ideas that simultaneously solve several difficulties, the joy in seeing the product grow. It felt like reading a good novel. And that’s kind of what it is, a programming book that is a narrative, not an exposition. Besides making for a good read, the narrative aspect of the book is important for showing how modern object-orientation practice takes place.

OO criticisms nowadays are a dime a dozen, and though they sometimes present good points, often many of the arguments are directed at practices that aren’t so common, or at least that shouldn’t be so common. For instance, inheritance hierarchies, rampant mutable state, and patternitis (the FactoryFactory syndrome), are not indictments of object orientation, just symptoms of bad programming practice. One of the great things about GOOS is that it provides a great example of actual non-trivial object-orientation. And the same goes for test-driven-development, any abstract discussion of the benefits of TDD is bound to seem hand-wavey; this book helps to ground the understanding in real coding, done step by step in front of the reader’s eyes.

Anyway, I've read some pretty good technical books this year, but this one was the best.

Where scala left me wanting

n the past few years, I've grown to enjoy more and more programming in a functional style. Even the java code I write nowadays is mostly free of mutable state. But before that, I spent a good chunk of time learning about object orientation and found some great ideas there. Ideas that can carry over to the functional world. One of them is Domain Driven Design, where we strive to have the symbols in the code respresent conncepts in the domain. Of course, I'm oversimplifying DDD, but the ideal of a bijection between the reality our software models and constructs in a programming language is one I believe we should maintain in many, perhaps most, applications.

A natural consequence is that our systems architecture tends to resemble an onion, with the domain model in the center and code to make it interact with the rest of the world surrounding it:

I've labeled the code that interacts with the rest of the world "Adapters", as it should do little more than adapt external representations into concepts the domain model can understand. This architecture is not in any way novel. In fact, it is a version of Alistair Cockburn Hexagonal Architecture. In the common case where the only interactions the domain model has with the rest of the world are through a Database on one end and an User Interface on the other, we've just described the 3-layer architecture that was so popular in the 90s. Some examples of adapters are — to increase our shot at the buzzword bingo — Web MVC Controllers, Repositories or DAOs, message endpoints, GUI listeners, etc., you get the picture.

Since Scala is a truly hybrid language, marrying quite elegantly OO and FP, it would appear to be the perfect vehicle to write this kind of software. I've found this premise to be correct, for the most part. As you might have guessed, there is an exception; (If there weren't, I wouldn't be writing this post, would I?). The problem is in the code for the adapters. As mentioned, they tend to be simple shims, translating some external representation into domain objects. Coding such translations every single time for every adapter instance in every application is very dull. We desperately need our old friend, lady abstraction, to help us out here. In the Java world, she lends us a hand though dynamic reflection, allowing the construction of objects and invocation of methods to be done generically. Unfortunately, Scala has no reflection API, so we have no alternative but to resort to Java reflection, in a sense reverse-engineering the Scala compilation process.

This isn't a major gripe, as Scala constructs tend to map to Java constructs in a straightforward fashion. Also, rumor has it that a Scala reflection API is being developed for 2.8.1. But that's only half the story. Powerful as it is, dynamic reflection alone is not enough to solve the Adapter problem once and for all. We often need to parameterize the translation in some way. For instance, when translating objects to database rows, we must discover the Column names corresponding to object properties. In many cases we can trust convention, for instance we could expect the columns to match exactly the names of the properties. But in other cases there is no option but to somehow configure the translation with an explicit mapping.

 In the Java world this kind of thing used to be done with verbose and annoying XML configuration files. As the language evolved, annotations were introduced, and they are now the main way to configure such translations.

Annotations are a great improvement over external XML files, but they can't be the end of the story. There is the relatively minor issue that annotations pollute the domain model we strive so much to keep clean and organized, specially when the same domain objects will be active in many adapters. Beyond that, there is the larger problem that annotations are just metadata. Sometimes we want to parameterize our adapters in richer ways. Take, for instance, the proposal for a typesafe API for database queries to be added to the new version of the Java Persistence API. It requires a special pre-processor to generate a metamodel that can be used to parameterize a database adapter. 

A better and more generic approach would be to have the language itself provide this metamodel: a kind of static reflection. I would love to see something like this in Scala. Some time ago I toyed with the idea of writing a compiler plugin to provide a static meta-model of Scala classes, but apparently compiler plugins have issues with non-transparent code generation. 

Postscript. As with all matters regarding programming languages on the web, we must tread lightly. I am not ranting against Scala, in fact I rather enjoy it, as can be gleaned from some of the previous posts. I am only relating a very specific domain where I believe the language can be much improved. 

On Exceptions, Mythological Monsters and Household Appliances

Philosophy aims at the logical clarification of thoughts. Philosophy is not a body of doctrine but an activity. A philosophical work consists essentially of elucidations. Philosophy does not result in 'philosophical propositions', but rather in the clarification of propositions. Without philosophy thoughts are, as it were, cloudy and indistinct: its task is to make them clear and to give them sharp boundaries.
Wittgenstein

If language is not correct, then what is said is not what is meant; if what is said is not what is meant, then what must be done remains undone; if this remains undone, morals and art will deteriorate; if justice goes astray, the people will stand about in helpless confusion. Hence there must be no arbitrariness in what is said. This matters above everything.
Confucius

Shit happens
Forrest Gump?

Exceptions are a controversial matter among programmers. We disagree about how to deal with them. We disagree if the should be checked by the compiler. We even disagree if they are useful at all. Sometimes controversy arises from real disagreement: I may argue that test-driven-development is the best way to write software while you might say that formal proofs are the way to go. Both sides can productively debate the issue and present thoughtful arguments either way. I posit that the polemics surrounding exceptions are not of this sort. They are, rather, brought about by the inherent confusion in the mechanism. Exceptions are the Hydra of current programming languages, under a simple banner — to handle exceptional conditions — they mix several mechanisms.

The heads

Let's begin with the humble throw term, which is a sort of procedure exit statement, causing the current routine to throw his hands up in the air, call it quits, and immediately return to the caller. The only other statement with a similar effect is return. Of course when calling return, we must return something! In statically typed languages, this something must have a type: the return type of the function or method. Exceptions muddle the picture, if we think of a throw as a kind of return then what is the type being returned? There can be many answers to this question, but let me put forth a proposal: the "return type" of a "throw" is, for all intents and purposes, a variant type, and that is the second head of the Hydra.

This warrants a small digression. Most functional languages have a way of saying "this type FOO is really either a BAR or a BAZ". So every place in the code that is handed a FOO must check whether it is a BAR or a BAZ and deal appropriately. We call FOO a variant type, and BAR and BAZ are its variants. The syntax for the check is called called a type-case¹.  And where have we seen syntax for checking the type of something and then acting on it? Catch clauses!

But of course many languages lack the concept of variant types. I'll argue that in practice, exception handling code in effect greenspuns variant types. First, we can observe that in languages with checked exceptions the declaration of the exceptions a method can throw — the throws clause in Java — is basically a variant type declaration in disguise. Even when working with unchecked exceptions variants lurk behind, encoded through subtyping. Just notice how little polymorphism is exploited in exception class hierarchies. Different exceptions are given different types for the sole reason of being distinguished in catch clauses. Just like variant types.

Ok, time for the next head of the monster. We'll continue to look at the catch clause, but this time we'll look at it as a control structure. As such, it is a strange beast, neither a repetition nor purely a decision structure. Once an exception is thrown, the calling method will stop everything it was doing and jump straight to a catch block. In a way, every exception-throwing method call can be seen as a regular call paired with a conditional jump. Dijkstra may not be rolling in his grave, but he is probably feeling a little tingly.

Since we've talked about control flow, let's direct our attention to the mischievous head of the Hydra: stack unwinding. After a throw happens, if the calling method doesn't care to handle an exception, his caller will have to bare the burden or leave it up to his caller, recursively unwinding the stack until some frame decides it can be bothered to catch the exception and deal with it. This is a pretty powerful feature, sometimes it's even exploited to simulate continuations in languages that lack them.²

One more head left: call stack introspection. Besides unwinding the call stack, exceptions may reify it. It is a limited form of reflection, typically used only to print stack traces to a log someplace where they can be attentively ignored. Common lore says capturing the stack frames is an expensive operation, and so should be executed only on, well, exceptional circumstances.

Exceptional Situations

The confusion bus doesn't stop here, I'm afraid. All the complexity we've examined so far is justified under a deceptively simple banner: to handle exceptional conditions. Hidden in these four words is the vast range of situations that may be interpreted as "exceptional". Like any piece that deals with the subject of exceptions, we can't escape the temptation to suggest a taxonomy. At least this one is short, these are the three main categories of "exceptional situations":

• Programming error: A traditional example would be code trying to index an array beyond it's bounds. Ideally we would design our abstractions so as to make it impossible to commit such coding errors. From formal grammars to dependent type systems this ideal has been driven much of the research on programming languages over the years. Unfortunately, at least for now, we cannot always evade the need for run-time checks.
• External failure: Something bad happened to some component not within control of the application. Examples: network cable cut, disk crapped out, database died, sysadmin was drunk and lost a round of UNIX Russian roulette.
• Invalid user entry: "If an error is possible, someone will make it. The designer must assume that all possible errors will occur and design so as to minimize
  the chance of the error in the first place, or its effects once it gets made. Errors should be easy to detect, they should have minimal consequences, and, if possible, their effects should be reversible." This is from Don Norman's superb The Design of Everyday Things. The book is so good I'm going to slip another quote in here: "The designer shouldn't think of a simple dichotomy between errors and correct behaviour; rather, the entire interaction should be treated as a cooperative endeavor between person and machine, one in which misconceptions can arise on either side"

Let's not forget we are classifying "exceptional situations", not exception types. Here is an example that I hope will make the distinction clear. Take the always popular FileNotFoundException, that is thrown when application code attempts to read a file and the file isn't there: where do we place it on our little taxonomy? Imagine the application is a boring old word processor and the user types in the wrong file name. That obviously is a case of invalid user entry. Now, consider the case of a mail server attempting to open the file where it stores all queued outgoing mail. This file was probably created by the server application itself, maybe during the installation procedure, and its absence would be an indication that something is very borked in the server environment. The same File Not Found event is now clearly an external failure. This matters, because the strategy used for dealing with the three categories is different, as we'll soon see.

The Inevitable Cross Product

Time to look at how the heads of the hydra fare up to our new categories:

Programming errors are, by definition, unexpected. If we aren't expecting them, there is no point in writing code to handle different exception types. What could you do when you know some code tripped a NullPointerException that's different from what you would do if the offence was an IndexOutOfBounds? In other words, the variant types head is useless here. Exceptions triggered by programming errors are usually caught in a generic stack frame, in the bowels of infrastructure code — through the magic of stack unwinding — and dealt with by logging the error type, message, and stack trace. After logging there is little left to do but to roll over and die. In multi-user systems, killing the whole process is not very polite, so we might nuke just the current work item. This is the gist of the fault barrier pattern. Call stack introspection is a very valuable aid to pinpoint the bug. So valuable that I go as far as point its absence as the single most important reason to avoid C for teaching programming to beginners.

Practical strategies for dealing with external failures aren't too different. We tend to exploit stack unwinding to catch the exceptions in a frame deep in infrastructure land and log them in pretty much the same way. But, every so often, we might want to code a different action, such as to retry the operation or to trigger a compensation transaction. In these relatively rare cases it might be useful to deal distinctly with different variants. Another aspect where external failures are distinct from programming errors is that call stack introspection isn't so helpful: if the failure is external, the exact place in the code where it was detected is of little significance. Of course, it is important that the exception carries enough information to enable a system administrator to find the origin of the fault by just looking at the log.

Invalid user input is a different beast altogether. Variants are definitely useful; after all it's important to let the user know if their request was over quota rather than under quota. On run-of-the mill interactive OLTP systems, we handle invalid user entry by providing immediate feedback on what was wrong. But not every case is so simple: in some circumstances it may be important to identify suspicious input and raise security flags; in others, we could offer suggestions for the user to correct the data; in asynchronous interactions, we might want to proceed with the action substituting a default or normalized value for the invalid input; and so on. Altogether, there is a wide range of handling strategies, mostly domain-specific. This is a key observation, as not only handling strategies, but also the detection and the very definition of what values are invalid, are full of domain specific decisions. I would go as far as proposing that the controversial domain exceptions all boil down to cases of invalid user input.

What about the other heads of the monster, are they helpful for invalid user inputs? Call stack introspection is clearly not needed. Moreover, the performance costs of reifying the stack can be detrimental for the not-so-rare instances of invalid input. This isn't just premature optimization, as the conscience of these costs can prevent users from even considering exceptions for the domain logic. What about stack unwinding? I'm afraid I don't really have an answer for this one. On the one hand, domain specific handling logic seems to fit naturally in the calling method. On the other paw, if all we do is inform the user that something happened, a generic code further down the stack is awfully convenient.³ 

So what?

We've seen that the feature by the name exceptions is really a hodgepodge of language constructs. Beyond the technical aspects, the very raison d'être for the feature is also muddled. It's as if some crazy nineteenth century inventor decided that since people need to be protected from the heat and, hey, food needs cooling to be preserved as well, why not make buildings like large refrigerators? Just keep the food near the bottom where it's colder, the humans near the top where it's not freezing, and presto! His solution is absurd, but that is because the problem itself is ill-formulated: "to keep stuff cool" is not an useful problem statement.  "To deal with exceptional circumstances" isn't either.

I don't know if the above ramblings are of any practical significance. I certainly don't have any concrete proposal to make things better. My only hope is that language designers will keep in mind context and necessity when devising new constructs.

¹ In languages where pattern matching is available, it subsumes type-cases. In fact, one way of thinking about pattern matching is that it is a sophisticated form of type-case.

² Some languages without full support for exceptions do offer a feature called long jumps, that is very close to what I'm calling stack unwinding.

³ To further complicate the issue, languages offering variant types usually allow different program structuring techniques. For instance, instead of a direct chain of method calls we may thread a monad through a series of functions.

Type-safe printf in scala ‽

All the excitement surrounding Scala's next big release - 2.8.0 - reminds me of another big release a few years ago. Java 5 was poised to be a huge leg up for developers, long feature lists were all over the java blogosphere: generics, enums, enhanced for-loops, autoboxing, varargs, util.concurrent, static imports, and so on; in sum, a whole heap of goodness. But there were a few odd ducks hidden in the feature lists, most notably "an interpreter for printf-style format strings".

Anyone who has been around curly-brace-land long enough knows that printf (or java.util.Formatter) serves one main purpose: it's a poor substitute for built-in variable interpolation in the language. Unfortunatly Scala also lacks variable interpolation and there isn't much we can do about it: our BDFL has ruled. But, there is another use for printf, as a shortcut for applying special formatting for some value types. We can specify a currency format using the string "%.2f USD", or a short american date format with "%tD". We can even go wild and use the two together:

In Java:

String.format("%.2f USD at %tD", 133.7, Calendar.getInstance());

In Scala:

"%.2f USD at %tD" format (133.7, Calendar.getInstance)

This snippet is saying that 133.7 should be formatted as a decimal with two digits after the point, and Calendar.getInstance() - the horrific Javaism for "right now" - should be formatted as a date, not a time. What always trips me up is that the order of the values must exactly correspond to the order of the format specifiers. It's a simple task, but my tiny little brain keeps messing it up. Let's see if our good friend the Compiler can help.

Formatters
The first step is to have our logic leave it's String hideout and show itself. So, instead of "%.2f", we'll say F(2)^*, and instead of "%tD" we'll say D(T). D and F are now Formatters:

trait Formatter[E] {
 def formatElement(e: E):String
}
 
case class F(precision: Int) extends Formatter[Double] {
 def formatElement(number: Double) = ("%."+precision+"f") format number
}
 
import java.util.Calendar
abstract class DateOrTime
object T extends DateOrTime
object D extends DateOrTime

case class T(dateOrTime: DateOrTime) extends Formatter[Calendar] {
 def formatElement(calendar: Calendar) = dateOrTime match {
   case Time => "%tR" format calendar
  case Date => "%tD" format calendar
 }
}

Chains
Next we tackle the issue of how to chain formats together. The best bet here is to use a composite, very similar to scala.List ^***. We even have a :: method to get right-associative syntax. This is how it looks like:

val fmt = F(2) :: T(D) :: FNil

And this is the actual, quite unremarkable, code:

trait FChain {
  def :(formatter: Formatter) =
    new FCons(formatter, this)

  def format(elements: List[Any]):String
}

object FNil extends FChain {
  def format(elements: List[Any]):String = ""
}

case class FCons(formatter: Formatter, tail: FChain) extends FChain { 
  def format(elements: List[Any]):String = 
    formatter.formatElement(elements.head) + tail.format(elements.tail)
}

There is still a missing piece. Remember "%.2f USD at %tD"? We have no way of chaining the " USD at" to our formatters. This is what we want to be able to write:

val fmt = F(2) :: " USD at " :: T(D) :: FNil

The solution is simple, we overload :: in FChain:

trait FChain {
  ...
  def ::(constant: String) =
    new FConstant(constant, this)
  ...
}

and create a new type of format chain that appends the string constant:

case class FConstant(constant:String, tail: FChain) extends FChain { 
  def format(elements: List[Any]):String = 
    constant + tail.format(elements)
}

Cool, that works. But wait; what have we gained so far? The problem was to match the types of the formatters with the types of the values, and we aren't really using types at all. The remedy, of course, is to keep track of them.

Cue the Types
We want to check the types of the values passed to the FChain.format(), but this method currently takes a List[Any], a List of anything at all. We could try to parameterize it somehow and make it take a List[T], a list of some type T, instead. But, if we take a List[T], it means all values must be of the same type T, and that's not what we want. For instance, in our running example we want a list of a Double and a Calendar and nothing more.

So, List doesn't cut it. Fortunately, the great Jesper Nordenberg created an awesome library, metascala, that contains an awesomest class: HList. It is kind of like a regular List, with a set of similar operations. But it differs in an important way: HLists "remember" the types of all members of the list. That's what the H stands for, Heterogeneous. Jesper explains how it works here.

We'll change FChain to remember the required type of the elements in a type member, and to require this type in the format() method:

trait FChain {
  type Elements <: HList
  ...
  def format(elements: Elements):String
}

FNil is pretty trivial, it can only handle empty HLists (HNil):

object FNil extends FChain {
   type Elements = HNil

   def format(elements: Elements):String = "" 
}

FCons is somewhat more complicated, it is parameterized on the type of the head element, and on the type of the rest of the chain:

case class FCons[E, TL <: HList](formatter: Formatter[E], tail: FChain { type Elements = TL }) extends FChain { 
  type Elements = HCons[E, TL]

  def format(elements: Elements):String = 
    formatter.formatElement(elements.head) + tail.format(elements.tail)
}

We also had to tighten-up the types of the constructor parameters: formatter is now Formatter[E] ^** — so it can format elements of type E, and tail is now FChain{type Elements=TL} — so it can format the rest of the values. The Elements member is where we build up our list of types. It is an HCons: a cons pair of a head type - E - and another HList type - TL. We changed how to FCons constructor parameters, so we also need to change the point where we instantiate it, in FChain:

trait FChain {
  type Elements <: HList
   ...
  def ::[E](formatter: Formatter[E]) =
    new FCons[E, Elements](formatter, this)
  ...
}

Just passing along the type of the formatter and of the elements so far to FCons. FConstant has to be changed in an analogous way. This is it, now format() only accepts a list whose values are of the right type. Check out an interpreter session:

scala> (F(2) :: " USD at " :: T(D) :: FNil) format (133.7 :: Calendar.getInstance :: HNil)
res5: java.lang.String = 133.70 USD at 10/10/09

scala> (F(2) :: " USD at " :: T(D) :: FNil) format (Calendar.getInstance :: 133.7 :: HNil)
:25: error: no type parameters for method ::: (v: T)metascala.HLists.HCons[T,metascala.HLists.HCons[Double,metascala.HLists.HNil]] exist so that it can be applied to arguments (java.util.Calendar)
 --- because ---
no unique instantiation of type variable T could be found
       (F(2) :: " USD at " :: T(D) :: FNil) format (Calendar.getInstance :: 133.7 :: HNil)

Some random remarks

• The interpreter session above nicely showcases the Achilles heel of most techniques for compile-time invariant verification: the error messages are basically impenetrable.
• A related issue with this kind of metaprogramming is that it's just plain hard. The code in this post looks pretty simple (compared with, say, JimMcBeath's beautiful builders), but it took me days of fiddling around with metascala to find an adequate implementation.
• Take the above two points together and it's clear that we are talking about a niche technique. Powerful, but not for everyday coding.
• Since I've mentioned the word coding, the FChain structure showed here looks like a partial encoding of a stack-automaton. Stack automata can represent context-free grammars, if I remember my college classes correctly. That said, I don't see any particular use for this tidbit of information.
• Since this is random remarks section, I'll randomly remark that we can implement the whole thing without type members and refinements. Just good old type parameters in action.
• Since it is possible to use nothing but type parameters, and Java has type parameters, would it be possible to implement our type-safe Printf in Java? Quite possibly, a good way to start would be with Rúnar Óli's HLists in Java. Just take care not to get cut wading through all those pointy brackets. 
• A powerful type system without type inference is useless. Quoting  Benjamin Pierce: "The more interesting your types get, the less fun it is to write them down".
  

---

The whole code:

import metascala._
import HLists._

object Printf {
 trait FChain {
  type Elements <: HList

  def ::(constant: String) =
   new FConstant[Elements](constant, this)
  
  def ::[E](formatter: Formatter[E]) =
   new FCons[E, Elements](formatter, this)

  def format(elements: Elements):String
 }

 case class FConstant[ES <: HList](constant:String, tail: FChain { type Elements = ES }) extends FChain { 
  type Elements = ES

  def format(elements: Elements):String = 
   constant + tail.format(elements)
 
 }
 
 object FNil extends FChain {
  type Elements = HNil
  def format(elements: Elements):String = ""
 }

 case class FCons[E, TL <: HList](formatter: Formatter[E], tail: FChain { type Elements = TL }) extends FChain { 
  type Elements = HCons[E, TL]

  def format(elements: Elements):String = 
   formatter.formatElement(elements.head) + tail.format(elements.tail)
 
 }
 
  trait Formatter[E] {
   def formatElement(e: E):String
  }
   
  case class F(precision: Int) extends Formatter[Double] {
   def formatElement(number: Double) = ("%."+precision+"f") format number
  }
   
  import java.util.Calendar
  abstract class DateOrTime
  object T extends DateOrTime
  object D extends DateOrTime
  
  case class T(dateOrTime: DateOrTime) extends Formatter[Calendar] {
   def formatElement(calendar: Calendar) = dateOrTime match {
     case T => "%tR" format calendar
    case D => "%tD" format calendar
   }
  }
 }

* Or F(precision=2) thanks to Scala 2.8 awesome named parameter support.

** In fact, the previous untyped version didn't even compile, as Formatter has always been generic. Sorry for misleading you guys.

*** If you are unfamiliar with how Scala lists are built, check out this article.