The Horrors of Adopting Standards

Morals of the story

• Do the simplest thing that works;
• Adopting a standard is not necessarily the simplest thing that works, depending on how you do it;
• Look for reference implementation(s);
• Look for intelligible documentation;

The actual story

Some of you know that I have been working on the capstone project since September 2005. We're 10 people in the team and our project is about creating an extensible, web-based, software project management system, using Java Servlet and JSP technology. Eliott and I have been in charge of the task scheduler component since the very beginning.

Being the 2nd biggest standard-compliance freak in the team, I have decided that the task scheduler needs to support some existing calendar format. After some research, Eliott and I have agreed that the RFC 2445 iCalendar format would be the way to go for a few reasons:

• It was already supported by major calendar programs (e.g. Mozilla Sunbird, Microsoft Outlook, and Apple's iCal);
• There was already a Java library (iCal4j) for working with iCalendar files;
• The specifications for the format were publicly available.

We were quite excited because iCalendar seemed like a "big thing", so we quickly grabbed the iCal4j library and started experimenting with it, while it was still at version 0.9.16. Soon though, we began running into problems.

First of all, learning to use the library was a disaster. We had no clue where to start and none of us knew the iCalendar format very well. The only documentation that the library had was some semi-complete JavaDoc. There was no tutorial and the introduction page on the official iCal4j website had some strange sample code that didn't make much sense (e.g. one example involved creating an event, but then the sample code never showed how to insert that newly created event into the calendar object). This was our first mistake: to believe that the iCal4j library would be as good as an official reference implementation. This, combined with our inability to decipher the RFC 2445 specifications, made our life pretty miserable. Actually, I suspect the IETF is to blame for not writing intelligible specifications. =P

Once we began to get the hang of iCal4j, we produced some rough design, by deciding what functionality we needed and hence the classes for datasource and domain logic. We started coding and soon realised something fishy going on. Because the iCalendar file does not function like a database, everytime a single event needs to be read or added to the calendar, the whole iCalendar file needs to be read from disk, converted to a Calendar object, and once the event is added to the Calendar object, the iCalendar file is completely overwritten. This operating scheme is, of course, extremely inefficient and cannot be used with our system, where we expect hundreds of concurrent users. The fact that Mozilla Sunbird had switched its main storage from the iCalendar format to a simple database system confirmed our suspicion. Our solution was then to create a CalendarRegistry, which manages the reads and writes of iCalendar files, and at the same time provides protection against the concurrency issue of lost update.

The solution was far from perfect, however, as we found out soon after that the Java Virtual Machine has a limit on the amount of memory it can use, which seems to be 64 MB by default. Having the CalendarRegistry keep too many Calendar objects in memory was therefore also a bad idea.

In the end, we realised that the simplest thing to do was to just use a database for storage and add the functionality of importing from and exporting to iCalendar files. A few months of development time were lost, and we were sent back to the sketch board.