Jeff Mesnil
Weblog · About

In-Depth Snow Leopard Review by Ars Technica

September 2, 2009

As usual, the review of Mac OS X 10.6 by Ars Technica is awesome and full of gold nuggets.

It gives a nice description of the changes under hood. I was particularly interested by:

Now, I just need to find another pet project to play with all these new features.

"Read Lature": A Web Application to Save Articles and Read Them Later

August 25, 2009

As an exercice when I learn a new programming language or a new environment, I develop simple applications to get a feel on the language, the platform, its libraries, etc. When I lack ideas for these applications, I redevelop applications that I use.

In order to learn Clojure and Google App Engine, I have developed Read Lature, a web application to save articles and read them later (very similar to Instapaper)1.

The idea is simple: I want to keep reference on articles or blog posts that can be interesting but I don't want or don't have time to read them immediately. I do not use bookmarks for that (either using my browser bookmarks or delicious): my bookmarks are long-lived while I want to keep a reference to these articles only for a short period (and keeping them open in tabs does not scale).

To save articles, you just need to drag a bookmarklet to your bookmarks bar. When you find a good article or blog post, you click on the bookmarklet to save it. It will be added to Read Lature so that you can read it when you have some time. Once it is read, it is automatically moved to the archive. Articles can also be starred for further readings.

The second way to save article to Read Lature is to use Google Reader. Setup a custom link and you can send articles to Read Lature directly from Google Reader (as explained in the instructions).

I mainly use Read Lature to save articles and read them later on my iPhone when I am away:

Reading from the iPhone

The application is hosted on Google "Cloud" and leverages Google accounts for user credentials.
To try it, go to Read Lature, login using your Google (or Gmail) account, and follow the instructions.

The application is written using Clojure, Compojure and jQuery and is less than 500 SLOC (75% is Clojure code, the rest is JavaScript).
Read Lature code is released the code under Apache 2.0 License on GitHub.

I will write later a review of the book "Progamming Clojure", that I have just finished reading, and some thoughts about Clojure base on my experience with it so far.

  1. So similar in fact that I first named the project Instapapure for "Instapaper in Clojure" but I did not like it and renamed it Read Lature (for "Read Later with Clojure"). Naming projects is a dark art that I do not master... ↩

How Clojure Forces Me to Rewire My Brain (In a Good Way)

August 19, 2009

Part of my process to learn a new language is to write an application/library complex enough to be confronted with interesting challenges.

My current learning language is Clojure and I am writing a web application using the Compojure framework.

I was confronted with a simple challenge recently. I had a map containing HTTP parameters, all of the values being either strings or nil:

;; my HTTP parameters
user=>(def params {:a "true" :b nil :c "false" :d "3"})
#'user/params
user=>(params :a)
-> "true"
user=>(params :b)
-> nil
user=>(params :d)
-> "3"

If you are not familiar with Clojure:

  • the character ; is used for comment
  • user=> is the prompt where you type the code
  • -> is the result of the evaluation. For example (params :a) evaluates to the string "true"
  • :a is a symbol. It is used to identify keys in the map
  • evaluating (m k) returns the value associated to the key k in the map m.

I have a map with values which are either strings or nil.
But I want to persist these parameters in my storage with types. For example, values of :a and :b must be stored as booleans, and value of :d as a double.

; I want to persist:
(def persisted-params {:a true :b nil :c false :d 3.0})

How can I do that using Clojure?1

First, I want a function which will return the map with the value of a key replaced by a boolean if the value is not nil:

(defn to-b
  "replace the value of key in the map by a Boolean instance if the value is not nil"
  [map key]
  (if (not (nil? (map key)))
    (assoc map key (Boolean. (map key)))
    map))

Once you know the Clojure idioms, the code is straightforward: if the value of key in the map - (map key) - is not nil, returns the map with the key associated to the value - (Boolean. (map key)) -, else return the map unchanged - map -.

Another Clojure idiom:

  • (Boolean. value) corresponds to calling the java.lang.Boolean contructor with the specified value.

Let's try it:

user=> (to-b params :a)
-> {:a true, :b nil, :c "false", :d "3"}
user=> (to-b params :b)
-> {:a "true", :b nil, :c "false", :d "3"}
user=> (to-b params :c)
-> {:a "true", :b nil, :c false, :d "3"}

It works: when we call it on :a it changes its value from the String "true" to the boolean true. It does the same for :c but it did not change the value of :b which is nil.

Second step is to call this function, not for a single key, but for a list of keys and returns the map where all the values corresponding to the keys' list have been "booleanified". Clojure provides a basic function for that: reduce.

(defn booleanify [map klist]
  "Replace all the non-nil values in the map for the keys in klist by their Boolean equivalent"
  (reduce to-b map klist))

Starting with map as the initial value, reduce will return the result of applying to-b to to the map and the first item of klist, then apply to-b to that result and the 2nd item, etc. This works as we expect:

; booleanify nothing
user=> (booleanify params [])        
-> {:a "true", :b nil, :c "false", :d "3"}

; booleanify value of :a
user=> (booleanify params [:a])      
-> {:a true, :b nil, :c "false", :d "3"}

; booleanify values of :a :b & :c
user=> (booleanify params [:a :b :c])
-> {:a true, :b nil, :c false, :d "3"}

That's good: I have solved my problem for the boolean fields. But I wanted to also have a double type for :d.

How can I leverage what I have just done for the booleans?

This is very similar to my previous problem, the only changing part is the function used to modify the value. Let's rewrite booleanify using a more general function modify-values:

(defn modify-values [map fun klist]
  "Apply a fun to the non-nil values in the map for the keys in klist"
  (reduce
    (fn [m k]
      (if (not (nil? (m k)))
        (assoc m k (fun (m k)))
        m))
    map klist))

; booleanify call modify-values with a anonymous function to create booleans:
(defn booleanify [map klist]
  (modify-values map #(Boolean. %) klist))

Another Clojure idiom:

  • #(Boolean. %) is an anonymous function which invokes the java.lang.Boolean constructor with a parameter

This is "almost" plain english: to booleanify a list, we modify the values in map and replace them with Boolean instances for all the keys in the list (although the name modify-values is ill-suited: the original map is never modified, instead the function builds a new map from the content of the original one).

It is now straightforward to "doublify" :d:

(defn doublify [map klist]
  (modify-values map #(Double. %) klist))

This works as expected:

user=> (doublify params [:d])
-> {:a "true", :b nil, :c "false", :d 3.0}

I like the way modify-values is written, it almost reads like English2: accumulate - reduce - in the map the association - assoc - of the key - k - and a new value - (fun (m k))) - or the unchanged map - m - if the value is nil, and do this for all the keys in the list - klist -.

Simple, isn't it?
...
Alright, I admit it: this is a lie: This is not the way I solved this problem and found the final code.

I would have followed this reasoning if my brain was used to declarative languages, but that's not the case. I'm so accustomed to think imperatively (using Java) that I was not able to figure out how I want to solve this problem "functionally".

What I really did was look at the source of a function with a similar behavior, rename-keys, which was returning a map with some renamed keys. I realized this was the same kind of a behavior than mine, except that I wanted to "rename" values rather than keys. I studied the function and when I understood it, I was finally able to know how I wanted to write booleanify.

While I found the final code elegant and simple, writing it was anything but straightforward.
It seems this is my major obstacle to use Clojure. The language is good, the documentation is good, the book - Programming Clojure - is a great introduction (I plan to review it later) but my brain is not good at thinking declaratively. I need to rewire my brain to adapt it to solve problems using functional languages such as Clojure.

I rarely think "declaratively" in my daily job using Java. I started thinking this way when I learnt Ruby and Javascript (closures and functions as 1st class-citizen). With these languages, I am still able to fall back to imperative statements if I was not able to solve a problem functionally (or if that does not make sense to do it that way).
With Clojure, it is more difficult.

To complement this exerice, I wrote a solution in Java too:

public class MapTest extends TestCase {

   public void testMap() {
      Map params = new HashMap();
      params.put("a", "true");
      params.put("b", null);
      params.put("c", "false");
      params.put("d", "3.0");

      assertEquals("true", params.get("a"));
      assertNull(params.get("b"));
      assertEquals("false", params.get("c"));
      assertEquals("3.0", params.get("d"));

      Map booleanified = booleanify(params, "a", "b", "c");
      assertEquals(true, booleanified.get("a"));
      assertNull(booleanified.get("b"));
      assertEquals(false, booleanified.get("c"));

      Map doublified = doublify(params, "d");
      assertEquals(3.0, doublified.get("d"));
   }

   private static Map booleanify(Map params, String... keys) {
      return modify(params, new Modifyable() {
         public Object modify(Object original) {
            return new Boolean(original.toString());
         }
      }, keys);
   }

   private static Map doublify(Map params, String... keys) {
      return modify(params, new Modifyable() {
         public Object modify(Object original) {
            return new Double(original.toString());
         }
      }, keys);
   }

   private static Map modify(Map params, Modifyable fun, String... keys) {
      for (String key : keys) {
         Object value = params.get(key);
         if (value != null) {
            params.put(key, fun.modify(value));
         }
      }
      return params;
   }

   private interface Modifyable {
      public Object modify(Object original);
   }

I find Clojure code more simple, elegant and powerful than the corresponding Java solution (albeit more difficult to read at fist glance). I wrote the Java code after the Clojure code and it shows. Had I started thinking in Java, it is likely my code would have been different. I would have designed the code differently to avoid that challenge in the first place (e.g. using explicit parameters instead of a Map and typify at the top of the chain).

I also find interesting that the declarative language (Clojure) is closer to the natural language (English) I used to describe the problem and, still, my first instinct is to translate it in an imperative language (Java) first3.

I sometimes uses this type of "functional" Java code in test cases when I want to have common code and a simple change in behavior in many tests but I rarely use it in production code.

Learning Clojure is a fun exercise as I have a special fondness for Lisp and its ability to express data and code together.
But the important part of this exercise is to increase my ability to solve problems, to learn to think differently (to think "declaratively" in addition to "imperatively") to approach challenges from different angles and have an open mindset to solve them. That's the fun part of software engineering.

  1. That I find this a challenge really shows my inexperience with Clojure and functional languages... ↩
  2. English is not my native language. As you can see, I am very open-minded on what reads like English!  ↩
  3. I am reading Gödel, Bach, Escher and pay lot of attention these days to the relations between languages. ↩

Nice Clojure Outline in TextMate

August 14, 2009
+

TextMate is my favorite text editor and my preferred IDE when I'm not writing Java or Objective-C (for these two, I use Eclipse and Xcode).

I am currently learning Clojure and installed a bundle to have proper syntax highlighting and macros in TextMate.

I found a cool feature by accident yesterday. My file was starting to be "big" and I added comments to separate the different "sections":

;; ************* ;;
;; Configuration ;;
;; ************* ;;

;; Configuration-related code

;; ********* ;;
;; DB Layer ;;
;; ********* ;;

;; DB-related code

;; *********** ;;
;; Web  Layer ;;
;; *********** ;;

;; Web-related code

;; ******* ;;
;; Startup ;;
;; ******* ;;

(db/init)
(server/start)

While I was writing code, I wanted to move from one part of the file to another and I used TextMate's outline for that (the outline is the right part of the status bar at the bottom of the window, when you click it, it displays a list of all defined functions in the file). I noticed that these special comments were also displayed:

Clojure Outline in TextMate (click for full size)

I don't know if the "magic" comes from TextMate or the Clojure bundle. To make it appear in the outline, the comment must end with ;;:

;; This will appear in TextMate outline ;;

;; This won't

I tried several comment formats in Ruby code but I was not able to make them appear in the outline. I assume the kudos should go the the bundle's authors rather than to TextMate1.

Outline the "logical" structure of the code

It is a very good idea to display these comments in the outline as they help to show the logical structure of a file.

For example in Java, the code is often organized as public/protected/private methods but it is does not give enough information on the logical composition of the class. I add comments to delimit the various interfaces implementations:

class Foo extends Thread implements IFoo, IBar {

    // IFoo implementation
    ...
    // IBar implementation
    ...
    // Thread overrides
    ...
    // Public
    ...
    // etc.
    ...
}

This valuable information is not displayed in Eclipse, in its outline or any other views2. This valuable information is lost while it would greatly improve the outline and make it more useful.

I already encountered such "special comments" when I was developing an iPhone application with Xcode. It supports a special preprocessor directive to display text in the outline:

#pragma mark This will appear in Xcode outline

// This is a regular Objective-C comment
// it won't appear in the outline

I used them to split my Objective-C class in different logical sections (e.g. controller implementation, UI actions, protocol delegate):

Objective-C Outline in Xcode (click for full size)

I rarely uses code outline. Most of the time, I navigate in the code using the keyboard or the mouse wheel. Naturally, I prefer the outline to not clutter the UI when I do not use it.
However, when I want to use it, I prefer when it shows a "logical" outline of the code and not only a simple listing.

Compared to TexMate or Xcode, I find that Eclipse's outline is cluttered:

Java Outline in Eclipse (click for full size)

I can not use it without either maximizing the view or scroll it horizontally (both requires too much UI gestures before reading the outline). I also find that it is overloaded with too much information:

  • different icons for private/protected/public fields
  • icon overlays for static, final modifiers or warnings
  • nested hierarchy losing valuable UI space

All these details brings noise instead of information: an outline should display a general view of the class, not all these tiny details.

Using quite often these three softwares, I find TextMate and XCode approach to outlines more adapted to my workflow:

  • they do not show it until you require it
  • It displays a logical view of the code (using the comments/directive to show the logical sections)
  • it does not show too much details not relevant to a broad view of the code
  1. To be fair, TextMate deserves a meta-kudos to allow such cool features through its bundle mechanism! ↩
  2. I have not check if NetBeans or IDEA can display "special comments" in their outlines but I never heard about such a thing... ↩

HTML Glyphs Cheatsheet

August 12, 2009

I don't often need HTML glyphs apart from the usual suspects (&amp; for &, &lt; for <, etc.) but when I need to find a specific one, I always waste time finding it again.

I found a good reference which lists all the glyphs. However it is slow to load all the glyphs and the default font size is too slow to see the details of the glyphs. I need to increase the text size (and reload the whole page) several times to have a better look at the glyphs, making it really slow.

I finally decided to have my own cheatsheet to reference the glyphs I sometimes need (and some I may use one day...).
I will never waste time again to find the Command key ⌘ (&#8984;).

Concurrency Is My New Memory Management Criteria

July 6, 2009

Way back in school, I started studying object-oriented programming language with C++. I found the object concepts simple and powerful (especially after learning procedural programming with Fortran) but I did not like C++.

I browsed the internet to find if there were other OO languages that I could use instead of C++. I quickly found Java (1.0.x at the time, if I remember correctly) and preferred it for 3 reasons:

  1. the concept and syntax were similar to C++ but simpler and well-defined. I could still leverage what I learn with C/C++ and applied it to Java
  2. the documentation was good: with the javadoc, the whole platform API was one-click away
  3. it had an automatic garbage collector

So I started to study Java seriously during my spare time, I found internships where I could learn from other Java developers and I ended up using it for most of my professional career.

Retrospectively, reasons #1 and #2 were nice to have but automatic garbage collection was the real decision criteria.

When I develop an application, most of the time I don't care when I should allocate objects, if should I take ownership or release them, etc. I just want to have an object and use it1.

But C++ forced me to think about all these details while Java made it simpler: new Foo() and that's it. Back then, I was just starting to learn programming languages and I preferred to let the languages handle the memory: it would be less error-prone than if I had to do it manually.

My student friends told me that my Java programs would consume more memory than the C++ equivalent and likely run slower. Maybe it was true at the time, but the Java programs were simpler and faster to write, less buggy and less likely to leak memory during their lifetime.
So I chose Java and never looked back to C++.

Back to the present.

We now have multiple cores in our machines and to use them efficiently, we need to increase the concurrency of our code.

And I'm not enthusiastic with Java. The java.util.concurrent library helps a lot but I find really hard to write correct concurrent code and too easy to screw it up. I'm in the same state of mind that when C++ was "forcing" me to think about memory management.

I have added other languages to my toolbox (mainly Ruby/JRuby and Javascript). They complement nicely Java but they are not more fit for concurrency. There should be a language/platform that can handle concurrency for me or at least help me write correct concurrent code.

Which languages/platform will I use for that? Erlang, Clojure2, Scala, Reia, other?
Which concepts should I use? STM, messages, actors, fork/join, Grand Central Dispatch, other?
I don't even know if I am looking for a new platform or another language on top of the Java platform.

But I know that I am looking for something to help me with concurrency the same way Java helps me manage the memory.

Concurrency is likely to be one of the main decision criteria when I choose which next language/platform to learn. I envision it being as important as memory management was when I started to learn OO and how it tipped the balance for Java instead of C++.

Choosing Java over C++ shaped the beginning of my career. Choosing a concurrent language/platform may have the same importance for the next part of it...

  1. I waited the release of Objective-C 2.0 to learn about Mac programming. I wanted to learn this language since I bought my first Powerbook but I waited until they added a garbage collector because I was not interested to manually manage the memory. Finally I still ended up having to learn it since the garbage collector is not available on the iPhone. Unsurprisingly, I have to track down a few memory leaks in the iPhone applications I develop. ↩
  2. I received the printed version of Programming Clojure a few days ago. I like what I am reading so far. ↩

jmx4r 0.0.8: "Through the Looking Glass"

June 15, 2009

jmx4r 0.0.8 has just been released.

Why "Through the Looking Glass"?
Until this version, jmx4r was a library which made it super easy to write simple Ruby scripts to manage Java applications using JMX.

With this version, the perspective has changed, we went through the looking glass and are now on the other side: jmx4r makes it super easy to directly manage Ruby applications by leveraging JRuby and the Java platform.

A 30-line example is worth 10,000 words:

#!/usr/bin/env jruby
require 'rubygems'
require 'jmx4r'

# a regular Ruby object we want to manage
class Foo < JMX::DynamicMBean
   operation "a polite management operation"
   parameter :string, "how do you want to be called?"
   returns :string
   def hello(name="world")
      "hello, #{name}!"
   end

   operation "double the value"
   parameter :long
   returns :long
   def double(value)
     value * 2
   end
end

# Java objects to register the Ruby object in the platform
import java.lang.management.ManagementFactory
import javax.management.ObjectName

foo = Foo.new
# each managed object needs an unique 'ObjectName'
object_name = ObjectName.new "foo:type=Foo" 
ManagementFactory.platform_mbean_server.register_mbean foo, object_name

# we keep the script running to manage it using jconsole
puts "open jconsole to manage foo registered under 'foo:type=Foo'"
gets
ManagementFactory.platform_mbean_server.unregister_mbean object_name

Save this script and run it using JRuby (tested it with version 1.3.0):

$  jruby bean.rb
open jconsole to manage foo registered under 'foo:type=Foo'

That's it: you have a Ruby application which can be managed using any Java management console.

For example, if we start jconsole to manage the bean.rb script:

jconsole_connection

In the MBeans tab, we can expand Foo to see the management operation:

jconsole_operation

And we can finally invoke it (e.g. by pushing the hello or double" button) and get a result after the method is called on the Ruby object:

jconsole_result

How to manage a Ruby object

To be manageable from a management console, the Ruby object must inherit from JMX::DynamicMBean

Since Java is statically typed, you also need to give some hints to help Java calls the Ruby object.

Each method you want to expose as a management operation must be annotated with:

  • an operation (and an optional description)
  • a list of parameter (with a mandatory type and optional name and description)
  • a returns type

The returns and parameter type must be one of :boolean, :byte, :int, :long, :float, :double, :list, :map, :set, :string, and :void

For example, if we have a method which prints the name and age of an user, it can be minimally exposed as:

   operation
   parameter :string
   parameter :int
   returns :void
   def display(name, age)
     puts "#{name} is #{age} years old
   end

Exposing Ruby attributes for management is even simpler:

class AttributeTypesMBean < JMX::DynamicMBean
   rw_attribute :my_attr, :string, "a read/write String attribute"
   r_attribute :another_attr, :int, "a readonly int attribute"

    ...
  end

the rw_attribute declares a Ruby attribute (using attr_accessor) which is also exposed for management and can be read and write from a management console. Likewise, r_attribute declares a read-only Ruby attribute (using attr_reader) which can only be read from a management console.

DynamicMBean RDoc contains a description of all these annotations.

Finally, the code to register/unregister the Ruby object is taken directly from the Java library using ManagementFactory.platform_mbean_server to access the Java Platform's MBean Server.
Each managed object must be registered with an unique ObjectName that is created using the javax.management.ObjectName class (the JMX Best Practices is a good start for an overview of JMX, the umbrella name for management in Java).

Most of this code was taken from the jmx module of jruby-extras. I fixed some issues with it but, from now on, the remaining bugs are likely written by me!

jmx4r was a simple and small library to write Ruby scripts to manage Java applications.
It is now also a simple and small library to manage Ruby applications.

With the success of JRuby and the rise of Ruby applications running on Java such as Torquebox, GitHub:fi, etc., I believe it can be very useful to leverage the features provided by the Java platform to manage Ruby applications.

As usual, to get this new release, just update the rubygem:

jruby -S gem install jmx4r

and do not hesitate to contribute:

git clone [git://github.com/jmesnil/jmx4r.git][jmx4r]