ClojureScript, D3, and Reagent

I recently set out to see what a small ClojureScript app might look like. I have a need at work for a React-based app that will ingest quite a bit of data and display it in a dashboard format for the end-user. I figured it would be a good time to take a another look at ClojureScript and see how things have evolved and whether the ecosystem is stable enough to support us.

A Few Words About ClojureScript

I should start by saying that I’ve been looking for an excuse to try ClojureScript out. There’s a great deal I like about it, not the least of which is bringing some sanity to an insane language. (To see that insanity, watch the Wat lightning talk by Gary Bernhardt starting near 1:22 into the movie. It’s a short video and well worth watching, if not for laughs then for the sheer craziness of some languages.)

I also like that:

• Data is immutable by default
• There is a wealth of operations you get to operate against said data
• There is potential for smaller, faster code due to the Google Closure compiler being used under-the-hood. One of the many advantages is that Google Closure can do dead-code elimination, allowing anything that isn’t being used to be removed
• It’s a functional language

I should also mention I tried ClojureScript a while back, but didn’t feel like ClojureScript itself was polished enough yet and that the tooling around developing ClojureScript apps was fragile. Kudos to those who kept at it though, because I think the tool support is much better today–though there is still room for improvement.

That said, there are some downsides:

• You need to pull in a fairly large Java ecosystem to use ClojureScript
• It’s a language on top of JavaScript, which needs it’s own runtime, so it’s not clear what kind of code compression you can really get in the final result
• The stack traces tend to suck
• It’s easy to lose sight of where you need interop. This is important because many other JavaScript libraries cannot cope with ClojureScript’s collections
• If you’re not used to reading Lisp, it can be a hurdle. I still don’t know what elegant, documented ClojureScript looks like. Most of what I find is hard to follow, dense, and lacks any source code comments
• The compiler seems to suffer some fairly serious size regressions occasionally. My app was about 280 KiB, and then jumped to nearly 450 KiB due to several changes in the compiler.

All of that said, things are pretty well-maintained–David Nolen is a beast of a developer… I really don’t know where he finds the time. And some of it, you’ll have to face even use straight-up JavaScript. For instance, you’ll like need npm, uglifyjs, browserify, and others to build your JavaScript application–and that’s no small amount of tooling or code that you’re bringing in either.

UPDATE: The size regression is not a regression. The compiler did indeed have a bug where it was not including foreign libraries correctly. So it seems I’m paying about 130 KiB in overhead to have ClojureScript involved on this otherwise small app. However, I am getting the full benefit of immutable data structures (which you would have to include something like Immutable.js to get, adding another 55 KiB to the size), and better infrastructure for future optimizations when the app grows (and it will grow) and the Google Closure library which is optimized too. So the cost/benefit analysis still isn’t clear.

D3

D3 is a visualization library written in JavaScript, and has taken a while for me to get my head wrapped around (I’m not quite sure I’m complete there yet!). There’s a lot to like about it. The sheer number of utility functions it gives you to help do visualization is phenomenal, and is only getting better. It can handle quite a bit of the painful parts for you, but it also leaves a substantial portion up to you. This is by design: it doesn’t limit you to only a handful of visualizations, it wants to support the full breadth of what you want to explore.

There’s a pretty large gallery of examples available for perusing–just to give you an idea of what is possible.

For my app, I need smaller (roughly 208px by 208px) graphics that are rich with information about that part of the system. In some cases, I have 4 or 5 pieces of relevant data that need to go into that small area that are easy to parse at a glance, and D3 was an excellent solution for that. Later, as we incorporate more metrics, we’ll likely lean on D3 again to help us draw a few more charts.

The trick here is that in order to use the advanced compilation in Google Closure, you really need an externs file to avoid the compiler mangling the external names. Fortunately, some folks have put together a repository to make this simpler for users of ClojureScript. Not every JavaScript package is in there, but many popular ones are, including D3 and React.

React and Reagent

If you haven’t looked at React, you really need to. I like many of the ideas there, and it’s pretty easy to get started with the in-browser compiler version. The in-browser compiler is not the way you want to go for production, and that’s where things start getting hairy. To properly packaged up your JavaScript app, you end up employing npm, babel, browserify, uglifyjs, and possibly minifify. Our main app is a Python daemon, so all of this is a substantial deviation from that territory. And after looking at all I had to do here, it started to feel like we could make the transition to ClojureScript with the same amount (or less!) tooling changes.

On real hangup with React is the integration with D3. D3 really wants a DOM node, but you don’t get real DOM nodes in the render routine. The standard practice is to implement a componentDidUpdate() method that will call out to your D3 code to draw the visualization. This wasn’t too hard to coordinate, though it does involve a bit of boilerplate.

With ClojureScript in the picture, it also made sense to look at Om and Reagent. Both build on top of React, but each takes a different strategy to exposing React. I don’t think Om is the right API, and apparently, neither does David Nolen. He’s been working hard on Om.Next, and I have to say that I really like the direction it’s going in. Unfortunately, it’s in an alpha state right now, and I can’t honestly propose it for use yet in our project. I’ll be keeping an eye on it though.

So, I chose to use Reagent instead. Reagent encourages keeping your global state in an atom, and then divvying out that state to the various components. It’s not hard to see how this fits our application well: we have a large amount of state that we want to partition between many components. However, some components need to interact with D3, and that’s where the pain began.

It turns out that it’s hard to find a good example of what to do here, so I relied on my previous work with React to help guide the way. That only got me so far though, as Reagent’s model is different. It likes to use a special kind of atom and dereferencing that in the render method is how Reagent pays attention to whether a component should be re-rendered or not. While there’s a quite a bit said around this, none of the documentation comes and states specifically that it has to be dereferenced in the render method. My guess is that in the 90% use-case, the function you are writing is the render method, so it doesn’t need any more clarification. But if you need to create something more specialized, this fact is really important.

In the end, I settled on a form like this for my D3 components:

(defn d3-gauge [args ...]
  (let [dom-node (r/atom nil)]
    (r/create-class
      {:component-did-update
       (fn [this old-argv]
         (let [[_ args ...] (r/argv this)]
           ;; This is where we get to actually draw the D3 gauge.
           (draw-d3-gauge @dom-node args ...)))

       :component-did-mount
       (fn [this]
         (let [node (r/dom-node this)]
           ;; This will trigger a re-render of the component.
           (reset! dom-node node)))

       :reagent-render
       (fn [args ...]
         ;; Necessary for Reagent to see that we depend on the dom-node r/atom.
         ;; Note: we don't actually use any of the args here.  This is because
         ;; we cannot render D3 at this point.  We have to wait for the update.
         @dom-node
         [:div.gauge [:svg]])})))

It took a while to really understand everything that needed to happen, so let me explain it. First, we really need a dom node to render the D3 gauge. To capture this, we use a Reagent atom to capture the dom node and we set the value in the component-did-mount function using reset!. Unfortunately, just setting this won’t cause Reagent to try and re-render our component so that we can draw the gauge. Instead, we must deref dom-node in the reagent-render method so that Reagent knows that we depend on it. Also note that a Hiccup-style notation is used to specify the HTML elements in the reagent-render method.

Now that we have that in place, we can draw the D3 component when an update happens. Next, we add a component-did-update method to draw the gauge. Now, I tried lots of things involving cursors into the state, but it was all overly complicated, and none of it was as efficient as it could be. I had to dig into Reagent’s source and really examine what was happening. It turns out that Reagent–probably rightfully–doesn’t peek inside arguments that are cursors to see the value. Instead, it works best to use the cursors in the parent, and deref them in the parent’s render routine. In my case, it may look like:

(defn my-component []
  (let [necessary-state (r/cursor global-state ["foo" "bar"])]
    [d3-gauge @necessary-state]))

The advantage of this is that Reagent will do some really smart things. First, the should-update-component default method will do a lot for you. It’ll compare the previous args to d3-gauge to the current ones to determine if the component needs to be re-rendered. Since we have a substantial amount of state, we really want this–I don’t want anything to be re-rendered unless the data behind it has changed. Creating cursors in the render method and the derefing them defeated this optimization, which is why I steered away from doing it.

The next headache was that component-did-update provides the old arguments to the routine, but not the new ones. It’s easy to get to, you just need to use reagent.core’s argv function to obtain them. argv will return everything, not just the arguments to the function, but a reference to d3-gauge as well (it’s the entire contents of the Hiccup form). You’ll need to peel off the first value and discard it, then pass the rest to the function that will actually draw the D3 visualization, along with the dom node to use as the parent.

Wrapping Up

I felt like I spent far to long trying to figure out how to make this work–a better example would have done wonders to improve the situation. Hopefully, someone can benefit from my pain and find some use for what I’ve managed to work out. Now that I finally understand what is happening, how it’s happening, and why, I think most everything should fall into place. I’m still not quite convinced ClojureScript is the solution I want to go forward with though–the size regression mentioned earlier was really disappointing and is eating away the majority of the reason for considering ClojureScript in the first place. I filed a ticket for the issue, so let’s hope that there’s an easy resolution to the problem, and we can keep moving forward with this solution.

Despite some of the heartache, and the incredible amount of time I spent tracking down all kinds of little issues, it’s been quite fun. I find that I like LISP-y languages–they remind me an awful lot of Python. In the case of Clojure, I feel like it’s a better Python–you get the power of the JVM and its JIT engine, along with the optional typing facilities to help speed things along. It was really nice to see how much the tooling around ClojureScript has come along. There are still snaggles but, on-the-whole, it’s worlds better than it was.

Best of all, it was fun to see how working with such a functional language changed the design. In many ways, I believe it’s more succinct. In others, I believe it’s worse–the “what does elegant Clojure code look like” problem. I don’t think it’s anything that can’t be addressed, but it’d be nice to have some help paving that path with useful style guides and examples.