Evaluating JavaScript in a Node.js REPL from an Emacs Buffer

Written by J. David Smith
Published on 01 June 2014

For my internship at IBM, we're going to be doing a lot of work on Node.js. This is awesome: Node is a great platform. However, I very quickly discovered that the state of Emacs ↔ Node.js integration is dilapidated at best (as far as I can tell, at least).

A Survey of Existing Tools

One of the first tools I came across was the swank-js / slime-js combination. However, when I (after a bit of pain) got both setup, slime promptly died when I tried to evaluate the no-op function: (function() {})().

Many pages describing how to work with Node in Emacs seem woefully out of date. However, I did eventually find nodejs-repl via package.el. This worked great right out of the box! However, it was missing what I consider a killer feature: evaluating code straight from the buffer.

Buffer Evaluation: Harder than it Sounds

Most of the languages I use that have a REPL are Lisps, which makes choosing what code to run in the REPL when I mash C-x C-e pretty straightforward. The only notable exceptions are Python (which I haven't used much outside of Django since I started using Emacs) and JavaScript (which I haven't used an Emacs REPL for before). Thankfully, while the problem is actually quite difficult, a collection of functions from js2-mode, which I use for development, made it much easier.

The first thing I did was try to figure out how to evaluate things via Emacs Lisp. Thus, I began with this simple function:

(defun nodejs-repl-eval-region (start end)
  "Evaluate the region specified by `START' and `END'."
  (let ((proc (get-process nodejs-repl-process-name)))
    (comint-simple-send proc (buffer-substring-no-properties start end))))

It worked! Even better, it put the contents of the region in the REPL so that it was clear exactly what had been evaluated! Whole-buffer evaluation was similarly trivial:

(defun nodejs-repl-eval-buffer (&optional buffer)
  "Evaluate the current buffer or the one given as `BUFFER'.

`BUFFER' should be a string or buffer."
  (let ((buffer (or buffer (current-buffer))))
    (with-current-buffer buffer
      (nodejs-repl-eval-region (point-min) (point-max)))))

I knew I wasn't going to be happy with just region evaluation, though, so I began hunting for a straightforward way to extract meaning from a js2-mode buffer.

js2-mode: Mooz is my Savior

Mooz has implemented JavaScript parsing in Emacs Lisp for his extension js2-mode. What this means is that I can use his tools to extract meaningful and complete segments of code from a JS document intelligently. I experimented for a while in an Emacs Lisp buffer. In short order, it became clear that the fundamental unit I'd be working with was a node. Each node is a segment of code not unlike symbols in a BNF. He's implemented many different kinds of nodes, but the ones I'm mostly interested in are statement and function nodes. My first stab at function evaluation looked like this:

(defun nodejs-repl-eval-function ()
  (let ((fn (js2-mode-function-at-point (point))))
    (when fn
      (let ((beg (js2-node-abs-pos fn))
            (end (js2-node-abs-end fn)))
        (nodejs-repl-eval-region beg end)))))

This worked surprisingly well! However, it only let me evaluate functions that the point currently resided in. For that reason, I implemented a simple reverse-searching function:

(defun nodejs-repl–find-current-or-prev-node (pos &optional include-comments)
  "Locate the first node before `POS'.  Return a node or nil.

If `INCLUDE-COMMENTS' is set to t, then comments are considered
valid nodes.  This is stupid, don't do it."
  (let ((node (js2-node-at-point pos (not include-comments))))
    (if (or (null node)
            (js2-ast-root-p node))
        (unless (= 0 pos)
          (nodejs-repl–find-current-or-prev-node (1- pos) include-comments))

This searches backwards one character at a time to find the closest node. Note that it does not find the closest function node, only the closest node. It'd be pretty straightforward to incorporate a predicate function to make it match only functions or statements or what-have-you, but I haven't felt the need for that yet.

My current implementation of function evaluation looks like this:

(defun nodejs-repl-eval-function ()
  "Evaluate the current or previous function."
  (let* ((fn-above-node (lambda (node)
                         (js2-mode-function-at-point (js2-node-abs-pos node))))
        (fn (funcall fn-above-node
              (point) (lambda (node)
                        (not (null (funcall fn-above-node node))))))))
    (unless (null fn)
      (nodejs-repl-eval-node fn))))

You Know What I Meant!

My next step was to implement statement evaluation, but I'll leave that off of here for now. If you're really interested, you can check out the full source.

The final step in my rather short adventure through buffevaluation-land was a *-dwim function. DWIM is Emacs shorthand for Do What I Mean. It's seen throughout the environment in function names such as comment-dwim. Of course, figuring out what the user means is not feasible – so we guess. The heuristic I used for my function was pretty simple:

This is succinctly represent-able using cond:

(defun nodejs-repl-eval-dwim ()
  "Heuristic evaluation of JS code in a NodeJS repl.

Evaluates the region, if active, or the first statement found at
or prior to the point.

If the point is at the end of a line, evaluation is done from one
character prior.  In many cases, this will be a semicolon and will
change what is evaluated to the statement on the current line."
   ((use-region-p) (nodejs-repl-eval-region (region-beginning) (region-end)))
   ((= (line-end-position) (point)) (nodejs-repl-eval-first-stmt (1- (point))))
   (t (nodejs-repl-eval-first-stmt (point)))))

The Beauty of the Emacs Development Process

This whole adventure took a bit less than 2 hours, all told. Keep in mind that, while I consider myself a decent Emacs user, I am by no means an ELisp hacker. Previously, the extent of my ELisp has been one-off advice functions for my .emacs.d. Being a competent Lisper, using ELisp has always been pretty straightforward, but I did not imagine that this project would end up being so simple.

The whole reason it ended up being easy is because the structure of Emacs makes it very easy to experiment with new functionality. The built-in Emacs Lisp REPL had me speeding through iterations of my evaluation functions, and the ability to jump to functions by name with a single key-chord was invaluable. This would not have been possible if I had been unable to read the context from the sources of comint-mode, nodejs-repl and js2-mode. Even if I had just been forced to grep through the codebases instead of being able to jump straight to functions, it would have taken longer and been much less enjoyable.

The beautiful part of this process is really how it enables one to stand on the shoulders of those who came before. I accomplished more than I had expected in far, far less time than I had anticipated because I was able to read and re-use the code written by my fellows and precursors. I am thoroughly happy with my results and have been using this code to speed up prototyping of Node.js code. The entire source code can be found here.

World of Warcraft's Recruit-a-Friend Reward Structure is Flawed

Written by J. David Smith
Published on 05 April 2014

What instigated this post?

Last night, an unnamed redditor asked the WoW sub-reddit what the fastest way to level these days is. Why? Because their girlfriend "has been wanting to start playing wow with me". Seems reasonable, right? S/he goes on to ask about RaF.

I immediately jump in and try to head off a disaster in the making. "What disaster?" one may ask. Simple: RaF dungeon spamming isn't fun. In fact, I wrote that "Personally, I wouldn't even use RaF because of how it completely screws up the structure of the early game." This set the gears in my head to whizzing frantically. What changed that made a really cool system actively harm the game? And – more importantly – how can it be fixed?

What is Recruit-a-Friend?

In order to answer those questions, it is important to understand what the RaF system actually does. Blizzard's FAQ does a good job of describing the system. There are actually a lot of perks to using RaF, but there is one in particular that really hurts the game: triple XP.

For levels 1 - 85, while in a group and similarly leveled the recruiter and recruitee gain 3 times the normal amount of experience. This isn't simply mob kill experience either: quest experience is also affected. The result these days is that – if you aren't spamming dungeons to power-level – you out-level zones just as you're starting to get their stories. To understand the impact of this effect, we need to first dig deeper into what the reward structure for WoW is.

I Saved Westfall and all I got was this stupid T-Shirt!

World of Warcraft is not unique in its structure. You help people, kill monsters and collect rewards. There are two general classes of rewards in WoW:

  1. Power-increasing rewards

    These rewards increase the player's overall power level (although perhaps not immediately). Examples of this are loot (literal character power), gold (economic power) and experience (character power – albeit slightly delayed).
  2. Emotional rewards

    These rewards tug on the player's heart-strings. Whether it's saving an adorable little orphan boy or laughing maniacally as you help Theldurin punch Deathwing in the face, these ones make you feel good (or bad) for having done whatever it was you did. Type 1 rewards are a subset of this reward class.

In my experience, the latter are much more important than the former. This is upheld by observations of the reaction to the Madness of Deathwing fight and Deathwing in general. While players got more powerful than ever before, there was something missing. Emotional reward was lacking, and it showed.

How does this relate to Recruit-a-Friend?

The RaF system increases the gain rate of a particular Type 1 reward: experience. However, it not only causes problems with the rate of gain of other Type 1 rewards, but often outright prevents the gain of Type 2 rewards!

Recently, I leveled through Darkshore. Starting at level 11, I finished the quest achievement at level 24. Had I been using RaF, I'd have only made it through the first 1/3rd of the quests in that time. This would have left the story hanging and broken the illusion of world-changing impact that Blizzard has worked so hard to create.

As a result, emotional investment can become a liability preventing enjoyment rather than a boon aiding it. It's like reading the first third of every Spider-Man comic in order to 'catch up' to the current. Sure, one would reach your goal faster, but at the cost of enjoying the process of reading comic books. Even once you were caught up, you wouldn't understand all of the stuff going on in the current issue.

I've seen situations where one player wants to get their significant other into the game using RaF. In every case I've seen where the core benefit of RaF is used to its fullest (ie by dungeon spamming), the SO quits playing. Therefore, I believe that the overall benefit of RaF for the new player is non-existent and in many cases it even causes damage to their perception and enjoyment of the game.

Two Birds, One Stone

The solution to this problem is relatively simple. While simply removing the XP bonus would go a long way towards preventing the damage currently being done by RaF, why stop at simple prevention when it can be used to make the game genuinely more enjoyable?

Think back, ye die-hard WoW fans: what problem always crops up when questing as a group? Yes, that one. You know it well. Someone plays while the others are away, gets ahead in both experience and quests and is either forced to wait for the group to catch up, retread the content you just did, or leave the group behind.

With long-time players, this isn't much of a problem. We have alts, we have mains, and we can always do something else while the group is offline. For a new player, however, such options are severely lacking. PvP grants experience, dungeons grant experience, even gathering mats to level crafting grants experience these days! Imagine if the Priest class is the only one that really clicks with your friend. Are you going to ask them to not play when you aren't online? To roll an alt? A second priest?

This problem is solved relatively well by the combination of massively boosted XP and level granting: the increased XP rate encourages moving on to other quest chains with relative frequency and level granting ensures that the older player can keep up (most of the time). However, if triple XP is removed from the system, then the problem again rears its ugly head because the player no longer has such an incentive to move on in the middle of a quest chain.

Sure, the two players can remain evenly leveled, but what about quest progress? Forcing the new player to retread content is not exactly ideal, so why not allow the new player to catch the older one up not only in levels but also in quests?

What I am proposing is this:

This would prevent XP gain from completely overriding any other sort of reward in the game and would allow new players to continue questing with their friends without worrying about quest dependencies and level discrepancies. To my view, this would be superior to the current system – especially since the store is now the go-to way to pay for a fast 90. However, one question remains to be answered.

Why was it designed this way in the first place?

World of Warcraft is not the game that it once was. In ye olden days, when Azeroth was yet young and paladins still only had 2 buttons for the first 40 levels, there were fewer quest chains and it was common – up til Outland, at least – to complete a zone without having out-leveled it. In that era, there were far fewer tales of merit told in the quests.

Way back then – near a full 6 years ago – tripling the experience rate made sense. It meant that you'd have to do one zone to get through a level range instead of 2.5-3. Still, those days are gone and now, with the world designed to take one player through a level range in one zone, it no longer makes sense.

Here's hoping that Blizzard fixes this system soon. It bothers me to think of the people potentially missing a great experience because something that should be rewarding can easily become the opposite. With all of the dramatic WoD changes incoming, this could be the perfect time to do it!

New Site (Built with Stasis)

Written by J. David Smith
Published on 03 March 2014

For a long time I've had my blog hosted on Wordpress.com, but today that has come to an end. While I've had a domain and server set up since mid-2013, I hadn't had the opportunity to decide how to build my new blog. However, when stasis was announced at the end of January, I realized that I may have found my solution.

First off: why not Wordpress?

Nothing against Automatic, but after having run several WP blogs I sympathize with this guy:

Wondering how I managed to end up building a Wordpress site today. For those of you that do this regularly, you have my deepest sympathies.

— Daniel Grant (@danieljohngrant) February 25, 2014

I don't want to run another WP blog and I don't want to have to hack any more PHP. The solution?

Static Site Generation

Static Site Generation is a pretty simple concept. You have some templates and some content, you want to put the content in the templates, and only want to do so once. The site content is transformed into HTML once by the site owner (aka me) and then served without any extra work by the server.

This has some big advantages. First, it makes a very fast website, as the restriction is not HTML generation time but simple transmission time. Second, it is extremely secure because malicious content serving is impossible short of someone gaining root access on my server (or someone hijacking Disqus; I trust Disqus' security people to do better than I could – it is their job, after all).

Even better, because the code doesn't need to interact with the server, I am not restricted to things that play nicely with the server (which Clojure actually does through Java, but that's not a place I'd like to go right now).

I only had one big requirement – that I be able to write my posts in Org format – but I also wanted something that I could hack on. Clojure is the language I'm most interested in right now, so I started looking in that direction. I toyed around with several options – even going so far as to fork nakkaya's static – but eventually settled on magnar's stasis.

The biggest problem I had with static was how it dealt with posts. This snippet says it best:

(defn list-files [d]
  (let [d (File. (dir-path d))]
    (if (.isDirectory d)
       (FileUtils/listFiles d (into-array ["markdown"
                                           "html"]) true)) [] )))

(defn create-latest-posts
  "Create and write latest post pages."
  (let [posts-per-page (:posts-per-page (config))
        posts (partition posts-per-page
                         (reverse (list-files :posts)))
        pages (partition 2 (interleave (reverse posts) (range)))
        [_ max-index] (last pages)]

As you can see, the posts list is created by using partition on what amounts to a directory listing. While this isn't a huge problem, my blog posts aren't organized that way and I didn't want to change that. Having dates in the file name looks ugly to me – never mind the fact that it duplicates the #+DATE headers that are in all of my posts.

This is where stasis comes in. It's a no-batteries-included framework, which means basically all it does is apply the templates to my sources. This leaves designing the templates, template framework and sources to me. I used the whattheemacsd source as my stasis-basis and built from there.

The biggest thing I had to do was implement conversion of Org files into HTML. While not the fastest option (in terms of running time), I opted to simply leave that to emacs by calling it in batch mode. The #+STUFF headers are trivial to parse using regexp, so pulling in my #+DATE's was a non-issue.

Ultimately, I'm pretty happy with how things turned out. This is the first post I've written using the new system and it's worked great!

What next?

There are a couple of features that I want to build, starting with category and tag views. After that, I may look at implementing an elisp command to replace my current deployment method (a shell script) so that I can deploy directly from the editor.

Technology & Style Credits

The full source code is available on github.