søndag 23. februar 2014

A Groovy Light Table client - Step 2: Evaluating Code

Background

This is the second post in my series "A Groovy Light Table client". A blog series about steps I take when trying to build a Groovy plugin for Light Table

In this post I will take you through some of the steps I went through to get Light Table to evaluate groovy (script) code and show results inline in the editor.


How did we get here ?

Evaluate contents of editor (cmd/ctrl + shift + enter)

(behavior ::on-eval
          :desc "Groovy: Eval current editor"
          :triggers #{:eval}
          :reaction (fn [editor]
                      (object/raise groovy :eval! {:origin editor
                                                   :info (assoc (@editor :info)
                                                           :code (ed/->val editor)
                                                           :meta {:start 0, :end (ed/last-line editor)})})))

This behavior triggers on ":eval", which is triggered to any editor (on cmd/ctrl + shift + enter in default key mapping). We just get hold of the text from the editor and gather some meta info  and trigger a ":eval!" behavior on the groovy "mother" object defined in the last blog post.

Evaluate current line/selection

(behavior ::on-eval.one
          :desc "Groovy: Eval current selection"
          :triggers #{:eval.one}
          :reaction (fn [editor]
                      (let [pos (ed/->cursor editor)
                            info (conj (:info @editor) 
                                  (if (ed/selection? editor) 
                                    {:code (ed/selection editor) :meta {:start (-> (ed/->cursor editor "start") :line)
                                                                        :end (-> (ed/->cursor editor "end") :line)}}
                                    {:pos pos :code (ed/line editor (:line pos)) :meta {:start (:line pos) :end (:line pos)}}))]
                        (object/raise groovy :eval! {:origin editor :info info}))))

The only difference here is that we gather the code for the current line or current selection. Then we trigger the same behavior as for evaluating the whole editor.


Our groovy Eval!

(behavior ::eval!
          :triggers #{:eval!}
          :reaction (fn [this event]
                      (let [{:keys [info origin]} event
                            client (-> @origin :client :default)]
                        (notifos/working "Evaluating groovy...")
                        (clients/send (eval/get-client! {:command :editor.eval.groovy
                                                         :origin origin
                                                         :info info
                                                         :create try-connect})
                                      :editor.eval.groovy info 
                                      :only origin))))

This behavior is what actually sends off a eval request to the groovy client. Quite a lot happens under the hood (by help of inbuilt LightTable behaviors):

  • It tries to find a client (connection) for the editor
  • If no connection exists it will try to create a new one. On create it will invoke the try-connect function that we defined for the gui connect/connect bar behavior in the previous blog post
  • Once connected it will jsonify our parameters and send them off to our groovy client
The JSON might look something like:
[89,
"editor.eval.groovy",
  {"line-ending":"\n",
   "name":"sample.groovy",
   "type-name":"Groovy",
   "path":"/Users/mrundberget/Library/Application Support/LightTable/plugins/Groovy/sample.groovy",
   "mime":"text/x-groovy",
   "tags":["editor.groovy"],
   "code":"println \"hello\"",
   "meta":{"start":22,"end":22}}]


  • The first param is the client id for the editor that triggered the behavior. This client Id doesn't represent the same as a connection id (ref previous blog post). Many editors may share the same connection !
  • The second param is the command (our groovy client will of course support many different commands, this is one of them)
  • The third and last parameter is our info. The code is the essential bit, but some of the meta information, like line info comes in handy when handling the response later on

The actual groovy evaluation

Command dispatch


ltClient.withStreams { input, output ->
  try {
    input.eachLine { line ->
    def (currentClientId, command, data) = new JsonSlurper().parseText(line)
    switch (command) {
    case "client.close":
      stop()
      break
    case "editor.eval.groovy":
      evalGroovy(data, currentClientId)
      break
   default:
     log "Invalid command: $command"
  }
  // ...


    

We parse any lines received from Light Table and based on the command received invokes the appropriate handler. In this case evalGroovy.

Eval groovy


private void evalGroovy(data, currentClientId) {
  def evalResult = scriptExecutor.execute(data.code)

  def resultParams = [meta: data.meta]
  if (evalResult.out) {
    resultParams << [out: evalResult.out]
  }
  if(evalResult.exprValues) {
    resultParams << [result: convertToClientVals(evalResult.exprValues)]
  }

  if (!evalResult.err) {
    data = [currentClientId?.toInteger(), "groovy.res", resultParams]
  } else {
    data = [currentClientId?.toInteger(), "groovy.err", [ex: evalResult.err] + resultParams]
  }
  sendData data
}

The first and most significant line is where we evaluate the groovy code received. This post would be too long if we went into all the details of what it does, but here's a high-level summary:

  • We basically create a GroovyShell and compile our code to a script. Normally that would just compile a Script class. However we  wish to collect a lot more information than you typically would get from default groovy script execution. So we do an AST transformation on the script class and add a custom abstract script class as a base class for the compiled script class.  This allows us to inject behavior and wrap statement execution (all compiled into the script for optimal performance).  That way we are able to collect information about values for most types of statements. We collect line number and value (each line could end up having many values :-) )
  • We run the script (capturing system.out and system.err). 
  • The function returns:
    • Anything written to standard out (println etc)
    • Errors if any and line number for error where possible
    • A list for of maps with line number and value(s)  

Most of the AST stuff is not something I've written. It's been contributed by Jim White after I posted a question on the groovy-user mailing list. I asked for advice on which way to proceed and the response from the groovy community was awesome. Jim in particular was more than eager to contribute to the plugin. OpenSource rocks ! So when I say we, I sometimes mean we literally.

Anyways, based on the results of the script execution we notify Light Table to trigger either a ":groovy.res" behavior or a "groovy.err" behavior.

The json response for sendData for a successful execution might look something like:

[89,
 "groovy.res",
 {"meta":{"start":22,"end":23},"out":"hello\nmama\n","result":[{"line":1,"values":["null"]},{"line":2,"values":["null"]}]}]


Handling the evaluation results in Light Table


(defn notify-of-results [editor res]
  (doseq [ln (:result res)]
    (let [lineNo (+ (:line ln) (-> res :meta :start) -1)]
      (object/raise editor :editor.result (clojure.string/join " " (:values ln)) {:line lineNo :start-line lineNo}))))

(behavior ::groovy-res
          :triggers #{:groovy.res}
          :reaction (fn [editor res]
                      (notifos/done-working)
                      (when-let [o (:out res)] (.log js/console o))
                      (notify-of-results editor res)))

(defn notify-of-error [editor res]
  (let [lineNo (+ (-> res :ex :line) (-> res :meta :start) -1)]
    (object/raise editor :editor.exception (:ex res) {:line lineNo :start-line lineNo'})))

(behavior ::groovy-err
          :triggers #{:groovy.err}
          :reaction (fn [editor res]
                      (object/raise editor :groovy.res res)
                      (notify-of-error editor res)))

These are the behavior definitions that handles either successful or evaluation of scripts with errors. Basically we:
  • Print to the Light Table Console anything that was captured to system.out/system.err by our groovy evaluation
  • Show inline results for each line, multiple results for a line are space separated. For showing inline results we are using a predefined Light Table behavior (:editor.result)
  • If the behavior is to handle an error, we show evaluation results up until the script exception. In addition we display details (stack trace) for the exception at the line in the script it occurred

Wiring it all up

groovy.behaviors

{:+ {:app [(:lt.objs.plugins/load-js ["codemirror/groovy.js", "groovy_compiled.js"])]
     :clients []
     :editor.groovy [:lt.plugins.groovy/on-eval
                     :lt.plugins.groovy/on-eval.one
                     :lt.plugins.groovy/groovy-res
                     :lt.plugins.groovy/groovy-err
                     [:lt.object/add-tag :watchable]]
     :files [(:lt.objs.files/file-types
              [{:name "Groovy" :exts [:groovy] :mime "text/x-groovy" :tags [:editor.groovy]}])]
     :groovy.lang [:lt.plugins.groovy/eval!
                   :lt.plugins.groovy/connect]}}

The eval and results/err behaviors are defined for the editor tag. So they are only applicable for editors marked as groovy editors. Any editor open with a file name ending in .groovy will automatically be attached to a editor.groovy tag. (You can also set it manually cmd+space -> "Editor: Set current editor syntax").
The ":eval!" behavior is defined for the :groovy.lang tag. Its tied to our groovy mother object just like the connect behavior. These behaviors are totally groovy client specific, whilst the other behaviors are less so (although not exactly generic as they are now…)

Wrap up

A little bit of plumbing was needed to get this set up. But the hard parts was really coming up with the groovy AST transformation stuff. I guess by now you might have started getting an inkling that Light Table is fairly composable ? It really is super flexible. You don't like the behavior for handling inline results for the groovy plugin ? You could easily write your own and wire it up in your user.behaviors file in Light Table. It's wicked cool, actually it really is your editor !

Yesterday I released version 0.0.2 of the Groovy LightTable plugin. Its available through the Light Table plugin manager, or if you wish to play with the code or maybe feel like contributing feel free to fork the repo at : https://github.com/rundis/LightTable-Groovy. Pull requests are welcome.

So where to next ? I'd really like to try and create an InstaRepl editor for the plugin. A groovy script editor that evaluates code as you type. There's gotta be one or two challenges related to that. A quick win might be to provide groovy api documentation from inside Light Table. I'll let you know what happens in the next post.

Disclaimer: I might have misunderstood some nuances of LIght Table, but hopefully I'm roughly on track. If you see anything glaringly wrong, do let me know.

søndag 16. februar 2014

A Groovy Light Table client - Step 1: Connecting the client

Building a plugin i Light Table

This is the first post in (hopefully) a series of blog posts about the various steps I go through when trying to create a plugin for Light Table. I have decided to try to create a Groovy plugin. I chose Groovy to ensure there was at least one technology fairly familiar to me. I have just started using Light Table, I have no previous experience with ClojureScript and I have just recently started writing some Clojure beyond the basic tutorials.

The short term goal is for the plugin to provide inline results and maybe an instarepl of some sort for groovy scripts.

LightTable-Groovy is the name of my plugin project and you can find the latest source there. It might be a couple of steps ahead of the blog posts though ! 

Documentation

Light Table was made open source in january and documentation for plugin developers is a little sparse.  So to have something to go by I decided to use some of the other language plugins as inspiration:
I haven't worked with any of the above mentioned languages, but they did provide enough inspiration to deduce how a Light Table client might interact.

BTW. A quick starter just to get you up an running with a hello world plugin could be this screen cast by Mike Haney.


Connecting a client - Process overview

Before we dwelve into the code It's a good idea to have a high level understanding of what we are trying to achieve ! 

A couple of use cases that needs to be supported:
  • Evaluate current selection or current line of groovy code and present results (preferably inline)
  • Evaluate contents of current editor and present results
    • Provide as much info about the results of each statement as possible
    • (Maybe need to evaluate line/statement by statement)
  • For a future instarepl, any change in the editor will trigger an evaluation
It becomes evident that our plugin needs to provide some kind of process that reacts to events from light table. A default pattern for achieving this has been devised for Light Table and roughly equates to the following steps:
  1. A connect event is triggered from Light Table (you need to set up your plugin to trigger that event…). Typically the connect event can be invoked manually from the connect bar in light table, or it can be triggered implicetly when evaluating code. 
  2. You fire of a process - Using inbuilt support from Light Table you start a process either a shell script or whatever really. I created a shell script that sets some environment stuff and then basically kicks off a groovy script. Light table provides a tcp/ip port and a unique client id which you need to forward to the process.
  3. Create a tcp client: In your process you create a tcp client using the given port 
  4. Send ack message: Send a json message with client id and an event name (behavior) to Light Table (through the tcp connection!)
  5. Confirm handshake for process: In your process (i.e. not the tcp connection!) write "Connected" to standard out. ("Connected" is just what the other plugins use, you could use  anything you like as long as it matches the connect behaviors(handlers) you provide inside light table.)
  6. Listen for events: Now you are connected and given you have set up your behaviors in Light Table correctly, your new connection should be reported as connected and shown in the Light Table connect bar. Now you listen for events on your tcp client and provides appropriate responses back to Light Table accordingly. (Handling this is the subject of a future blog post)

Behaviors for connecting (from groovy.cljs): 


(defn run-groovy[{:keys [path name client] :as info}]
  (let [obj (object/create ::connecting-notifier info)
        client-id (clients/->id client)
        project-dir (files/parent path)]
    (object/merge! client {:port tcp/port
                           :proc obj})
    (notifos/working "Connecting..")
    (proc/exec {:command binary-path
                :args [tcp/port client-id project-dir]
                :cwd plugin-dir
                :env {"GROOVY_PATH" (files/join (files/parent path))}
                :obj obj})))

(defn check-groovy[obj]
  (assoc obj :groovy (or (::groovy-exe @groovy)
                         (.which shell "groovy"))))

(defn check-server[obj]
  (assoc obj :groovy-server (files/exists? server-path)))

(defn handle-no-groovy [client]
  (clients/rem! client)
  (notifos/done-working)
  (popup/popup! {:header "We couldn't find Groovy."
                 :body "In order to evaluate in Groovy files, Groovy must be installed and on your system PATH."
                 :buttons [{:label "Download Groovy"
                            :action (fn []
                                      (platform/open "http://gvmtool.net/"))}
                           {:label "ok"}]}))

(defn notify [obj]
  (let [{:keys [groovy path groovy-server client]} obj]
    (cond
     (or (not groovy) (empty? groovy)) (do (handle-no-groovy client))
     :else (run-groovy obj))
    obj))

(defn check-all [obj]
  (-> obj
      (check-groovy)
      (check-server)
      (notify)))

(defn try-connect [{:keys [info]}]
  (.log js/console (str "try connect" info))
  (let [path (:path info)
        client (clients/client! :groovy.client)]
    (check-all {:path path
                :client client})
    client))


(object/object* ::groovy-lang
                :tags #{:groovy.lang})


(def groovy (object/create ::groovy-lang))

(scl/add-connector {:name "Groovy"
                    :desc "Select a directory to serve as the root of your groovy project... then again it might not be relevant..."
                    :connect (fn []
                               (dialogs/dir groovy :connect))})
(behavior ::connect
                  :triggers #{:connect}
                  :reaction (fn [this path]
                              (try-connect {:info {:path path}})))



  • scl/add-connector: This statement adds a connect dialog to our groovy plugin. You select a root directory and upon selection the ::connect behavior is triggered
  • ::connect basically responds with invoking a method for connecting. This does some sanity checks and if all goes well ends up invoking  run-groovy.
  • run-groovy : Fires up our groovy (server) process
  • def groovy   is basically the "mother" object of our plugin. It helps us scope behaviors and commands

The server part (LTServer.groovy)

import groovy.json.*

params = [
  ltPort:   args[0].toInteger(),
  clientId: args[1].toInteger() // light table generated id for the client (connection)
]

logFile = new File("server.log")

def log(msg) {
  logFile << "${new Date().format('dd.MM.yyyy mm:hh:sss')} - $msg\n"
}

client = null
try {
  client = new Socket("127.0.0.1", params.ltPort)
} catch (Exception e) {
  log "Could not connect to port: ${params.ltPort}"
  throw e
}

def sendData(data) {
  client << new JsonBuilder(data).toString() + "\n"
}
// ack to Light Table
sendData (
  [
    name: "Groovy",
    "client-id": params.clientId,
    dir: new File("").absolutePath,
    commands: ["editor.eval.groovy"],
    type: "groovy"
  ]
)
println "Connected" // tells lighttable we're good

client.withStreams {input, output ->
  while(true) {
  // insert code to listen for events from light table and respond to those (eval code etc)
  }
}


Notification of successful conncetion


(behavior ::on-out
          :triggers #{:proc.out}
          :reaction (fn [this data]
                      (let [out (.toString data)]
                        (object/update! this [:buffer] str out)
                        (if (> (.indexOf out "Connected") -1)
                          (do
                            (notifos/done-working)
                            (object/merge! this {:connected true}))
                          (object/update! this [:buffer] str data)))))

(behavior ::on-error
          :triggers #{:proc.error}
          :reaction (fn [this data]
                      (let [out (.toString data)]
                        (when-not (> (.indexOf (:buffer @this) "Connected") -1)
                          (object/update! this [:buffer] str data)
                          ))
                      ))

(behavior ::on-exit
          :triggers #{:proc.exit}
          :reaction (fn [this data]
                      ;(object/update! this [:buffer] str data)
                      (when-not (:connected @this)
                        (notifos/done-working)
                        (popup/popup! {:header "We couldn't connect."
                                       :body [:span "Looks like there was an issue trying to connect
                                              to the project. Here's what we got:" [:pre (:buffer @this)]]
                                       :buttons [{:label "close"}]})
                        (clients/rem! (:client @this)))
                      (proc/kill-all (:procs @this))
                      (object/destroy! this)
                      ))

(object/object* ::connecting-notifier
                :triggers []
                :behaviors [::on-exit ::on-error ::on-out]
                :init (fn [this client]
                        (object/merge! this {:client client :buffer ""})
                        nil))



The above behaviors basically handles signaling success, error or connection exits for our groovy client. As you can see in ::on-out this is where we check standard out from the process for the string "Connected", to signal success.


Wiring up behaviors (behaviors.groovy)


{:+ {:app [(:lt.objs.plugins/load-js ["codemirror/groovy.js", "groovy_compiled.js"])]
     :clients []
     :editor.groovy []
     :files [(:lt.objs.files/file-types
              [{:name "Groovy" :exts [:groovy] :mime "text/x-groovy" :tags [:editor.groovy]}])]
     :groovy.lang [:lt.plugins.groovy/connect]}}

The important part in terms on the connection is the wiring of the connect behavior to ":groovy.lang". This is needed for groovy to appear as a connection item in the light table connect bar.

"codemirror/groovy.js" deserves a special mention. This is what provides syntax highlighting for our groovy files (defined in the :files vector). The syntax highlighting is provided by the groovy mode module from CodeMirror

Wrapping up

So what have we achieved. Well we now have a connection to Light Table from an external process that can listen and respond to events from Light Table. For the purposes of this blog post series, its a Groovy client that hopefully pretty soon will be able to evaluate groovy scripts and respond with evaluation results. We didn't pay much attention to it, but we also got syntax highlighting of our Groovy files complements of CodeMirror. 

It took a while to grok how the connection part worked. Once I did understand roughly what was needed I was a bit annoyed with myself for messing about so much. I'm hoping this post might help others to avoid some of the mistakes I stumbled into.

søndag 9. februar 2014

Getting my feet wet with Clojure using Light Table

Background

I've been meaning to get into Clojure for quite some time now. I have been reading books, played with 4clojure and even went to a Clojure workshop with Stuart Sierra. But I never really found time and inspiration to percolate a hobby project to try it out in anger… Until my company (Kodemaker) decided to start a project to rewrite our website in RYO style using Clojure. I asked if I could have a go at one of the features (a really tiny one).

The last time I decided to properly learn a language I tried to learn Emacs at the same time. That didn't end all that well, or rather somewhere along the road I lost sight of Emacs. I'm a IntelliJ user in my day job so I've tried both LaClojure and Cursive a little bit. A couple of my colleagues have mentioned Light Table in the past and after watching some presentations and tutorials I was intrigued. When Light Table went open source I was determined to give it a good go.


The first steps

I already had Leiningen installed (and its just a "brew install leiningen" away on my mac anyway). So installing Light Table was a breeze and when I started it and I tried a few commands, it quickly became apparent that I was quite a few keyboard shortcuts short. There are plugins for Emacs and Vim shortcuts, but I figured I'd assign my own. That freedom proved daunting as well, and I'm pretty sure my setup will change a lot before it stabilizes.

ctrl + space and type "user key" and select User keymap

{:+ {:app {
           "ctrl-shift-t" [:tabset.new]
           "ctrl-shift-left" [:tabs.prev]
           ;; etc
           }
     :editor {"alt-w" [:editor.watch.watch-selection]
              "alt-shift-w" [:editor.watch.unwatch]
              "ctrl-k" [:editor.kill-line]
              ;; etc
             }}}



The first map :app defines shortcuts that available across editors. Ie if you have focus in an inline browser window, these shortcuts will still work. The :editor map are shortcuts that requires an editor to have focus. It's really nice that there is code completion (and instant docs) for all available actions and saved changes are immediately applied. And yes you could do fancy stuff by doing multiple actions for one shortcut, its a clojure vector !
I have created a github project for my current settings to keep things in synch across machines.

The paredit shortcuts are worth a special mention. You'll quickly discover that keeping those pesky parenthesizes in order can prove to be a challenge. Light Table comes bundled with a plugin that provides some support to make your life easier. You might need to update the plugin through the plugin manager (ctrl + space in case you were wondering how to find it).

I assume that from now on that either you have a shortcut or use ctrl+space to find the appropriate actions…


Get my project up and running

Summarized it goes like this:
  1. Open folder and select project folder for your clojure project
  2. "Connect: Add connection"
  3. Select clojure and then your leiningen project.clj file
  4. Open a clj file (mac-hint: cmd+o). And evaluate it (mac-hint: cmd+shift+enter). Behold the inline results !
  5. Assuming you have a ring app, you might want to start your web server and be able to change things on the fly:
    1. Create a repl file (see sample below)
    2. Open an clojure instarepl 
    3. enter: (use '<the_name_space:for_the_repl_file>) 
    4. enter: (start-server)
    5. voila, you can now kill/clear the instarepl if you like
  6. Add a browser tab (if you have the space why not assign it to a separate tabset so you can have code and browser side by side !) 
  7. Start hacking your clojure and evaluate statement (mac-hint: cmd+enter) or evaluate the whole clojure file. Refresh your browser to see the changes (mac-hint: cmd+r) 
It is really neat once you get into it. Not quite like my old WebSphere days. If you aren't sure what a function actually returns ? Just fire up an instarepl and try it out ! Can't remember exactly how that clojure function works… instarepl. Curious about the documentation for a clojure function try out : "Docs: Toggle documentation at cursor". 


Don't have a (web) project ? Try opening a terminal and type "lein new compojure-app sample".


Sample repl file

(ns repl
  (:require [mycompany-com.web :refer [app]]
            [ring.adapter.jetty]))

(defn start-server []
  (defonce server (ring.adapter.jetty/run-jetty #'app {:port 3000 :join? false}))
  (.start server))

(defn stop-server []
  (.stop server))


Plugins

Light Table comes bundled with a few plugins, but there are already quite a few others out there that are definitely worth a look. I've already installed Claire which is a emacs inspired file browser/finder. There are ruby, python and haskell plugins out there and probably many more coming.

Wrap-up

It really didn't take that long to get up and running with Light Table. Fumbled around a bit before I grasped how to get live code reloading in my running web app from the IDE. The instarepl is really nice for a Clojure noob, and the inline watches (results) are neat as well. Learning Clojure will obviously depend on much more than the editor, but cool tools that help you along the way can't be wrong.

Light Table is supposed to be super extensible, so I'm planning to delve into that. I've already started playing around with a Groovy plugin for Light Table and it's great fun. But anyways that belongs to another blog post.