onsdag 18. juni 2014

A Groovy Light Table client - Step 5: Gradle dependencies in Light Table with dagre-D3

Background

This is the fifth post in my series "A Groovy Light Table client". A blog series about steps I take when trying to build a Groovy plugin for Light Table.

Tapping more into the Potential of Light Table

So far the Groovy Light Table plugin hasn't really showcased the real power of the Light Table Editor. What feature could showcase more of Light Table and at the same time prove useful in many scenarios ? For most projects I have worked on, the number of dependencies and their relationships have usually been non trivial. A couple of years back I wrote a post about showing gradle dependencies as a graphwiz png. Wouldn't it be cool if I could show my gradle dependencies  inline in Light Table ? It would be even cooler if the graph was interactive and provided more/different value than the default dependency reporting you got from Gradle itself

dagre-D3

So what library should I choose for laying out my planned dependency diagram ? My first instinct was something related to D3. However laying out a dot-graph sensibly on my own didn't seem like a challenge I was quite up to. Luckily I found dagre-D3 and it looked to be just the thing I needed.  Of course I would have loved to have found something more clojurish and ideally something that supported an immediate mode ui (akin to Facebook React, but for graphing).  Maybe I didn't look long or well enough but I couldn't find anything obvious so I settled for dagre-D3.  

Gradle dependencies

The second challenge I faced before even getting started was: How would I go about retrieving rich dependency information for my gradle projects using the tooling-api ? The information about dependencies default provided through the tooling api is fairly limited and wouldn't have produced a very informative graph at all. Luckily I found through dialog with the Gradle guys that it should be possible to achieve what I wanted through a custom gradle model.


It's all about the data

When I initially started developing the custom gradle model for retrieving dependency information I designed a data structure that resembled the dependency modelling in Gradle. However after prototyping with dagre and later trying to display multi project dependency graphs I decided to change the design. I ended up with a data structure more similar to that of a graph with nodes and edges.

Custom Gradle Model

To create a Custom Gradle Model you need to create a Gradle Plugin. My plugin got the very informative name "Generic Gradle Model" (naming is hard!).

class GenericGradleModelPlugin implements Plugin {
    final ToolingModelBuilderRegistry registry;

    @Inject
    public GenericGradleModelPlugin(ToolingModelBuilderRegistry registry) {
        this.registry = registry;
    }

    @Override
    void apply(Project project) {
        registry.register(new CustomToolingModelBuilder())
    }
}

The important bit above is registering my custom tooling builder to make it available to the tooling api !


private static class CustomToolingModelBuilder implements ToolingModelBuilder {

// .. other private methods left out for brevity

Map confDiGraph(Configuration conf) {
  def nodeTree = conf.allDependencies
    .findAll {it instanceof ProjectDependency}
    .collect {getProjectDepInfo(it as ProjectDependency)} +
    conf.resolvedConfiguration
        .firstLevelModuleDependencies
        .collect { getDependencyInfo(it) }
    
    def nodes = nodeTree.collect {collectNodeEntry(it)}.flatten().unique {nodeId(it)}
    def edges = nodeTree.collect {
      collectEdge(conf.name, it)
    }.flatten().unique()
    
    [nodes: nodes, edges: edges]
  }
  
  Map projectDeps(Project project) {
    [
      name: project.name,
      group: project.group,
      version: project.version,
      configurations: project.configurations.collectEntries{Configuration conf ->
          [conf.name, confDiGraph(conf)]
      }
    ]
  }
    
  public boolean canBuild(String modelName) {
    modelName.equals(GenericModel.class.getName())
  }
  
  public Object buildAll(String modelName, Project project) {
    new DefaultGenericModel(
      rootDependencies: projectDeps(project),
      subprojectDependencies: project.subprojects.collect {projectDeps(it)}
  }
}

The custom tooling model builder harvests information about all dependencies for all defined configurations in the project. If the project is a multi-project It will collect the same information for each subproject in addition to collect information about interdependencies between the sub projects.


Applying the plugin to gradle projects we connect to 

Before we can retrieve our custom gradle model, we need to apply the plugin to the project in question. I could ask the users to do it themselves, but that wouldn't be particularly user friendly.
Luckily Gradle provides init scripts that you can apply to projects and the tooling api supports doing so. Init scripts allows you to do... well ... init stuff for your projects. Applying a plugin from the outside falls into that category.

initscript {
  repositories {
    maven { url 'http://dl.bintray.com/rundis/maven' }
  }
  dependencies { classpath "no.rundis.gradle:generic-gradle-model:0.0.2" }
}

allprojects {
  apply plugin: org.gradle.tooling.model.generic.GenericGradleModelPlugin
}


Retrieving the model

def genericModel = con.action(new GetGenericModelAction())
                            .withArguments("--init-script", new File("lib/lt-project-init.gradle").absolutePath)
                            .addProgressListener(listener)
                            .run()

private static class GetGenericModelAction implements Serializable, BuildAction {
  @Override
  GenericModel execute(BuildController controller) {
    controller.getModel(GenericModel)
  }
}


To retrieve the model we use a custom build action and applies the plugin implementing the custom model using the --init-script command line argument for gradle.

Voila we have the data we need and we return the dependency info (async) after you have connected to a gradle project.


Show me a graph

The dependency graph and associated logic was separated out to a separate namespace (graph.cljs).
We'll quickly run through some of the highlights of the LightTable clojurescript parts for displaying the dependency graph.


Graph object

(defui dependency-graph-ui [this]
  [:div.graph
   [:div.dependency-graph
    [:svg:svg {:width "650" :height "680"}
     [:svg:g {:transform "translate(20,20)"}]]]])

(object/object* ::dependency-graph
                :tags [:graph.dependency]
                :name "Dependency graph"
                :init (fn [this]
                        (load/js (files/join plugin-dir "js/d3.v3.min.js") :sync)
                        (load/js (files/join plugin-dir "js/dagre-d3.js") :sync)
                        (let [content (dependency-graph-ui this)]
                          content)))

The first step was to create and object that represents the view (and is able to hold the dependency data). The init method is responsible for loading the required graphing libs and then it creates the initial placeholder markup for the graph.


Some behaviours

(behavior ::on-dependencies-loaded
          :desc "Gradle dependencies loaded for selected project"
          :triggers #{:graph.set.dependencies}
          :reaction (fn [this rootDeps subDeps]
                      (object/merge! this {:rootDeps rootDeps
                                           :subDeps subDeps})))


(behavior ::on-show-dependencies
          :desc "Show dependency graph"
          :triggers #{:graph.show.dependencies}
          :reaction (fn [this root-deps]
                      (tabs/add-or-focus! dependency-graph)
                      (default-display this)))

The first behavior is triggered when the groovy backend has finished retrieving the project info, and more specifically the dependencies. If the project is a single project only the rootDeps will contain data.

The second behavior is triggered (by a command) when the user wishes to view the dependency graph for a connected gradle project.

Render Multiproject graph Hightlighs

For multi projects the plugin renders an overview graph where you can see the interdependencies between you sub projects. 

(defn create-multiproject-graph [this]
  (let [g (new dagreD3/Digraph)]
    (doseq [x (:nodes (multi-proj-deps this))]
      (.addNode g (dep-id x) #js {:label (str "<div class='graph-label clickable' data-proj-name='"  
                                              (:name x) "' title='" 
                                              (dep-id x) "'>" 
                                              (:name x) "<br/>" 
                                              (:version x) 
                                              "</div>")}))
    (doseq [x (:edges (multi-proj-deps this))]
      (.addEdge g nil (:a x) (:b x) #js {:label ""}))
    g))

(defn render-multi-deps [this]
  (let [renderer (new dagreD3/Renderer)
        g (dom/$ :g (:content @this))
        svg (dom/$ :svg (:content @this))
        layout (.run renderer (create-multiproject-graph this) (d3-sel g))
        dim (dimensions this)]
    (unbind-select-project this)
    (bind-select-project this)
    (.attr (d3-sel svg) "width" (+ (:w dim) 20))
    (.attr (d3-sel svg) "height" (+ (:h dim) 20))))


The first function shows how we use dagre-D3 to create a logical dot graph representation. We basically add nodes and edges (dep->dep). Most of the code is related to what's rendered inside each node.

The second function shows how we actually layout and display the graph.  In addition we bind click handlers to our custom divs inside the nodes. The click handlers allows for drill down into a detailed graph about each dependency configuration.


End results

Multiproject sample : Ratpack



Project configuration dependencies



Conclusion

I think we achieved some pretty cool things. Maybe not a feature that you need everyday, but its certainly useful to get an overview of your project dependencies. For troubleshooting transitive dependency issues and resolution conflicts etc you might need more details though.

We have certainly showcased that you can do some really cool things with Light Table that you probably wouldn't typically do (easily) with a lot of other editors and IDE's. We have also dug deeper into the gradle tooling api. The gradle tooling api when maturing even more will provide some really cool new options for  JVM IDE integrations. A smart move by gradleware that opens up for integrations from a range of editors, IDE's and specialised tools and applications.

The end result of the dependency graph integration became the largest chunk of the 0.0.6 release.









mandag 26. mai 2014

A Groovy Light Table client - Step 4: Exploring new avenues with Gradle

Background

This is the fourth post in my series "A Groovy Light Table client". A blog series about steps I take when trying to build a Groovy plugin for Light Table.

Initial ponderings

Gradle ships with a Tooling API that makes it fairly easily to integrate with your Gradle projects. Initially I thought that Gradle integration should be a separate plugin that other jvm language plugins could depend on, starting with the Groovy plugin. However after much deliberation I decided to start out with bundling the gradle integration with the Groovy plugin. There is certainly a degree of selecting the easy option to that decision. However I still consider the integration exploratory and I'm not sure how it will pan out. I've settled for a strategy of keeping it logically fairly separate, with a mind to separating gradle specifics out to its own plugin when things become clearer.

Classpath Integration for Groovy "REPL"

In part 3 I talked about some REPL like features where variables that result in bindings are stored in a editor session and used as input to the next evaluation. Since then I've also added the feature of caching method definitions (albeit as closures so I'm sure there are gotchas to that approach as well). 

Anyways wouldn't it be nice If I could also explore my project classes and my projects third party library dependencies in a REPL like fashion ? Hence the idea of providing a Gradle integration. With the Tooling API I should be able to retrieve a class path. So this is where i started. 
Before anyone potentially asking; I will not bother with maven or ant at any point in time, I'll leave that to someone else.

Retrieving the class path as a list from a Gradle project

// Step 1: Connecting to project
def con = GradleConnector.newConnector()
  .forProjectDirectory(projectDir)
  .connect()

// Step 2: Get hold of a project model, for now a IdeaModel provides what we need
def ideaProject = con.model(IdeaProject)
  .addProgressListener(listener)
  .get()

// Step 3: Get list of dependencies
def deps = ideaProject.children
  .dependencies
  .flatten()
  .findAll { it.scope.scope == "COMPILE" }
  .collect {
    [
      name   : it.gradleModuleVersion?.name,
      group  : it.gradleModuleVersion?.group,
      version: it.gradleModuleVersion?.version,
      file   : it.file?.path,
      source : it.source?.path,
      javadoc: it.javadoc?.path
    ] 
  }

def classpathList = deps.file + [new File(projectDir, "build/classes/main").path]

The above code is actually wrapped in a class. Connection and model instances are cached for performance reasons.
  1. We connect to our gradle project. If the project ships with a gradle wrapper (which it should IMO), the gradle connector will use that version (download the distribution even if need be). Otherwise it will use the gradle version of the tooling-api. At the time of writing that's 1.12
  2. The tooling api doesn't really expose as much information by default as you might wish. However it ships with an IdeaModel and an EclipseModel that provides what we need for the purposes of creating a class path. As an Idea user the IdeaModel seemed the right choice ! There is also added a progress listener, which is a callback from the api reporting progress. The progress listener returns each progress event as a string to Light Table so that we can display progress information
  3. We basically navigate the model and extract information about dependencies and put it in a list of maps for ease of jsonifying (useful later !). The location of our projects custom compiled classes are added manually to the class path list (ideally should have been retrieved from the model as well...)

Adding the class path list to our groovy shell before code invocation

private GroovyShell createShell(Map params) {
def transform = new ScriptTransform()
def conf = new CompilerConfiguration()
  conf.addCompilationCustomizers(new ASTTransformationCustomizer(transform))
  conf.addCompilationCustomizers(ic)

  if(params.classPathList) {
    conf.setClasspathList(params.classPathList)
  }

  new GroovyShell(conf)
}


Its basically just a matter of adding the class path list to the CompilerConfiguration we initialise our GroovyShell with. Sweet !
Voila your groovy scripts can invoke any class in your project´s class path.


This addition basically resulted in version 0.0.4

Reporting progress

Groovy

class ProgressReporter implements LTProgressReporter {
    final LTConnection ltCon

    ProgressReporter(LTConnection ltCon) { this.ltCon = ltCon }

    @Override
    void statusChanged(ProgressEvent event) {
        if (event.description?.trim()) {
            reportProgress(event.description)
        }
    }

    void reportProgress(String message) {
        ltCon.sendData([null, "gradle.progress",[msg: message]])
    }
}


  • statusChanges is called by gradle (LTProgressReporter extends the Gradle ProgressListener interface)
  • reportProgress sends the progress information to Light Table 

Light Table

(behavior ::on-gradle-progress
  :desc "Reporting of progress from gradle related tasks"
  :triggers #{:gradle.progress}
  :reaction (fn [this info]
              (notifos/msg* (str "Gradle progress: " (:msg info)) {:timeout 5000})))

The progress behaviour just prints a message to the Light Table status bar.


Executing Gradle Tasks 

There are two parts to this puzzle. One is to retrieve information about what tasks are actually available for the given project. The other is to actually invoke the task (tasks in the future).

Listing tasks Groovy/Server

 // Step 1: Retrieve generic Gradle model
def gradleProject = con.model(GradleProject)
  .addProgressListener(listener)
  .get()

// Step 2: Get list of available tasks
gradleProject.tasks.collect{
  [
    name: it.name,
    displayName: it.displayName,
    description: it.description,
    path: it.path
  ]
}

// Step 3: Send task list to client (omitted, you get the general idea by now !)

Listing tasks in Light Table

The list of tasks is actually retrieved by the Light Table plugin once you select to connect to a gradle project. Furthermore the list is cached in an atom.

(behavior ::on-gradle-projectinfo
  :desc "Gradle project model information"
  :triggers #{:gradle.projectinfo}
  :reaction (fn [this info]
              (object/merge! groovy {::gradle-project-info info})
              (object/assoc-in! cmd/manager [:commands :gradle.task.select :options] (add-selector))))

When the groovy server has finished retrieving the tasks (and other project info) the above behaviour is triggered in Light Table:

  • We store the project info in our Groovy object (an atom)
  • We also update the command for selecting tasks with the new list of tasks. See the section below for details.
(behavior ::set-selected
  :triggers #{:select}
  :reaction (fn [this v]
              (scmd/exec-active! v)))

(defn selector [opts]
  (doto (scmd/filter-list opts)
    (object/add-behavior! ::set-selected)))

(defn get-tasks []
  (->@groovy ::gradle-project-info :tasks))

(defn add-selector []
  (selector {:items (get-tasks)
             :key :name
             :transform #(str "<p>" (:name %4) "</p>"
                              "<p class='binding'>" (:description %4) "</p>")}))

(cmd/command {:command :gradle.task.select
              :desc "Groovy: Select Gradle task"
              :options (add-selector)
              :exec (fn [item]
                      (object/raise groovy :gradle.execute item))})


The above code adds a sub panel to the default sidebar command panel. When you select the command :gradle.task.select it will show a child panel listing the tasks from the get-tasks function.


;; Behavior to actually trigger execution of a selected task from the list above
(behavior ::on-gradle-execute
  :desc "Gradle execute task(s)"
  :triggers #{:gradle.execute}
  :reaction (fn [this task]
              (clients/send
                (clients/by-name "Groovy")
                :gradle.execute
                {:tasks [(:name task)]})))

Once you have selected a task the above behaviour is triggered. We get hold of an editor agnostic groovy client and send an execute task message with a list of task (currently always just one). The data we send will be extended in the future to support multiple tasks and build arguments.

Server side Task execution 

// Generic execute task function
def execute(Map params, Closure onComplete) {
    def resultHandler = [
        onComplete: {Object result ->
            onComplete status: "OK"
        },
        onFailure: {GradleConnectionException failure ->
            onComplete status: "ERROR", error: failure
        }
    ] as ResultHandler


    con.newBuild()
    .addProgressListener(listener)
    .forTasks(params.tasks as String[])
    .run(resultHandler)
}

Here we use the asynch features of the Gradle Tooling API. Executing a task may actually take a while so it certainly makes sense. Callers of the execute method will receive a callback (onComplete) once task execution is completed successfully (of failed).


projectConnection.execute(params) {Map result ->    
    ltConnection.sendData([
        null,
        result.status == "ERROR" ? "gradle.execute.err" : "gradle.execute.success",
        result
    ])
}

We invoke the execute method with a closure argument and return the results (success/failure) back to Light Table.


This brings us pretty much up to version 0.0.5

Summary

Well we covered a lot of ground here. We can now call any class that's in your Gradle project's class path from a groovy editor in Light Table. We've also started on providing Gradle features that are language agnostic. Starting with support for listing and executing tasks in your gradle project.
We've added decent progress reporting and performance seems to be pretty good too. Looks like we have something we can build further upon !

I have lots of ideas; Infinitesting,  single test with inline results, compile single file, grails integration ? etc etc. I also really want to show project dependencies in a graph. However before I can do any of those things I need to extend the tooling api with custom models ... and/or maybe I should see if I can contribute to the gradle project in extending the tooling-api with a richer generic project model. 

We'll have to wait and see. Next week I'm off to gr8conf.eu in Copenhagen. Really looking forward to meeting up with all the great Groovy dudes/dudettes.  And who knows maybe the hackergarten evening will result in something new and exciting ! 

mandag 12. mai 2014

A Groovy Light Table client - Step 3: Running out of quick wins

Background

This is the third post in my series "A Groovy Light Table client". A blog series about steps I take when trying to build a Groovy plugin for Light Table.

Running out of quick wins

After 0.0.2 of the plugin was released I was pretty happy. I had something that I could actually use as an alternative Groovy Console. However I was keen to keep up the flow so I figured I would try to implement an autocomplete feature. The task proved rather daunting. Not so much from the Light Table side of things, but rather from the Groovy side. First I tried to see if I could reuse anything from GroovySh, but that didn't look to promising. After that I tried to get my head around whether I could somehow reuse something from IntelliJ or Eclipse. I failed to see the light so I gave up that endeavour. Secondly I tried to see if there was an easy way to provide an inline documentation feature. Sadly I couldn't find much reusable and something with a small foot print here either. Someone should make a REST based doc search feature for Groovy Docs one day !

I turned my attention to a couple of other plugins that I thought would be useful for Light Table. I created a Buster.JS plugin InstaBuster for easily running JavaScript tests. I also created a snippets/code templates plugin lt-snippets and some snippet collections, among them a small collection of groovy snippets.

There is just no way I could ever compete with the mainstream IDE's, but then again that was never really the original intention. But even with limited capacity it should still be possible to provide useful groovy support and maybe even something fairly unique within the super hackable framework of Light Table.

A (small) step towards a Groovy REPL

After working with Light Table and the Clojure/ClojureScript REPL I have grown very fond of that exploratory nature of working. Is there anything I could do with the plugin to give a more REPL like feel ? Well a small but helpful step would be to be able to remember what I previously have evaluated ...

Groovy Bindings

A simple but still useful mechanism would be to cache bindings from script execution between evals.

// from the evalGroovy method
def evalResult = scriptExecutor.execute(data.code, clientSessions.get(currentClientId))
clientSessions.put(currentClientId, evalResult.bindings)

Each editor in Light Table gets it's own unique Id. So I just created a simple cache "ClientSessions" that hold a map of binding variables, mapped my that Id. When executing a script the current binding variables are applied to the script and after the script has executed the resulting binding variables are added/replaced in the cache. Dead simple really.

Clearing bindings - Light Table

I figured it would be handy to be able to clear any stored bindings. So a new command and behaviour was created in Light Table

;; Behavior for clearing bindings
(behavior ::on-clear-bindings
          :desc "Clear cached bindings for this editor"
          :triggers #{:on.clear.bindings}
          :reaction (fn [editor]
                      (let [info (:info @editor)
                            cl (eval/get-client! {:command :editor.clear.groovy
                                                    :origin editor
                                                    :info info
                                                    :create try-connect})]
                          (clients/send cl
                                        :editor.clear.groovy info
                                        :only editor))))

;; Command that allows a new keyboard bindable action for invoking the behaviour above
(cmd/command {:command :clear-bindings
              :desc "Groovy: Clear bindings for current editor"
              :exec (fn []
                      (when-let [ed (pool/last-active)]
                        (object/raise ed :on.clear.bindings)))})

The command retrieves the currently active editor and triggers the behaviour. The behaviour retrieves a client connection (or creates one if one shouldn't exist) and calls the server (groovy).


// Wiring up the behaviour in groovy.behaviors
:editor.groovy [:lt.plugins.groovy/on-eval
                     :lt.plugins.groovy/on-eval.one
                     :lt.plugins.groovy/on-clear-bindings
                     :lt.plugins.groovy/groovy-res
                     :lt.plugins.groovy/groovy-err
                     [:lt.object/add-tag :watchable]]

The final piece of the puzzle from the Light Table side is to attach the behavior to the :editor.groovy tag. This enables the behavior to be available from any editor that is tagged with this tag.

Clearing bindings - Groovy

// The command dispatch got a new command
case "editor.clear.groovy":
   clientSessions.clear(currentClientId)
   break;

The code above will just nuke any stored binding variables.

Conclusions

A tiny step that allows you to eval groovy expressions step by step. Anything that results in a binding is stored between evals. Obviously it's a bit limited in that you'll run into the familiar trap of trying to use def and be surprised(/annoyed) that it won't remember that or if you define a class it won't remember that either. It's probably possible to cater for some of these traps, but maybe not within the realms of a quick win. 

Anyways the end result is Version 0.0.3 !

Next steps

Firstly there is a Screencast brewing. After that I think a Light Table Gradle plugin is coming before the Groovy plugin gets any further updates. A pluggable Gradle plugin would enable the Groovy plugin to quite easily get the class path for you project. This would allow you to  explore your projects code in a REPL (-like) way. Exploratory testing FTW !




onsdag 7. mai 2014

Creating and Using Snippets in Light Table


An introduction on how to create and use simple and more elaborate snippets with the lt-snippets Light Table plugin. If you can't see the video above here's the link directly to youtube

søndag 23. februar 2014

A Groovy Light Table client - Step 2: Evaluating Code

Background

This is the second post in my series "A Groovy Light Table client". A blog series about steps I take when trying to build a Groovy plugin for Light Table

In this post I will take you through some of the steps I went through to get Light Table to evaluate groovy (script) code and show results inline in the editor.


How did we get here ?

Evaluate contents of editor (cmd/ctrl + shift + enter)

(behavior ::on-eval
          :desc "Groovy: Eval current editor"
          :triggers #{:eval}
          :reaction (fn [editor]
                      (object/raise groovy :eval! {:origin editor
                                                   :info (assoc (@editor :info)
                                                           :code (ed/->val editor)
                                                           :meta {:start 0, :end (ed/last-line editor)})})))

This behavior triggers on ":eval", which is triggered to any editor (on cmd/ctrl + shift + enter in default key mapping). We just get hold of the text from the editor and gather some meta info  and trigger a ":eval!" behavior on the groovy "mother" object defined in the last blog post.

Evaluate current line/selection

(behavior ::on-eval.one
          :desc "Groovy: Eval current selection"
          :triggers #{:eval.one}
          :reaction (fn [editor]
                      (let [pos (ed/->cursor editor)
                            info (conj (:info @editor) 
                                  (if (ed/selection? editor) 
                                    {:code (ed/selection editor) :meta {:start (-> (ed/->cursor editor "start") :line)
                                                                        :end (-> (ed/->cursor editor "end") :line)}}
                                    {:pos pos :code (ed/line editor (:line pos)) :meta {:start (:line pos) :end (:line pos)}}))]
                        (object/raise groovy :eval! {:origin editor :info info}))))

The only difference here is that we gather the code for the current line or current selection. Then we trigger the same behavior as for evaluating the whole editor.


Our groovy Eval!

(behavior ::eval!
          :triggers #{:eval!}
          :reaction (fn [this event]
                      (let [{:keys [info origin]} event
                            client (-> @origin :client :default)]
                        (notifos/working "Evaluating groovy...")
                        (clients/send (eval/get-client! {:command :editor.eval.groovy
                                                         :origin origin
                                                         :info info
                                                         :create try-connect})
                                      :editor.eval.groovy info 
                                      :only origin))))

This behavior is what actually sends off a eval request to the groovy client. Quite a lot happens under the hood (by help of inbuilt LightTable behaviors):

  • It tries to find a client (connection) for the editor
  • If no connection exists it will try to create a new one. On create it will invoke the try-connect function that we defined for the gui connect/connect bar behavior in the previous blog post
  • Once connected it will jsonify our parameters and send them off to our groovy client
The JSON might look something like:
[89,
"editor.eval.groovy",
  {"line-ending":"\n",
   "name":"sample.groovy",
   "type-name":"Groovy",
   "path":"/Users/mrundberget/Library/Application Support/LightTable/plugins/Groovy/sample.groovy",
   "mime":"text/x-groovy",
   "tags":["editor.groovy"],
   "code":"println \"hello\"",
   "meta":{"start":22,"end":22}}]


  • The first param is the client id for the editor that triggered the behavior. This client Id doesn't represent the same as a connection id (ref previous blog post). Many editors may share the same connection !
  • The second param is the command (our groovy client will of course support many different commands, this is one of them)
  • The third and last parameter is our info. The code is the essential bit, but some of the meta information, like line info comes in handy when handling the response later on

The actual groovy evaluation

Command dispatch


ltClient.withStreams { input, output ->
  try {
    input.eachLine { line ->
    def (currentClientId, command, data) = new JsonSlurper().parseText(line)
    switch (command) {
    case "client.close":
      stop()
      break
    case "editor.eval.groovy":
      evalGroovy(data, currentClientId)
      break
   default:
     log "Invalid command: $command"
  }
  // ...


    

We parse any lines received from Light Table and based on the command received invokes the appropriate handler. In this case evalGroovy.

Eval groovy


private void evalGroovy(data, currentClientId) {
  def evalResult = scriptExecutor.execute(data.code)

  def resultParams = [meta: data.meta]
  if (evalResult.out) {
    resultParams << [out: evalResult.out]
  }
  if(evalResult.exprValues) {
    resultParams << [result: convertToClientVals(evalResult.exprValues)]
  }

  if (!evalResult.err) {
    data = [currentClientId?.toInteger(), "groovy.res", resultParams]
  } else {
    data = [currentClientId?.toInteger(), "groovy.err", [ex: evalResult.err] + resultParams]
  }
  sendData data
}

The first and most significant line is where we evaluate the groovy code received. This post would be too long if we went into all the details of what it does, but here's a high-level summary:

  • We basically create a GroovyShell and compile our code to a script. Normally that would just compile a Script class. However we  wish to collect a lot more information than you typically would get from default groovy script execution. So we do an AST transformation on the script class and add a custom abstract script class as a base class for the compiled script class.  This allows us to inject behavior and wrap statement execution (all compiled into the script for optimal performance).  That way we are able to collect information about values for most types of statements. We collect line number and value (each line could end up having many values :-) )
  • We run the script (capturing system.out and system.err). 
  • The function returns:
    • Anything written to standard out (println etc)
    • Errors if any and line number for error where possible
    • A list for of maps with line number and value(s)  

Most of the AST stuff is not something I've written. It's been contributed by Jim White after I posted a question on the groovy-user mailing list. I asked for advice on which way to proceed and the response from the groovy community was awesome. Jim in particular was more than eager to contribute to the plugin. OpenSource rocks ! So when I say we, I sometimes mean we literally.

Anyways, based on the results of the script execution we notify Light Table to trigger either a ":groovy.res" behavior or a "groovy.err" behavior.

The json response for sendData for a successful execution might look something like:

[89,
 "groovy.res",
 {"meta":{"start":22,"end":23},"out":"hello\nmama\n","result":[{"line":1,"values":["null"]},{"line":2,"values":["null"]}]}]


Handling the evaluation results in Light Table


(defn notify-of-results [editor res]
  (doseq [ln (:result res)]
    (let [lineNo (+ (:line ln) (-> res :meta :start) -1)]
      (object/raise editor :editor.result (clojure.string/join " " (:values ln)) {:line lineNo :start-line lineNo}))))

(behavior ::groovy-res
          :triggers #{:groovy.res}
          :reaction (fn [editor res]
                      (notifos/done-working)
                      (when-let [o (:out res)] (.log js/console o))
                      (notify-of-results editor res)))

(defn notify-of-error [editor res]
  (let [lineNo (+ (-> res :ex :line) (-> res :meta :start) -1)]
    (object/raise editor :editor.exception (:ex res) {:line lineNo :start-line lineNo'})))

(behavior ::groovy-err
          :triggers #{:groovy.err}
          :reaction (fn [editor res]
                      (object/raise editor :groovy.res res)
                      (notify-of-error editor res)))

These are the behavior definitions that handles either successful or evaluation of scripts with errors. Basically we:
  • Print to the Light Table Console anything that was captured to system.out/system.err by our groovy evaluation
  • Show inline results for each line, multiple results for a line are space separated. For showing inline results we are using a predefined Light Table behavior (:editor.result)
  • If the behavior is to handle an error, we show evaluation results up until the script exception. In addition we display details (stack trace) for the exception at the line in the script it occurred

Wiring it all up

groovy.behaviors

{:+ {:app [(:lt.objs.plugins/load-js ["codemirror/groovy.js", "groovy_compiled.js"])]
     :clients []
     :editor.groovy [:lt.plugins.groovy/on-eval
                     :lt.plugins.groovy/on-eval.one
                     :lt.plugins.groovy/groovy-res
                     :lt.plugins.groovy/groovy-err
                     [:lt.object/add-tag :watchable]]
     :files [(:lt.objs.files/file-types
              [{:name "Groovy" :exts [:groovy] :mime "text/x-groovy" :tags [:editor.groovy]}])]
     :groovy.lang [:lt.plugins.groovy/eval!
                   :lt.plugins.groovy/connect]}}

The eval and results/err behaviors are defined for the editor tag. So they are only applicable for editors marked as groovy editors. Any editor open with a file name ending in .groovy will automatically be attached to a editor.groovy tag. (You can also set it manually cmd+space -> "Editor: Set current editor syntax").
The ":eval!" behavior is defined for the :groovy.lang tag. Its tied to our groovy mother object just like the connect behavior. These behaviors are totally groovy client specific, whilst the other behaviors are less so (although not exactly generic as they are now…)

Wrap up

A little bit of plumbing was needed to get this set up. But the hard parts was really coming up with the groovy AST transformation stuff. I guess by now you might have started getting an inkling that Light Table is fairly composable ? It really is super flexible. You don't like the behavior for handling inline results for the groovy plugin ? You could easily write your own and wire it up in your user.behaviors file in Light Table. It's wicked cool, actually it really is your editor !

Yesterday I released version 0.0.2 of the Groovy LightTable plugin. Its available through the Light Table plugin manager, or if you wish to play with the code or maybe feel like contributing feel free to fork the repo at : https://github.com/rundis/LightTable-Groovy. Pull requests are welcome.

So where to next ? I'd really like to try and create an InstaRepl editor for the plugin. A groovy script editor that evaluates code as you type. There's gotta be one or two challenges related to that. A quick win might be to provide groovy api documentation from inside Light Table. I'll let you know what happens in the next post.

Disclaimer: I might have misunderstood some nuances of LIght Table, but hopefully I'm roughly on track. If you see anything glaringly wrong, do let me know.