onsdag 18. juni 2014

A Groovy Light Table client - Step 5: Gradle dependencies in Light Table with dagre-D3

Background

This is the fifth post in my series "A Groovy Light Table client". A blog series about steps I take when trying to build a Groovy plugin for Light Table.

Tapping more into the Potential of Light Table

So far the Groovy Light Table plugin hasn't really showcased the real power of the Light Table Editor. What feature could showcase more of Light Table and at the same time prove useful in many scenarios ? For most projects I have worked on, the number of dependencies and their relationships have usually been non trivial. A couple of years back I wrote a post about showing gradle dependencies as a graphwiz png. Wouldn't it be cool if I could show my gradle dependencies  inline in Light Table ? It would be even cooler if the graph was interactive and provided more/different value than the default dependency reporting you got from Gradle itself

dagre-D3

So what library should I choose for laying out my planned dependency diagram ? My first instinct was something related to D3. However laying out a dot-graph sensibly on my own didn't seem like a challenge I was quite up to. Luckily I found dagre-D3 and it looked to be just the thing I needed.  Of course I would have loved to have found something more clojurish and ideally something that supported an immediate mode ui (akin to Facebook React, but for graphing).  Maybe I didn't look long or well enough but I couldn't find anything obvious so I settled for dagre-D3.  

Gradle dependencies

The second challenge I faced before even getting started was: How would I go about retrieving rich dependency information for my gradle projects using the tooling-api ? The information about dependencies default provided through the tooling api is fairly limited and wouldn't have produced a very informative graph at all. Luckily I found through dialog with the Gradle guys that it should be possible to achieve what I wanted through a custom gradle model.


It's all about the data

When I initially started developing the custom gradle model for retrieving dependency information I designed a data structure that resembled the dependency modelling in Gradle. However after prototyping with dagre and later trying to display multi project dependency graphs I decided to change the design. I ended up with a data structure more similar to that of a graph with nodes and edges.

Custom Gradle Model

To create a Custom Gradle Model you need to create a Gradle Plugin. My plugin got the very informative name "Generic Gradle Model" (naming is hard!).

class GenericGradleModelPlugin implements Plugin {
    final ToolingModelBuilderRegistry registry;

    @Inject
    public GenericGradleModelPlugin(ToolingModelBuilderRegistry registry) {
        this.registry = registry;
    }

    @Override
    void apply(Project project) {
        registry.register(new CustomToolingModelBuilder())
    }
}

The important bit above is registering my custom tooling builder to make it available to the tooling api !


private static class CustomToolingModelBuilder implements ToolingModelBuilder {

// .. other private methods left out for brevity

Map confDiGraph(Configuration conf) {
  def nodeTree = conf.allDependencies
    .findAll {it instanceof ProjectDependency}
    .collect {getProjectDepInfo(it as ProjectDependency)} +
    conf.resolvedConfiguration
        .firstLevelModuleDependencies
        .collect { getDependencyInfo(it) }
    
    def nodes = nodeTree.collect {collectNodeEntry(it)}.flatten().unique {nodeId(it)}
    def edges = nodeTree.collect {
      collectEdge(conf.name, it)
    }.flatten().unique()
    
    [nodes: nodes, edges: edges]
  }
  
  Map projectDeps(Project project) {
    [
      name: project.name,
      group: project.group,
      version: project.version,
      configurations: project.configurations.collectEntries{Configuration conf ->
          [conf.name, confDiGraph(conf)]
      }
    ]
  }
    
  public boolean canBuild(String modelName) {
    modelName.equals(GenericModel.class.getName())
  }
  
  public Object buildAll(String modelName, Project project) {
    new DefaultGenericModel(
      rootDependencies: projectDeps(project),
      subprojectDependencies: project.subprojects.collect {projectDeps(it)}
  }
}

The custom tooling model builder harvests information about all dependencies for all defined configurations in the project. If the project is a multi-project It will collect the same information for each subproject in addition to collect information about interdependencies between the sub projects.


Applying the plugin to gradle projects we connect to 

Before we can retrieve our custom gradle model, we need to apply the plugin to the project in question. I could ask the users to do it themselves, but that wouldn't be particularly user friendly.
Luckily Gradle provides init scripts that you can apply to projects and the tooling api supports doing so. Init scripts allows you to do... well ... init stuff for your projects. Applying a plugin from the outside falls into that category.

initscript {
  repositories {
    maven { url 'http://dl.bintray.com/rundis/maven' }
  }
  dependencies { classpath "no.rundis.gradle:generic-gradle-model:0.0.2" }
}

allprojects {
  apply plugin: org.gradle.tooling.model.generic.GenericGradleModelPlugin
}


Retrieving the model

def genericModel = con.action(new GetGenericModelAction())
                            .withArguments("--init-script", new File("lib/lt-project-init.gradle").absolutePath)
                            .addProgressListener(listener)
                            .run()

private static class GetGenericModelAction implements Serializable, BuildAction {
  @Override
  GenericModel execute(BuildController controller) {
    controller.getModel(GenericModel)
  }
}


To retrieve the model we use a custom build action and applies the plugin implementing the custom model using the --init-script command line argument for gradle.

Voila we have the data we need and we return the dependency info (async) after you have connected to a gradle project.


Show me a graph

The dependency graph and associated logic was separated out to a separate namespace (graph.cljs).
We'll quickly run through some of the highlights of the LightTable clojurescript parts for displaying the dependency graph.


Graph object

(defui dependency-graph-ui [this]
  [:div.graph
   [:div.dependency-graph
    [:svg:svg {:width "650" :height "680"}
     [:svg:g {:transform "translate(20,20)"}]]]])

(object/object* ::dependency-graph
                :tags [:graph.dependency]
                :name "Dependency graph"
                :init (fn [this]
                        (load/js (files/join plugin-dir "js/d3.v3.min.js") :sync)
                        (load/js (files/join plugin-dir "js/dagre-d3.js") :sync)
                        (let [content (dependency-graph-ui this)]
                          content)))

The first step was to create and object that represents the view (and is able to hold the dependency data). The init method is responsible for loading the required graphing libs and then it creates the initial placeholder markup for the graph.


Some behaviours

(behavior ::on-dependencies-loaded
          :desc "Gradle dependencies loaded for selected project"
          :triggers #{:graph.set.dependencies}
          :reaction (fn [this rootDeps subDeps]
                      (object/merge! this {:rootDeps rootDeps
                                           :subDeps subDeps})))


(behavior ::on-show-dependencies
          :desc "Show dependency graph"
          :triggers #{:graph.show.dependencies}
          :reaction (fn [this root-deps]
                      (tabs/add-or-focus! dependency-graph)
                      (default-display this)))

The first behavior is triggered when the groovy backend has finished retrieving the project info, and more specifically the dependencies. If the project is a single project only the rootDeps will contain data.

The second behavior is triggered (by a command) when the user wishes to view the dependency graph for a connected gradle project.

Render Multiproject graph Hightlighs

For multi projects the plugin renders an overview graph where you can see the interdependencies between you sub projects. 

(defn create-multiproject-graph [this]
  (let [g (new dagreD3/Digraph)]
    (doseq [x (:nodes (multi-proj-deps this))]
      (.addNode g (dep-id x) #js {:label (str "<div class='graph-label clickable' data-proj-name='"  
                                              (:name x) "' title='" 
                                              (dep-id x) "'>" 
                                              (:name x) "<br/>" 
                                              (:version x) 
                                              "</div>")}))
    (doseq [x (:edges (multi-proj-deps this))]
      (.addEdge g nil (:a x) (:b x) #js {:label ""}))
    g))

(defn render-multi-deps [this]
  (let [renderer (new dagreD3/Renderer)
        g (dom/$ :g (:content @this))
        svg (dom/$ :svg (:content @this))
        layout (.run renderer (create-multiproject-graph this) (d3-sel g))
        dim (dimensions this)]
    (unbind-select-project this)
    (bind-select-project this)
    (.attr (d3-sel svg) "width" (+ (:w dim) 20))
    (.attr (d3-sel svg) "height" (+ (:h dim) 20))))


The first function shows how we use dagre-D3 to create a logical dot graph representation. We basically add nodes and edges (dep->dep). Most of the code is related to what's rendered inside each node.

The second function shows how we actually layout and display the graph.  In addition we bind click handlers to our custom divs inside the nodes. The click handlers allows for drill down into a detailed graph about each dependency configuration.


End results

Multiproject sample : Ratpack



Project configuration dependencies



Conclusion

I think we achieved some pretty cool things. Maybe not a feature that you need everyday, but its certainly useful to get an overview of your project dependencies. For troubleshooting transitive dependency issues and resolution conflicts etc you might need more details though.

We have certainly showcased that you can do some really cool things with Light Table that you probably wouldn't typically do (easily) with a lot of other editors and IDE's. We have also dug deeper into the gradle tooling api. The gradle tooling api when maturing even more will provide some really cool new options for  JVM IDE integrations. A smart move by gradleware that opens up for integrations from a range of editors, IDE's and specialised tools and applications.

The end result of the dependency graph integration became the largest chunk of the 0.0.6 release.









mandag 26. mai 2014

A Groovy Light Table client - Step 4: Exploring new avenues with Gradle

Background

This is the fourth post in my series "A Groovy Light Table client". A blog series about steps I take when trying to build a Groovy plugin for Light Table.

Initial ponderings

Gradle ships with a Tooling API that makes it fairly easily to integrate with your Gradle projects. Initially I thought that Gradle integration should be a separate plugin that other jvm language plugins could depend on, starting with the Groovy plugin. However after much deliberation I decided to start out with bundling the gradle integration with the Groovy plugin. There is certainly a degree of selecting the easy option to that decision. However I still consider the integration exploratory and I'm not sure how it will pan out. I've settled for a strategy of keeping it logically fairly separate, with a mind to separating gradle specifics out to its own plugin when things become clearer.

Classpath Integration for Groovy "REPL"

In part 3 I talked about some REPL like features where variables that result in bindings are stored in a editor session and used as input to the next evaluation. Since then I've also added the feature of caching method definitions (albeit as closures so I'm sure there are gotchas to that approach as well). 

Anyways wouldn't it be nice If I could also explore my project classes and my projects third party library dependencies in a REPL like fashion ? Hence the idea of providing a Gradle integration. With the Tooling API I should be able to retrieve a class path. So this is where i started. 
Before anyone potentially asking; I will not bother with maven or ant at any point in time, I'll leave that to someone else.

Retrieving the class path as a list from a Gradle project

// Step 1: Connecting to project
def con = GradleConnector.newConnector()
  .forProjectDirectory(projectDir)
  .connect()

// Step 2: Get hold of a project model, for now a IdeaModel provides what we need
def ideaProject = con.model(IdeaProject)
  .addProgressListener(listener)
  .get()

// Step 3: Get list of dependencies
def deps = ideaProject.children
  .dependencies
  .flatten()
  .findAll { it.scope.scope == "COMPILE" }
  .collect {
    [
      name   : it.gradleModuleVersion?.name,
      group  : it.gradleModuleVersion?.group,
      version: it.gradleModuleVersion?.version,
      file   : it.file?.path,
      source : it.source?.path,
      javadoc: it.javadoc?.path
    ] 
  }

def classpathList = deps.file + [new File(projectDir, "build/classes/main").path]

The above code is actually wrapped in a class. Connection and model instances are cached for performance reasons.
  1. We connect to our gradle project. If the project ships with a gradle wrapper (which it should IMO), the gradle connector will use that version (download the distribution even if need be). Otherwise it will use the gradle version of the tooling-api. At the time of writing that's 1.12
  2. The tooling api doesn't really expose as much information by default as you might wish. However it ships with an IdeaModel and an EclipseModel that provides what we need for the purposes of creating a class path. As an Idea user the IdeaModel seemed the right choice ! There is also added a progress listener, which is a callback from the api reporting progress. The progress listener returns each progress event as a string to Light Table so that we can display progress information
  3. We basically navigate the model and extract information about dependencies and put it in a list of maps for ease of jsonifying (useful later !). The location of our projects custom compiled classes are added manually to the class path list (ideally should have been retrieved from the model as well...)

Adding the class path list to our groovy shell before code invocation

private GroovyShell createShell(Map params) {
def transform = new ScriptTransform()
def conf = new CompilerConfiguration()
  conf.addCompilationCustomizers(new ASTTransformationCustomizer(transform))
  conf.addCompilationCustomizers(ic)

  if(params.classPathList) {
    conf.setClasspathList(params.classPathList)
  }

  new GroovyShell(conf)
}


Its basically just a matter of adding the class path list to the CompilerConfiguration we initialise our GroovyShell with. Sweet !
Voila your groovy scripts can invoke any class in your project´s class path.


This addition basically resulted in version 0.0.4

Reporting progress

Groovy

class ProgressReporter implements LTProgressReporter {
    final LTConnection ltCon

    ProgressReporter(LTConnection ltCon) { this.ltCon = ltCon }

    @Override
    void statusChanged(ProgressEvent event) {
        if (event.description?.trim()) {
            reportProgress(event.description)
        }
    }

    void reportProgress(String message) {
        ltCon.sendData([null, "gradle.progress",[msg: message]])
    }
}


  • statusChanges is called by gradle (LTProgressReporter extends the Gradle ProgressListener interface)
  • reportProgress sends the progress information to Light Table 

Light Table

(behavior ::on-gradle-progress
  :desc "Reporting of progress from gradle related tasks"
  :triggers #{:gradle.progress}
  :reaction (fn [this info]
              (notifos/msg* (str "Gradle progress: " (:msg info)) {:timeout 5000})))

The progress behaviour just prints a message to the Light Table status bar.


Executing Gradle Tasks 

There are two parts to this puzzle. One is to retrieve information about what tasks are actually available for the given project. The other is to actually invoke the task (tasks in the future).

Listing tasks Groovy/Server

 // Step 1: Retrieve generic Gradle model
def gradleProject = con.model(GradleProject)
  .addProgressListener(listener)
  .get()

// Step 2: Get list of available tasks
gradleProject.tasks.collect{
  [
    name: it.name,
    displayName: it.displayName,
    description: it.description,
    path: it.path
  ]
}

// Step 3: Send task list to client (omitted, you get the general idea by now !)

Listing tasks in Light Table

The list of tasks is actually retrieved by the Light Table plugin once you select to connect to a gradle project. Furthermore the list is cached in an atom.

(behavior ::on-gradle-projectinfo
  :desc "Gradle project model information"
  :triggers #{:gradle.projectinfo}
  :reaction (fn [this info]
              (object/merge! groovy {::gradle-project-info info})
              (object/assoc-in! cmd/manager [:commands :gradle.task.select :options] (add-selector))))

When the groovy server has finished retrieving the tasks (and other project info) the above behaviour is triggered in Light Table:

  • We store the project info in our Groovy object (an atom)
  • We also update the command for selecting tasks with the new list of tasks. See the section below for details.
(behavior ::set-selected
  :triggers #{:select}
  :reaction (fn [this v]
              (scmd/exec-active! v)))

(defn selector [opts]
  (doto (scmd/filter-list opts)
    (object/add-behavior! ::set-selected)))

(defn get-tasks []
  (->@groovy ::gradle-project-info :tasks))

(defn add-selector []
  (selector {:items (get-tasks)
             :key :name
             :transform #(str "<p>" (:name %4) "</p>"
                              "<p class='binding'>" (:description %4) "</p>")}))

(cmd/command {:command :gradle.task.select
              :desc "Groovy: Select Gradle task"
              :options (add-selector)
              :exec (fn [item]
                      (object/raise groovy :gradle.execute item))})


The above code adds a sub panel to the default sidebar command panel. When you select the command :gradle.task.select it will show a child panel listing the tasks from the get-tasks function.


;; Behavior to actually trigger execution of a selected task from the list above
(behavior ::on-gradle-execute
  :desc "Gradle execute task(s)"
  :triggers #{:gradle.execute}
  :reaction (fn [this task]
              (clients/send
                (clients/by-name "Groovy")
                :gradle.execute
                {:tasks [(:name task)]})))

Once you have selected a task the above behaviour is triggered. We get hold of an editor agnostic groovy client and send an execute task message with a list of task (currently always just one). The data we send will be extended in the future to support multiple tasks and build arguments.

Server side Task execution 

// Generic execute task function
def execute(Map params, Closure onComplete) {
    def resultHandler = [
        onComplete: {Object result ->
            onComplete status: "OK"
        },
        onFailure: {GradleConnectionException failure ->
            onComplete status: "ERROR", error: failure
        }
    ] as ResultHandler


    con.newBuild()
    .addProgressListener(listener)
    .forTasks(params.tasks as String[])
    .run(resultHandler)
}

Here we use the asynch features of the Gradle Tooling API. Executing a task may actually take a while so it certainly makes sense. Callers of the execute method will receive a callback (onComplete) once task execution is completed successfully (of failed).


projectConnection.execute(params) {Map result ->    
    ltConnection.sendData([
        null,
        result.status == "ERROR" ? "gradle.execute.err" : "gradle.execute.success",
        result
    ])
}

We invoke the execute method with a closure argument and return the results (success/failure) back to Light Table.


This brings us pretty much up to version 0.0.5

Summary

Well we covered a lot of ground here. We can now call any class that's in your Gradle project's class path from a groovy editor in Light Table. We've also started on providing Gradle features that are language agnostic. Starting with support for listing and executing tasks in your gradle project.
We've added decent progress reporting and performance seems to be pretty good too. Looks like we have something we can build further upon !

I have lots of ideas; Infinitesting,  single test with inline results, compile single file, grails integration ? etc etc. I also really want to show project dependencies in a graph. However before I can do any of those things I need to extend the tooling api with custom models ... and/or maybe I should see if I can contribute to the gradle project in extending the tooling-api with a richer generic project model. 

We'll have to wait and see. Next week I'm off to gr8conf.eu in Copenhagen. Really looking forward to meeting up with all the great Groovy dudes/dudettes.  And who knows maybe the hackergarten evening will result in something new and exciting ! 

mandag 12. mai 2014

A Groovy Light Table client - Step 3: Running out of quick wins

Background

This is the third post in my series "A Groovy Light Table client". A blog series about steps I take when trying to build a Groovy plugin for Light Table.

Running out of quick wins

After 0.0.2 of the plugin was released I was pretty happy. I had something that I could actually use as an alternative Groovy Console. However I was keen to keep up the flow so I figured I would try to implement an autocomplete feature. The task proved rather daunting. Not so much from the Light Table side of things, but rather from the Groovy side. First I tried to see if I could reuse anything from GroovySh, but that didn't look to promising. After that I tried to get my head around whether I could somehow reuse something from IntelliJ or Eclipse. I failed to see the light so I gave up that endeavour. Secondly I tried to see if there was an easy way to provide an inline documentation feature. Sadly I couldn't find much reusable and something with a small foot print here either. Someone should make a REST based doc search feature for Groovy Docs one day !

I turned my attention to a couple of other plugins that I thought would be useful for Light Table. I created a Buster.JS plugin InstaBuster for easily running JavaScript tests. I also created a snippets/code templates plugin lt-snippets and some snippet collections, among them a small collection of groovy snippets.

There is just no way I could ever compete with the mainstream IDE's, but then again that was never really the original intention. But even with limited capacity it should still be possible to provide useful groovy support and maybe even something fairly unique within the super hackable framework of Light Table.

A (small) step towards a Groovy REPL

After working with Light Table and the Clojure/ClojureScript REPL I have grown very fond of that exploratory nature of working. Is there anything I could do with the plugin to give a more REPL like feel ? Well a small but helpful step would be to be able to remember what I previously have evaluated ...

Groovy Bindings

A simple but still useful mechanism would be to cache bindings from script execution between evals.

// from the evalGroovy method
def evalResult = scriptExecutor.execute(data.code, clientSessions.get(currentClientId))
clientSessions.put(currentClientId, evalResult.bindings)

Each editor in Light Table gets it's own unique Id. So I just created a simple cache "ClientSessions" that hold a map of binding variables, mapped my that Id. When executing a script the current binding variables are applied to the script and after the script has executed the resulting binding variables are added/replaced in the cache. Dead simple really.

Clearing bindings - Light Table

I figured it would be handy to be able to clear any stored bindings. So a new command and behaviour was created in Light Table

;; Behavior for clearing bindings
(behavior ::on-clear-bindings
          :desc "Clear cached bindings for this editor"
          :triggers #{:on.clear.bindings}
          :reaction (fn [editor]
                      (let [info (:info @editor)
                            cl (eval/get-client! {:command :editor.clear.groovy
                                                    :origin editor
                                                    :info info
                                                    :create try-connect})]
                          (clients/send cl
                                        :editor.clear.groovy info
                                        :only editor))))

;; Command that allows a new keyboard bindable action for invoking the behaviour above
(cmd/command {:command :clear-bindings
              :desc "Groovy: Clear bindings for current editor"
              :exec (fn []
                      (when-let [ed (pool/last-active)]
                        (object/raise ed :on.clear.bindings)))})

The command retrieves the currently active editor and triggers the behaviour. The behaviour retrieves a client connection (or creates one if one shouldn't exist) and calls the server (groovy).


// Wiring up the behaviour in groovy.behaviors
:editor.groovy [:lt.plugins.groovy/on-eval
                     :lt.plugins.groovy/on-eval.one
                     :lt.plugins.groovy/on-clear-bindings
                     :lt.plugins.groovy/groovy-res
                     :lt.plugins.groovy/groovy-err
                     [:lt.object/add-tag :watchable]]

The final piece of the puzzle from the Light Table side is to attach the behavior to the :editor.groovy tag. This enables the behavior to be available from any editor that is tagged with this tag.

Clearing bindings - Groovy

// The command dispatch got a new command
case "editor.clear.groovy":
   clientSessions.clear(currentClientId)
   break;

The code above will just nuke any stored binding variables.

Conclusions

A tiny step that allows you to eval groovy expressions step by step. Anything that results in a binding is stored between evals. Obviously it's a bit limited in that you'll run into the familiar trap of trying to use def and be surprised(/annoyed) that it won't remember that or if you define a class it won't remember that either. It's probably possible to cater for some of these traps, but maybe not within the realms of a quick win. 

Anyways the end result is Version 0.0.3 !

Next steps

Firstly there is a Screencast brewing. After that I think a Light Table Gradle plugin is coming before the Groovy plugin gets any further updates. A pluggable Gradle plugin would enable the Groovy plugin to quite easily get the class path for you project. This would allow you to  explore your projects code in a REPL (-like) way. Exploratory testing FTW !




onsdag 7. mai 2014

Creating and Using Snippets in Light Table


An introduction on how to create and use simple and more elaborate snippets with the lt-snippets Light Table plugin. If you can't see the video above here's the link directly to youtube

søndag 23. februar 2014

A Groovy Light Table client - Step 2: Evaluating Code

Background

This is the second post in my series "A Groovy Light Table client". A blog series about steps I take when trying to build a Groovy plugin for Light Table

In this post I will take you through some of the steps I went through to get Light Table to evaluate groovy (script) code and show results inline in the editor.


How did we get here ?

Evaluate contents of editor (cmd/ctrl + shift + enter)

(behavior ::on-eval
          :desc "Groovy: Eval current editor"
          :triggers #{:eval}
          :reaction (fn [editor]
                      (object/raise groovy :eval! {:origin editor
                                                   :info (assoc (@editor :info)
                                                           :code (ed/->val editor)
                                                           :meta {:start 0, :end (ed/last-line editor)})})))

This behavior triggers on ":eval", which is triggered to any editor (on cmd/ctrl + shift + enter in default key mapping). We just get hold of the text from the editor and gather some meta info  and trigger a ":eval!" behavior on the groovy "mother" object defined in the last blog post.

Evaluate current line/selection

(behavior ::on-eval.one
          :desc "Groovy: Eval current selection"
          :triggers #{:eval.one}
          :reaction (fn [editor]
                      (let [pos (ed/->cursor editor)
                            info (conj (:info @editor) 
                                  (if (ed/selection? editor) 
                                    {:code (ed/selection editor) :meta {:start (-> (ed/->cursor editor "start") :line)
                                                                        :end (-> (ed/->cursor editor "end") :line)}}
                                    {:pos pos :code (ed/line editor (:line pos)) :meta {:start (:line pos) :end (:line pos)}}))]
                        (object/raise groovy :eval! {:origin editor :info info}))))

The only difference here is that we gather the code for the current line or current selection. Then we trigger the same behavior as for evaluating the whole editor.


Our groovy Eval!

(behavior ::eval!
          :triggers #{:eval!}
          :reaction (fn [this event]
                      (let [{:keys [info origin]} event
                            client (-> @origin :client :default)]
                        (notifos/working "Evaluating groovy...")
                        (clients/send (eval/get-client! {:command :editor.eval.groovy
                                                         :origin origin
                                                         :info info
                                                         :create try-connect})
                                      :editor.eval.groovy info 
                                      :only origin))))

This behavior is what actually sends off a eval request to the groovy client. Quite a lot happens under the hood (by help of inbuilt LightTable behaviors):

  • It tries to find a client (connection) for the editor
  • If no connection exists it will try to create a new one. On create it will invoke the try-connect function that we defined for the gui connect/connect bar behavior in the previous blog post
  • Once connected it will jsonify our parameters and send them off to our groovy client
The JSON might look something like:
[89,
"editor.eval.groovy",
  {"line-ending":"\n",
   "name":"sample.groovy",
   "type-name":"Groovy",
   "path":"/Users/mrundberget/Library/Application Support/LightTable/plugins/Groovy/sample.groovy",
   "mime":"text/x-groovy",
   "tags":["editor.groovy"],
   "code":"println \"hello\"",
   "meta":{"start":22,"end":22}}]


  • The first param is the client id for the editor that triggered the behavior. This client Id doesn't represent the same as a connection id (ref previous blog post). Many editors may share the same connection !
  • The second param is the command (our groovy client will of course support many different commands, this is one of them)
  • The third and last parameter is our info. The code is the essential bit, but some of the meta information, like line info comes in handy when handling the response later on

The actual groovy evaluation

Command dispatch


ltClient.withStreams { input, output ->
  try {
    input.eachLine { line ->
    def (currentClientId, command, data) = new JsonSlurper().parseText(line)
    switch (command) {
    case "client.close":
      stop()
      break
    case "editor.eval.groovy":
      evalGroovy(data, currentClientId)
      break
   default:
     log "Invalid command: $command"
  }
  // ...


    

We parse any lines received from Light Table and based on the command received invokes the appropriate handler. In this case evalGroovy.

Eval groovy


private void evalGroovy(data, currentClientId) {
  def evalResult = scriptExecutor.execute(data.code)

  def resultParams = [meta: data.meta]
  if (evalResult.out) {
    resultParams << [out: evalResult.out]
  }
  if(evalResult.exprValues) {
    resultParams << [result: convertToClientVals(evalResult.exprValues)]
  }

  if (!evalResult.err) {
    data = [currentClientId?.toInteger(), "groovy.res", resultParams]
  } else {
    data = [currentClientId?.toInteger(), "groovy.err", [ex: evalResult.err] + resultParams]
  }
  sendData data
}

The first and most significant line is where we evaluate the groovy code received. This post would be too long if we went into all the details of what it does, but here's a high-level summary:

  • We basically create a GroovyShell and compile our code to a script. Normally that would just compile a Script class. However we  wish to collect a lot more information than you typically would get from default groovy script execution. So we do an AST transformation on the script class and add a custom abstract script class as a base class for the compiled script class.  This allows us to inject behavior and wrap statement execution (all compiled into the script for optimal performance).  That way we are able to collect information about values for most types of statements. We collect line number and value (each line could end up having many values :-) )
  • We run the script (capturing system.out and system.err). 
  • The function returns:
    • Anything written to standard out (println etc)
    • Errors if any and line number for error where possible
    • A list for of maps with line number and value(s)  

Most of the AST stuff is not something I've written. It's been contributed by Jim White after I posted a question on the groovy-user mailing list. I asked for advice on which way to proceed and the response from the groovy community was awesome. Jim in particular was more than eager to contribute to the plugin. OpenSource rocks ! So when I say we, I sometimes mean we literally.

Anyways, based on the results of the script execution we notify Light Table to trigger either a ":groovy.res" behavior or a "groovy.err" behavior.

The json response for sendData for a successful execution might look something like:

[89,
 "groovy.res",
 {"meta":{"start":22,"end":23},"out":"hello\nmama\n","result":[{"line":1,"values":["null"]},{"line":2,"values":["null"]}]}]


Handling the evaluation results in Light Table


(defn notify-of-results [editor res]
  (doseq [ln (:result res)]
    (let [lineNo (+ (:line ln) (-> res :meta :start) -1)]
      (object/raise editor :editor.result (clojure.string/join " " (:values ln)) {:line lineNo :start-line lineNo}))))

(behavior ::groovy-res
          :triggers #{:groovy.res}
          :reaction (fn [editor res]
                      (notifos/done-working)
                      (when-let [o (:out res)] (.log js/console o))
                      (notify-of-results editor res)))

(defn notify-of-error [editor res]
  (let [lineNo (+ (-> res :ex :line) (-> res :meta :start) -1)]
    (object/raise editor :editor.exception (:ex res) {:line lineNo :start-line lineNo'})))

(behavior ::groovy-err
          :triggers #{:groovy.err}
          :reaction (fn [editor res]
                      (object/raise editor :groovy.res res)
                      (notify-of-error editor res)))

These are the behavior definitions that handles either successful or evaluation of scripts with errors. Basically we:
  • Print to the Light Table Console anything that was captured to system.out/system.err by our groovy evaluation
  • Show inline results for each line, multiple results for a line are space separated. For showing inline results we are using a predefined Light Table behavior (:editor.result)
  • If the behavior is to handle an error, we show evaluation results up until the script exception. In addition we display details (stack trace) for the exception at the line in the script it occurred

Wiring it all up

groovy.behaviors

{:+ {:app [(:lt.objs.plugins/load-js ["codemirror/groovy.js", "groovy_compiled.js"])]
     :clients []
     :editor.groovy [:lt.plugins.groovy/on-eval
                     :lt.plugins.groovy/on-eval.one
                     :lt.plugins.groovy/groovy-res
                     :lt.plugins.groovy/groovy-err
                     [:lt.object/add-tag :watchable]]
     :files [(:lt.objs.files/file-types
              [{:name "Groovy" :exts [:groovy] :mime "text/x-groovy" :tags [:editor.groovy]}])]
     :groovy.lang [:lt.plugins.groovy/eval!
                   :lt.plugins.groovy/connect]}}

The eval and results/err behaviors are defined for the editor tag. So they are only applicable for editors marked as groovy editors. Any editor open with a file name ending in .groovy will automatically be attached to a editor.groovy tag. (You can also set it manually cmd+space -> "Editor: Set current editor syntax").
The ":eval!" behavior is defined for the :groovy.lang tag. Its tied to our groovy mother object just like the connect behavior. These behaviors are totally groovy client specific, whilst the other behaviors are less so (although not exactly generic as they are now…)

Wrap up

A little bit of plumbing was needed to get this set up. But the hard parts was really coming up with the groovy AST transformation stuff. I guess by now you might have started getting an inkling that Light Table is fairly composable ? It really is super flexible. You don't like the behavior for handling inline results for the groovy plugin ? You could easily write your own and wire it up in your user.behaviors file in Light Table. It's wicked cool, actually it really is your editor !

Yesterday I released version 0.0.2 of the Groovy LightTable plugin. Its available through the Light Table plugin manager, or if you wish to play with the code or maybe feel like contributing feel free to fork the repo at : https://github.com/rundis/LightTable-Groovy. Pull requests are welcome.

So where to next ? I'd really like to try and create an InstaRepl editor for the plugin. A groovy script editor that evaluates code as you type. There's gotta be one or two challenges related to that. A quick win might be to provide groovy api documentation from inside Light Table. I'll let you know what happens in the next post.

Disclaimer: I might have misunderstood some nuances of LIght Table, but hopefully I'm roughly on track. If you see anything glaringly wrong, do let me know.

søndag 16. februar 2014

A Groovy Light Table client - Step 1: Connecting the client

Building a plugin i Light Table

This is the first post in (hopefully) a series of blog posts about the various steps I go through when trying to create a plugin for Light Table. I have decided to try to create a Groovy plugin. I chose Groovy to ensure there was at least one technology fairly familiar to me. I have just started using Light Table, I have no previous experience with ClojureScript and I have just recently started writing some Clojure beyond the basic tutorials.

The short term goal is for the plugin to provide inline results and maybe an instarepl of some sort for groovy scripts.

LightTable-Groovy is the name of my plugin project and you can find the latest source there. It might be a couple of steps ahead of the blog posts though ! 

Documentation

Light Table was made open source in january and documentation for plugin developers is a little sparse.  So to have something to go by I decided to use some of the other language plugins as inspiration:
I haven't worked with any of the above mentioned languages, but they did provide enough inspiration to deduce how a Light Table client might interact.

BTW. A quick starter just to get you up an running with a hello world plugin could be this screen cast by Mike Haney.


Connecting a client - Process overview

Before we dwelve into the code It's a good idea to have a high level understanding of what we are trying to achieve ! 

A couple of use cases that needs to be supported:
  • Evaluate current selection or current line of groovy code and present results (preferably inline)
  • Evaluate contents of current editor and present results
    • Provide as much info about the results of each statement as possible
    • (Maybe need to evaluate line/statement by statement)
  • For a future instarepl, any change in the editor will trigger an evaluation
It becomes evident that our plugin needs to provide some kind of process that reacts to events from light table. A default pattern for achieving this has been devised for Light Table and roughly equates to the following steps:
  1. A connect event is triggered from Light Table (you need to set up your plugin to trigger that event…). Typically the connect event can be invoked manually from the connect bar in light table, or it can be triggered implicetly when evaluating code. 
  2. You fire of a process - Using inbuilt support from Light Table you start a process either a shell script or whatever really. I created a shell script that sets some environment stuff and then basically kicks off a groovy script. Light table provides a tcp/ip port and a unique client id which you need to forward to the process.
  3. Create a tcp client: In your process you create a tcp client using the given port 
  4. Send ack message: Send a json message with client id and an event name (behavior) to Light Table (through the tcp connection!)
  5. Confirm handshake for process: In your process (i.e. not the tcp connection!) write "Connected" to standard out. ("Connected" is just what the other plugins use, you could use  anything you like as long as it matches the connect behaviors(handlers) you provide inside light table.)
  6. Listen for events: Now you are connected and given you have set up your behaviors in Light Table correctly, your new connection should be reported as connected and shown in the Light Table connect bar. Now you listen for events on your tcp client and provides appropriate responses back to Light Table accordingly. (Handling this is the subject of a future blog post)

Behaviors for connecting (from groovy.cljs): 


(defn run-groovy[{:keys [path name client] :as info}]
  (let [obj (object/create ::connecting-notifier info)
        client-id (clients/->id client)
        project-dir (files/parent path)]
    (object/merge! client {:port tcp/port
                           :proc obj})
    (notifos/working "Connecting..")
    (proc/exec {:command binary-path
                :args [tcp/port client-id project-dir]
                :cwd plugin-dir
                :env {"GROOVY_PATH" (files/join (files/parent path))}
                :obj obj})))

(defn check-groovy[obj]
  (assoc obj :groovy (or (::groovy-exe @groovy)
                         (.which shell "groovy"))))

(defn check-server[obj]
  (assoc obj :groovy-server (files/exists? server-path)))

(defn handle-no-groovy [client]
  (clients/rem! client)
  (notifos/done-working)
  (popup/popup! {:header "We couldn't find Groovy."
                 :body "In order to evaluate in Groovy files, Groovy must be installed and on your system PATH."
                 :buttons [{:label "Download Groovy"
                            :action (fn []
                                      (platform/open "http://gvmtool.net/"))}
                           {:label "ok"}]}))

(defn notify [obj]
  (let [{:keys [groovy path groovy-server client]} obj]
    (cond
     (or (not groovy) (empty? groovy)) (do (handle-no-groovy client))
     :else (run-groovy obj))
    obj))

(defn check-all [obj]
  (-> obj
      (check-groovy)
      (check-server)
      (notify)))

(defn try-connect [{:keys [info]}]
  (.log js/console (str "try connect" info))
  (let [path (:path info)
        client (clients/client! :groovy.client)]
    (check-all {:path path
                :client client})
    client))


(object/object* ::groovy-lang
                :tags #{:groovy.lang})


(def groovy (object/create ::groovy-lang))

(scl/add-connector {:name "Groovy"
                    :desc "Select a directory to serve as the root of your groovy project... then again it might not be relevant..."
                    :connect (fn []
                               (dialogs/dir groovy :connect))})
(behavior ::connect
                  :triggers #{:connect}
                  :reaction (fn [this path]
                              (try-connect {:info {:path path}})))



  • scl/add-connector: This statement adds a connect dialog to our groovy plugin. You select a root directory and upon selection the ::connect behavior is triggered
  • ::connect basically responds with invoking a method for connecting. This does some sanity checks and if all goes well ends up invoking  run-groovy.
  • run-groovy : Fires up our groovy (server) process
  • def groovy   is basically the "mother" object of our plugin. It helps us scope behaviors and commands

The server part (LTServer.groovy)

import groovy.json.*

params = [
  ltPort:   args[0].toInteger(),
  clientId: args[1].toInteger() // light table generated id for the client (connection)
]

logFile = new File("server.log")

def log(msg) {
  logFile << "${new Date().format('dd.MM.yyyy mm:hh:sss')} - $msg\n"
}

client = null
try {
  client = new Socket("127.0.0.1", params.ltPort)
} catch (Exception e) {
  log "Could not connect to port: ${params.ltPort}"
  throw e
}

def sendData(data) {
  client << new JsonBuilder(data).toString() + "\n"
}
// ack to Light Table
sendData (
  [
    name: "Groovy",
    "client-id": params.clientId,
    dir: new File("").absolutePath,
    commands: ["editor.eval.groovy"],
    type: "groovy"
  ]
)
println "Connected" // tells lighttable we're good

client.withStreams {input, output ->
  while(true) {
  // insert code to listen for events from light table and respond to those (eval code etc)
  }
}


Notification of successful conncetion


(behavior ::on-out
          :triggers #{:proc.out}
          :reaction (fn [this data]
                      (let [out (.toString data)]
                        (object/update! this [:buffer] str out)
                        (if (> (.indexOf out "Connected") -1)
                          (do
                            (notifos/done-working)
                            (object/merge! this {:connected true}))
                          (object/update! this [:buffer] str data)))))

(behavior ::on-error
          :triggers #{:proc.error}
          :reaction (fn [this data]
                      (let [out (.toString data)]
                        (when-not (> (.indexOf (:buffer @this) "Connected") -1)
                          (object/update! this [:buffer] str data)
                          ))
                      ))

(behavior ::on-exit
          :triggers #{:proc.exit}
          :reaction (fn [this data]
                      ;(object/update! this [:buffer] str data)
                      (when-not (:connected @this)
                        (notifos/done-working)
                        (popup/popup! {:header "We couldn't connect."
                                       :body [:span "Looks like there was an issue trying to connect
                                              to the project. Here's what we got:" [:pre (:buffer @this)]]
                                       :buttons [{:label "close"}]})
                        (clients/rem! (:client @this)))
                      (proc/kill-all (:procs @this))
                      (object/destroy! this)
                      ))

(object/object* ::connecting-notifier
                :triggers []
                :behaviors [::on-exit ::on-error ::on-out]
                :init (fn [this client]
                        (object/merge! this {:client client :buffer ""})
                        nil))



The above behaviors basically handles signaling success, error or connection exits for our groovy client. As you can see in ::on-out this is where we check standard out from the process for the string "Connected", to signal success.


Wiring up behaviors (behaviors.groovy)


{:+ {:app [(:lt.objs.plugins/load-js ["codemirror/groovy.js", "groovy_compiled.js"])]
     :clients []
     :editor.groovy []
     :files [(:lt.objs.files/file-types
              [{:name "Groovy" :exts [:groovy] :mime "text/x-groovy" :tags [:editor.groovy]}])]
     :groovy.lang [:lt.plugins.groovy/connect]}}

The important part in terms on the connection is the wiring of the connect behavior to ":groovy.lang". This is needed for groovy to appear as a connection item in the light table connect bar.

"codemirror/groovy.js" deserves a special mention. This is what provides syntax highlighting for our groovy files (defined in the :files vector). The syntax highlighting is provided by the groovy mode module from CodeMirror

Wrapping up

So what have we achieved. Well we now have a connection to Light Table from an external process that can listen and respond to events from Light Table. For the purposes of this blog post series, its a Groovy client that hopefully pretty soon will be able to evaluate groovy scripts and respond with evaluation results. We didn't pay much attention to it, but we also got syntax highlighting of our Groovy files complements of CodeMirror. 

It took a while to grok how the connection part worked. Once I did understand roughly what was needed I was a bit annoyed with myself for messing about so much. I'm hoping this post might help others to avoid some of the mistakes I stumbled into.

søndag 9. februar 2014

Getting my feet wet with Clojure using Light Table

Background

I've been meaning to get into Clojure for quite some time now. I have been reading books, played with 4clojure and even went to a Clojure workshop with Stuart Sierra. But I never really found time and inspiration to percolate a hobby project to try it out in anger… Until my company (Kodemaker) decided to start a project to rewrite our website in RYO style using Clojure. I asked if I could have a go at one of the features (a really tiny one).

The last time I decided to properly learn a language I tried to learn Emacs at the same time. That didn't end all that well, or rather somewhere along the road I lost sight of Emacs. I'm a IntelliJ user in my day job so I've tried both LaClojure and Cursive a little bit. A couple of my colleagues have mentioned Light Table in the past and after watching some presentations and tutorials I was intrigued. When Light Table went open source I was determined to give it a good go.


The first steps

I already had Leiningen installed (and its just a "brew install leiningen" away on my mac anyway). So installing Light Table was a breeze and when I started it and I tried a few commands, it quickly became apparent that I was quite a few keyboard shortcuts short. There are plugins for Emacs and Vim shortcuts, but I figured I'd assign my own. That freedom proved daunting as well, and I'm pretty sure my setup will change a lot before it stabilizes.

ctrl + space and type "user key" and select User keymap

{:+ {:app {
           "ctrl-shift-t" [:tabset.new]
           "ctrl-shift-left" [:tabs.prev]
           ;; etc
           }
     :editor {"alt-w" [:editor.watch.watch-selection]
              "alt-shift-w" [:editor.watch.unwatch]
              "ctrl-k" [:editor.kill-line]
              ;; etc
             }}}



The first map :app defines shortcuts that available across editors. Ie if you have focus in an inline browser window, these shortcuts will still work. The :editor map are shortcuts that requires an editor to have focus. It's really nice that there is code completion (and instant docs) for all available actions and saved changes are immediately applied. And yes you could do fancy stuff by doing multiple actions for one shortcut, its a clojure vector !
I have created a github project for my current settings to keep things in synch across machines.

The paredit shortcuts are worth a special mention. You'll quickly discover that keeping those pesky parenthesizes in order can prove to be a challenge. Light Table comes bundled with a plugin that provides some support to make your life easier. You might need to update the plugin through the plugin manager (ctrl + space in case you were wondering how to find it).

I assume that from now on that either you have a shortcut or use ctrl+space to find the appropriate actions…


Get my project up and running

Summarized it goes like this:
  1. Open folder and select project folder for your clojure project
  2. "Connect: Add connection"
  3. Select clojure and then your leiningen project.clj file
  4. Open a clj file (mac-hint: cmd+o). And evaluate it (mac-hint: cmd+shift+enter). Behold the inline results !
  5. Assuming you have a ring app, you might want to start your web server and be able to change things on the fly:
    1. Create a repl file (see sample below)
    2. Open an clojure instarepl 
    3. enter: (use '<the_name_space:for_the_repl_file>) 
    4. enter: (start-server)
    5. voila, you can now kill/clear the instarepl if you like
  6. Add a browser tab (if you have the space why not assign it to a separate tabset so you can have code and browser side by side !) 
  7. Start hacking your clojure and evaluate statement (mac-hint: cmd+enter) or evaluate the whole clojure file. Refresh your browser to see the changes (mac-hint: cmd+r) 
It is really neat once you get into it. Not quite like my old WebSphere days. If you aren't sure what a function actually returns ? Just fire up an instarepl and try it out ! Can't remember exactly how that clojure function works… instarepl. Curious about the documentation for a clojure function try out : "Docs: Toggle documentation at cursor". 


Don't have a (web) project ? Try opening a terminal and type "lein new compojure-app sample".


Sample repl file

(ns repl
  (:require [mycompany-com.web :refer [app]]
            [ring.adapter.jetty]))

(defn start-server []
  (defonce server (ring.adapter.jetty/run-jetty #'app {:port 3000 :join? false}))
  (.start server))

(defn stop-server []
  (.stop server))


Plugins

Light Table comes bundled with a few plugins, but there are already quite a few others out there that are definitely worth a look. I've already installed Claire which is a emacs inspired file browser/finder. There are ruby, python and haskell plugins out there and probably many more coming.

Wrap-up

It really didn't take that long to get up and running with Light Table. Fumbled around a bit before I grasped how to get live code reloading in my running web app from the IDE. The instarepl is really nice for a Clojure noob, and the inline watches (results) are neat as well. Learning Clojure will obviously depend on much more than the editor, but cool tools that help you along the way can't be wrong.

Light Table is supposed to be super extensible, so I'm planning to delve into that. I've already started playing around with a Groovy plugin for Light Table and it's great fun. But anyways that belongs to another blog post.