Metadata

Number:

GEP-2

Title:

AST Builder Support

Version:

7

Type:

Feature

Target:

1.7

Status:

Draft

Leader:

Hamlet D'Arcy

Created:

2009-04-01

Last modification:

2009-06-17

Abstract

Groovy 1.6 introduced the ability to perform local and global AST (Abstract Syntax Tree) transformations, allowing users to read and modify the AST of Groovy code as it is being compiled. Reading information in the AST is relatively easy in Groovy. The core library provides a strongly typed visitor called GroovyCodeVisitor. Nodes can be read and modified using the API provided through the subtypes of ASTNode. Writing new AST nodes is not as simple. The AST generated from source is not always obvious, and using constructor calls to generate trees of nodes can be verbose. This GEP proposes an ASTBuilder object that allows users to easily create AST.

The ASTBuilder object allows AST to be created from Strings containing Groovy source code, from a closure containing Groovy source, and from a closure containing an AST creation DSL. All three approaches share the same API: a builder object is instantiated and a build* method is invoked.

Approach

ASTNode from String

The simplest approach to implement is to provide an AST Builder with an API that takes a String and returns List<ASTNode>.

def builder = new AstBuilder()
List<ASTNode> statements = builder.buildFromString(
     CompilePhase.CONVERSION,
     true,
     """ println "Hello World" """
)

The above example produces the following AST:

 BlockStatement
 -> ExpressionStatement
     -> MethodCallExpression
        -> VariableExpression
        -> ConstantExpression
        -> ArgumentListExpression
           -> ConstantExpression

 Alternatives

def astTemplate = builder.buildAst ( "println $txt" ).head()

def constant = builder.buildAst ( "To be, or not to be: that is the question" ).head()

def methodCallExpression = astTemplate.apply(txt: constant)
// method call expression not contains println "To be ... "

     This templating approach adds complexity that may not be used. It overloads the GString $ operator, in that it is used here only with objects of type ASTNode but is used normally in GStrings with any Object type at all. Also, the templating approach can create order of precedence confusion. Consider source = "$expr * y", and later $expr is bound to "x+a". The result is "x + a * y", which was probably unintentional. At this time, the AST builder does not include such a feature.

ASTNode from Code Block

A useful API would be creating AST from code blocks.

AstBuilder builder = new AstBuilder()
def statementBlock = builder.buildFromCode (CompilePhase.CONVERSION, true) {
     println "Hello World"
}

The above example produces the following AST:

BlockStatement
 -> ExpressionStatement
    -> MethodCallExpression
       -> VariableExpression
       -> ConstantExpression
       -> ArgumentListExpression
          -> ConstantExpression

 Alternatives

@AstSource(CompilePhase.CONVERSION)
List<ASTNode> source = { println "compiled on: ${new Date()}" }

     This option seems helpful; however, annotations on local variables are not yet supported. This approach will not be implemented.

ASTNode from psuedo-specification

Building AST conditionally, such as inserting an if-statement or looping, is not easily accomplished in the String or code based builders. Consider this example:

def builder = new AstBuilder()
List<ASTNode> statements = builder.buildFromSpec {
    methodCall {
        variable "this"
        constant "println"
        argumentList {
            if (locale == "US") constant "Hello"
            if (locale == "FR") constant "Bonjour"
            else constant "Ni hao"
        }
    }
}

This library class is useful for several reasons:

Issues

Alternatives

Template Haskell and Boo provide a special syntax for AST building statements. Quasi-quote (or Oxford quotes) can be used to trigger an AST building operation:

 ConstantExpression exp = [| "Hello World" |]

 Those languages also supply a splice operator $(...) to turn AST back into code. This is not part of the AstBuilder work.

References

Mailing-list discussions

JIRA issues

Useful links

Reference Implementation

http://code.assembla.com/AstBuilderPrototype (very in-progress)

Test Case for AstBuilder from Code: http://subversion.assembla.com/svn/AstBuilderPrototype/src/test/org/codehaus/groovy/ast/builder/AstFactoryFromCodeTest.groovy

Test Case For AstBuilder from String: http://subversion.assembla.com/svn/AstBuilderPrototype/src/test/org/codehaus/groovy/ast/builder/AstFactoryFromStringTest.groovy

Test Case For AstBuilder from Specification: http://subversion.assembla.com/svn/AstBuilderPrototype/src/test/org/codehaus/groovy/ast/builder/AstBuilderFromSpecificationTest.groovyhttp://subversion.assembla.com/svn/AstBuilderPrototype/tests/org/codehaus/groovy/ast/builder/AstBuilderFromSpecificationTest.groovy