Xtext

from Wikipedia, the free encyclopedia
Xtext

logo
Screenshot

Domain-specific language for the implementation of state machines. Source text editor on the left, Outline view on the right
Basic data

Maintainer Eclipse Foundation
developer Eclipse Foundation , itemis AG , TypeFox
Publishing year 2006
Current  version 2.21.0
( March 3, 2020 )
operating system Cross-platform / Java Virtual Machine / Eclipse , IntelliJ IDEA
programming language Java , Xtend
category Framework , language workbench
License Eclipse Public License
eclipse.org/Xtext

Xtext is an open-source - Framework for the development of programming languages and domain-specific languages ( English d omain s pecific l anguage , DSL ) and part of the Eclipse Modeling Framework -project (EMF). In contrast to normal parser generators , Xtext not only generates a parser , but also an EMF metamodel (a class model for the abstract syntax tree ) and a text editor integrated in Eclipse as well as the necessary infrastructure for the implementation of a modern development environment for the developed language.

history

The first version of Xtext was published in 2006 as part of the openArchitectureWare (oAW) project . In the time that followed, new functions and concepts were integrated with each version. Version 4.3 of oAW contains the last oAW version of Xtext. Since the beginning of 2008, Xtext has been further developed under the Eclipse Modeling Project in the Textual Modeling Framework (TMF) . Employees from itemis AG and TypeFox are developing a new version of the framework here. The first official release took place on June 16, 2009, when Xtext appeared in version 0.7.0.

In June 2010 the framework reached version 1.0. Version 2.0 was released on June 27, 2011 together with Eclipse 3.7 (Indigo).

In June 2012, Version 2.3 of the Expression Language Xbase was presented. On the one hand, Xbase enables a seamless integration of a DSL into the Java type system and, on the other hand, provides typical expressions in a Java-like syntax for integration into a DSL. Another innovation of version 2.3 is the programming language Xtend , a programming and template language developed in Xtext based on Java. It has since been promoted to implement many language and IDE concepts.

The next big innovation from Xtext was presented on December 1st, 2015 with version 2.9. Xtext is now also available as a plug-in for the IntelliJ IDEA development environment . Both the version for Eclipse and the version for IntelliJ can be used to develop plug-ins for both platforms. It is also possible to generate a DSL editor for a web application. The use of the build systems is now (partially) possible.

Functionality

(Domain-specific) programming languages ​​are developed with Xtext. It is much more than a simple parser generator such as B. ANTLR , which is still used in Xtext to read source texts and derive an abstract syntax tree (AST). It offers possibilities to analyze and validate the AST, to transfer the AST into any other (textual) representation or to derive Java classes from it. In addition to these typical compiler functions , Xtext offers possibilities to implement many IDE concepts for the language. At the end of the development process there can be a plug-in for Eclipse or IntellJ, which makes a complete IDE available with the help of these platforms.

The following briefly describes the various functions and concepts that can be implemented with Xtext.

Workflow

Xtext uses an MWE2 workflow to configure the language. If the workflow is executed, the parser and the metamodel are generated. Furthermore, class and method bodies are created in order to implement various compiler and IDE concepts. If the concepts are not implemented, the standard implementations of Xtext are used.

Compiler functions

This section summarizes concepts that are typically part of the compiler. A compiler parses the source text and creates an abstract syntax tree (AST, Abstract Syntax Tree). Before this happens, a syntax analysis is carried out to find any syntax errors. The AST is only generated if the source text is free of syntax errors. Static analyzes are often carried out on the AST (e.g. type check, overlay / shading, visibility, etc.). These analyzes can reveal errors. If the AST is error-free, it will be converted into the output language. This can be any (textual) representation or source text in any (Java) other programming language.

Parser / grammar of the language

Xtext automatically generates an ANTLR parser for the language. The ANTLR parser generator is used for this. For this concept only the grammar of the language (syntactic rules) has to be defined. Xtext provides its own language for this. The special thing about the grammar language of Xtext is that it describes not only the concrete, but also the abstract syntax. Usually not only the ANTLR parser is generated from the grammar, but also an Ecore metamodel and its implementation in Java. It is used as an AST. The generated ANTLR TreeParser creates an instance of this model and returns it as AST. If a metamodel already exists and only the textual syntax is to be defined, the metamodel can be imported in the Xtext grammar language. Then no own metamodel is created.

The syntax of the grammar language is described below. Each grammar has a header that defines the name of the language and the parent grammar. Furthermore metamodels of the father languages ​​can be imported in order to use their rules. If no existing metamodel is used, the Ecore metamodel is also defined here: 

grammar org.eclipse.mydsl.MyDsl with org.eclipse.xtext.xbase.Xbase

import "http://www.eclipse.org/emf/2002/Ecore" as ecore
import "http://www.eclipse.org/xtext/common/JavaVMTypes"
import "http://www.eclipse.org/xtext/xbase/Xtype"
generate MyDSL "http://www.eclipse.org/xtext/mydsl/MyDsl"

The top-level (root) rule of the language follows the header. All other rules follow the root rule in any, but reasonable, order. A rule has a name that starts with an uppercase letter followed by a colon. A rule ends with a semicolon. Characters that are in the defined language, such as B. commas, periods, but also keywords, are placed in single quotation marks. If a rule contains further (complex) rules or has certain attributes, the name of the attribute is written first (starting with a small letter), followed by an equal sign and then the name of the child rule. A typical rule could e.g. B. look like this: 

 AnyRule:
  'any' element=ID 'will' 'be' value=Value ';';

We don't know what the Value rule looks like, but the code described here could look something like this: 

  any MyName will be 42;

An attribute must always contain exactly one value, unless it is marked as optional or it contains a list of values. To mark a value as optional, a question mark is appended to it, a list of values ​​that can also be empty is marked with an asterisk and a list that must contain at least one element is marked with a plus. The assignment operator for list attributes is the plus equals: 

 SeveralAttributesRule:
  mustBeSet=AttributeMandartory
  canBeSet=AttributeOptional?
  possiblyEmptyList+=AttributeInList*
  listWithAtLeastOneElement+=AttributeInList+;

After the grammar of the language has been implemented or changed, the MWE2 workflow needs to be executed in order for the parser and metamodel to be (re) generated.

Static analysis

As soon as the MWE2 workflow is executed for the first time, Xtext creates the class org.eclipse.xtext.mydsl.validation.MyDslValidator. This is initially empty and any number of methods can be implemented that can test or analyze individual nodes of the AST. Xtext finds these methods and executes them automatically depending on the context. In order to inform Xtext that a certain method contains an analysis, it must on the one hand be marked with the annotation @Checkand on the other hand it may only have exactly one parameter whose type is a type of the meta model. Such an analysis method could look something like this: 

 @Check
 def checkRootElement(MyDslRoot root) {
   // …
 }

These methods have no return value (void). If errors are found, one of the error(…)methods should be called. This receives some information and creates an error marker.

Analogous to errors, warnings ( warning(…)) or information ( info(…)) can also be generated (for the editor, see IDE concepts) . These results are displayed in the source text by the editor.

Code generation

For each language, Xtext creates a class that can be used to translate the language into a specific output. From version 2.3 there are two options for this. If the language z. If, for example, a data structure is set up, an XML output or a JSON object can be generated, or any other (textual) representation. Often it also happens that a language covers a small part of a large program. Then it makes sense to create Java classes. The ModelInferrer is used for this. It derives a Java class and thus integrates the language seamlessly into the Java type system (great advantages at IDE level).

MyDslGenerator

By default, a class is org.eclipse.xtext.mydsl.generator.MyDslGeneratorcreated first. This contains exactly one method doGenerate(Resource r, IFileSystemAccess fsa). The generation process that calculates the output of the compiler can be implemented here. This method is carried out automatically when the file is saved in the Eclipse Editor. The generated files are usually src-genstored in the directory , unless something else is set. The output format is arbitrary here.

MyDslModelInferrer

When using Xbase (e.g. to use cross-referencing to Java-types) a ModelInferrer is generated instead of the generator class, which has to be implemented. This enables the derivation of Java types from the language. The file itself does not have to be generated here, but only a transformation of the language AST to a Java AST is implemented. The actual serialization and translation into bytecode is done by Xtext.

MyDslInterpreter

Xtext also offers the option of implementing an interpreter instead of a code generator so that the code of the language can be executed directly.

IDE concepts

(tbd) In addition to the compiler components, some of which also implement IDE concepts or work on the editor, various IDE concepts can be implemented for the Xtext language.

editor

The MWE2 workflow automatically creates an editor view with basic syntax highlighting.

Syntax highlighting

This can be adapted / refined both syntactically and semantically. There are no limits to colors, fonts and text decorations. With a corresponding effort, a page for the language settings can be implemented in which the end user can adapt the highlighting to their own needs (e.g. dark theme or red-green visual impairment).

Auto formatting

The feature of autoformatting source text known from Eclipse can also be implemented for an Xtext language. Xtext provides an API for this. The patterns defined here are applied to the source text when the corresponding command is given or when a source text file is generated by building up the AST piece by piece and then serializing it.

Scoping

The feature of hyperlinks to jump to field, method or class definitions known from Eclipse can also be implemented for Xtext languages. Xtext provides an API for this. In connection with the Expression language Xbase and the JvmModelInferrer, a seamless integration of the language into the Java type system can be created at the IDE level. On the one hand, a hyperlink to a method of a Java class can be implemented, on the other hand, it is also possible to use the derived Java types in Java classes. If the hyperlink is clicked in the Java editor, the Xtext language editor even opens the corresponding file and marks the referenced part.

Display of analysis results

As described in the section on compiler concepts, errors, warnings, and information can be found in the source code. These are automatically displayed in the Eclipse editor.

Code folding

Xtext editors automatically support the collapse of source text blocks.

Autocomplete and quick fixes

Xtext provides a class to support the Eclipse autocomplete feature. This is how you can implement what content is proposed. In combination with the displayed errors and warnings, suggestions for improvement can even be generated to a certain extent. These can then replace the area of ​​a document marked as defective with the suggestion in order to remove errors.

Outline view

The so-called Outline View is known from Eclipse. This shows z. B. to all fields and methods of a Java class. Such an outline view can also be implemented for an Xtext language. This is then linked to the open file so that the courser jumps to the corresponding point in the text with a click on an element in the outline.

Rename refactoring

Rename refactoring is supported by Xtext editors.

Further processing of DSL models

DSLs are formal languages ​​and therefore need to be made executable. This is done in two different ways:

  • For code generation itself offers Xtend of which is developed by the same team and has hosted also in Eclipse Modeling. Any target platform can be used for code generation.
  • The interpretation of EMF models must be done with a language compatible with the Java Virtual Machine , since EMF and Xtext have only been implemented in Java so far. The target platform is therefore restricted to these languages.

Award

  • Eclipse Community Awards 2010: “Most Innovative New Feature or Eclipse Project”

literature

Web links

Individual evidence

  1. ↑ www.eclipse.org .
  2. ^ Eclipse Modeling News. eclipse.org, accessed March 4, 2010 .
  3. ↑ Eclipse Indigo with many innovations for Java developers. heise.de, accessed on June 28, 2011 .
  4. ^ The Future of Xtext. blog.efftinge.de, accessed on January 12, 2016 .
  5. ^ The Future of Xtext. speakerdeck.com, accessed January 12, 2016 .
  6. ↑ Xtext 2.9.0 Release Review. projects.eclipse.org, accessed January 12, 2016 .
  7. ↑ eclipse.org