Getting Started

DTG is a project aimed at simplifying the development of domain-specific languages by generating parts of their implementation (e.g. AST) and tools for using the language in an editor like VSCode (e.g. LSP server and editor extensions with syntax highlighting) based on an ANTLR4 grammar and a DSL config file that allows further customization.

Outputs

• Syntax Highlighting – a .tmLanguage.json file which contains a TextMate grammar for syntax highlighting supported by many editors, including VSCode.

• AST – C# source code of AST node classes (as records) and an AstBuilder class, which is a Visitor implementation that transforms parse trees (output by ANTLR-generated parsers) into ASTs ready for further analysis, validation and transformations.

• Language Server – a C# class that implements a basic LSP (Language Server Protocol) server for the language, which can be used to provide features like live diagnostics, code completion, hover information, etc.

• VSCode Extension – a ready-to-run VSCode extension that uses the generated TextMate grammar for syntax highlighting and provides integration with the language server.

  • includes an AST Explorer tree view (optional)

Install

DTG is available as a dotnet tool. You can install it using dotnet tool install --global DSLToolsGen and then run it as dtg.

Alternatively, run it directly without installation using dnx DSLToolsGen.

Usage

• dtg init to create a dtg.json config file
• dtg generate to run the configured generators once
  • or dtg watch, which automatically reruns the configured generators once their inputs change

Generate DSL tools using DTG

Detailed steps for adding DTG-generated tools into a C# project:

1. Create or open a C# project and add these NuGet packages as dependencies:

   • ANTLR runtime library: Antlr4.Runtime.Standard
   • OmniSharp LSP library: OmniSharp.Extensions.LanguageServer

   (so for example: dotnet new console --name AbcLS && dotnet add package Antlr4.Runtime.Standard && dotnet add package OmniSharp.Extensions.LanguageServer)

2. Run dtg init and then edit the generated dtg.json, primarily the grammar file name, VSCode extension ID, and output paths. See Configuration

3. Run dtg generate or dtg watch (to keep the generator running and regenerating whenever the inputs are modified)

4. Generate the VS Code extension using dtg generate vscodeExtension (it is not generated automatically by default because it’s usually enough to generate it once and then only regenerate the TextMate grammar for syntax highlighting; it’s also a little risky since there’s no clear separation of user-written vs autogenerated code)

   Caution

   In the current implementation, the VSCode extension generator writes to (and overwrites) the tasks.json and launch.json files in a .vscode folder outside the configured VSCode output directory (i.e. in the main project directory).

5. Add code to the top of Program.cs that runs the generated LanguageServer if the --ls argument is provided, for example:

   if (args.Contains("--ls"))
   {
       var connection = LspConnectionInfo.FromCommandLineArgs(args);
       bool loop = connection is LspConnectionInfo.TcpServer;
       do
       {
           await new XyzLanguageServer().RunAsync(connection);
           Console.Error.WriteLine("Language server stopped.");
       }
       while (loop);
   }

   Note

   If you want to use a different argument than --ls, make sure to also update languageServerArgs in activate() in the generated extension.ts (in the VSCode extension).

6. Execute cd vscode-extension && npm install

7. Launch the language server using dotnet watch -- --ls --tcpserver=42882 (or through Visual Studio; you can add a launch profile with the arguments)

8. Now you can edit code while running (using hot-reload) or restart the LSP server – the LSP client should try to reconnect automatically (or reconnect manually via the Xyz: Restart Language Server command)

9. Add a LanguageServer.cs file with the following contents (replace Xyz with the name of your language):

   using Microsoft.Extensions.DependencyInjection;
   using LSP = OmniSharp.Extensions.LanguageServer;
   using OmniSharp.Extensions.LanguageServer.Protocol.Models;
   using Xyz.Parser;
   using Xyz.AST;
   
   namespace Xyz.LanguageServer;
   
   partial class XyzLanguageServer
   {
       public async Task RunAsync(LspConnectionInfo connection)
       {
           DocumentManager documents = new();
   
           var server = await LSP.Server.LanguageServer.From(options => options
               .WithConnection(connection)
               .WithServices(s => s.AddSingleton(documents))
               .WithTextDocumentSyncHandler(HandleDocumentUpdate)
               .WithHandler<BasicHoverHandler>()
               .WithHandler<BasicCodeCompletionHandler>()
           );
   
           await server.WaitForExit.ConfigureAwait(false);
       }
   
       protected override async Task<Document> AnalyzeDocument(
           TextDocumentIdentifier documentId, string documentText,
           XyzParser parser, IList<Diagnostic> diagnostics)
       {
           // Convert parse tree to AST
           // TODO: replace `xyzFile` with the actual rule name from your grammar
           var ast = new AstBuilder().VisitXyzFile(parser.xyzFile());
   
           // Here you can perform any additional semantic analysis,
           // add errors and warnings to `diagnostics`,
           // attach data to AST nodes (use `partial` class declarations
           // to add new properties to the AST node classes), etc.
   
           return new Document(documentText, ast, parser);
       }
   }

10. Implement any custom Language Server functionality:

    • live diagnostics (error checking, warnings with automatic code fix suggestions, etc.)
      • lexical and syntax errors reported by ANTLR during parsing are automatically published as LSP diagnostics
      • publish custom diagnostics (semantic errors, warnings, …) from inside your AnalyzeDocument override
    • code completion
      • you can use the generated BasicCodeCompletionHandler base class
    • semantic highlighting
      • you can use the generated BasicSemanticTokensHandler base class
    • hover information
      • you can use the generated BasicHoverHandler base class
    • document outline
    • …

    Don’t forget to register custom handlers using .WithHandler<T>() in RunAsync.

Troubleshooting

If the syntax highlighting looks outdated after making a change to the grammar, you will need to restart the extension host instance of VSCode so that it loads the newly generated .tmLanguage.json file (make sure dtg watch or dtg generate printed Generating TextMate grammar… to the console).

It it still looks wrong, open the TextMate Scope Inspector in VSCode (Ctrl+Shift+P, Developer: Inspect Editor Tokens and Scopes) and look at what scope is being assigned to the token; the default scope names include the original lexer name at the end… there might be a rule conflict (shadowing) → add it into the configuration