Golang : How to tokenize source code with text/scanner package?



Tags : golang tokenizer scanner EOF

It has been a while since I touch the topic on tokenizer.....err... since the day I actually built a compiler back in the 90's in Ada95!

Ok, time to cut the grandma or grandpa story.

This short code example below demonstrate how to tokenize souce code with Golang's text/scanner package.

Here you go !

 package main

 import (
  "fmt"
  "strings"
  "text/scanner"
 )

 func main() {

  code := `<?php
 echo "Hello World!";
 ?>`

  codeReader := strings.NewReader(code)

  fmt.Println(code) 

 fmt.Println("------------------------------")
 fmt.Println("TOKENS : ")
 fmt.Println("------------------------------")

  var scn scanner.Scanner
  scn.Init(codeReader)

  tok := scn.Scan()
  fmt.Println(scn.TokenText())
  for tok != scanner.EOF {
 tok = scn.Scan()
 fmt.Println(scn.TokenText())
  }
 }

Play at : http://play.golang.org/p/R9pDAt2jOO

Affiliate :

Hire Golang Developers




Tags : golang tokenizer scanner EOF

By Adam Ng

IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.


Advertisement