Golang : How to tokenize source code with text/scanner package?

It has been a while since I touch the topic on tokenizer.....err... since the day I actually built a compiler back in the 90's in Ada95!

Ok, time to cut the grandma or grandpa story.

This short code example below demonstrate how to tokenize souce code with Golang's text/scanner package.

Here you go !

 package main

 import (
  "fmt"
  "strings"
  "text/scanner"
 )

 func main() {

  code := `<?php
 echo "Hello World!";
 ?>`

  codeReader := strings.NewReader(code)

  fmt.Println(code) 

 fmt.Println("------------------------------")
 fmt.Println("TOKENS : ")
 fmt.Println("------------------------------")

  var scn scanner.Scanner
  scn.Init(codeReader)

  tok := scn.Scan()
  fmt.Println(scn.TokenText())
  for tok != scanner.EOF {
 tok = scn.Scan()
 fmt.Println(scn.TokenText())
  }
 }

Play at : http://play.golang.org/p/R9pDAt2jOO