Golang : How to tokenize source code with text/scanner package?
It has been a while since I touch the topic on tokenizer.....err... since the day I actually built a compiler back in the 90's in Ada95!
Ok, time to cut the grandma or grandpa story.
This short code example below demonstrate how to tokenize souce code with Golang's text/scanner
package.
Here you go !
package main
import (
"fmt"
"strings"
"text/scanner"
)
func main() {
code := `<?php
echo "Hello World!";
?>`
codeReader := strings.NewReader(code)
fmt.Println(code)
fmt.Println("------------------------------")
fmt.Println("TOKENS : ")
fmt.Println("------------------------------")
var scn scanner.Scanner
scn.Init(codeReader)
tok := scn.Scan()
fmt.Println(scn.TokenText())
for tok != scanner.EOF {
tok = scn.Scan()
fmt.Println(scn.TokenText())
}
}
Play at : http://play.golang.org/p/R9pDAt2jOO
By Adam Ng
IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.
Advertisement
Tutorials
+15.1k Golang : How to extract links from web page ?
+5.7k Golang : Skip or discard items of non-interest when iterating example
+7.2k Golang : Auto-generate reply email with text/template package
+14.3k Golang : Get current time from the Internet time server(ntp) example
+11.4k Golang : Get absolute path to binary for os.Exec function with exec.LookPath
+20.2k Golang : Convert(cast) string to rune and back to string example
+10.4k Golang : How to flush a channel before the end of program?
+8.4k Golang : Qt Yes No and Quit message box example
+41.1k Golang : Get hardware information such as disk, memory and CPU usage
+17.7k Golang : Populate dropdown with html/template example
+9.3k Golang : How to get quoted string into another string?
+8.9k Golang : Test a slice of integers for odd and even numbers