Golang : Read large file with bufio.Scanner cause token too long error

There are couple of ways to read files with Golang and one of the common way to read in file is with bufio.NewScanner() function. For example :

 package main

 import (
  "bufio"
  "fmt"
  "os"
 )

 func main() {
  file, err := os.Open("somefile.dat")

  if err != nil {
 panic(err.Error())
  }

  defer file.Close()

  reader := bufio.NewReader(file)
  scanner := bufio.NewScanner(reader)

  scanner.Split(bufio.ScanBytes)

  for scanner.Scan() {
 fmt.Println(scanner.Bytes())
  }
 if err := scanner.Err(); err != nil {
 fmt.Println("error while reading :", err)
  }
 }

Most of the time, things should be fine and dandy. However, there will be cases when the program attempts to read in a very large file and it will throw out this error message :

"bufio.Scanner: token too long"

This error message was caused by the input token size that exceeded the bufio.MaxScanTokenSize buffer size. The allocated size is MaxScanTokenSize = 64 * 1024

To read in large files, it is recommended to use bufio.Reader.ReadLine instead.

References :

bufio.MaxTokenSize

Golang Bufio NewScanner example