Golang : How to handle file size larger than available memory panic issue
Problem:
You are trying to convert create a slice with a size of a very large file. The file size is larger than available memory and it caused your Golang program to panic during runtime. How to overcome this panic issue?
Solution:
It was once thought that 640KB of memory should be enough for all our computing needs, but that is no longer true. However, hitting a memory limit issue will always be there. Hence the solution is:
Get the available memory first before creating the slice.
If the file size is lower than the available memory than proceed as usual. However, if the file size is larger than available memory, chunk the file according to a sensible size and process each chunks without bursting the total available memory.
package main
import (
"fmt"
"net/http"
"os"
"github.com/shirou/gopsutil/mem"
)
func main() {
// memory
vmStat, err := mem.VirtualMemory()
if err != nil {
panic(err)
}
availableMemory := vmStat.Available
//availableMemory := uint64(655360) // uncomment to test low memory situation
fmt.Println("Available memory : ", availableMemory, " bytes")
fileToBeUploaded := "somebigfile"
file, err := os.Open(fileToBeUploaded)
if err != nil {
panic(err)
}
defer file.Close()
fileInfo, _ := file.Stat()
fileSize := uint64(fileInfo.Size())
fmt.Println("File size : ", fileSize, " bytes")
// previous tutorials did not check if the fileSize is larger than
// available memory. However, bear in mind that
// http.DetectContentType() only needs 512 bytes, not the entire file
if fileSize < availableMemory {
fileBytes := make([]byte, fileSize)
fmt.Println("File converted into []byte")
fileType := http.DetectContentType(fileBytes)
fmt.Println("File type : ", fileType)
} else {
// see https://golang.org/pkg/net/http/#DetectContentType
fmt.Println("Not enough memory to read the entire file into []byte!")
fileBytes := make([]byte, 512) // we only need 512 bytes
fileType := http.DetectContentType(fileBytes)
fmt.Println("File type : ", fileType)
}
// However, if http.DetectContentType() is not your concern then you will need to
// split the file to smaller pieces within the available memory and process it
// without causing panic
// such as
// https://www.socketloop.com/tutorials/golang-how-to-split-or-chunking-a-file-to-smaller-pieces
// https://www.socketloop.com/tutorials/golang-upload-big-file-larger-than-100mb-to-aws-s3-with-multipart-upload
// calculate the number of parts by dividing up the file size by 5MB
const fileChunk = 5242880 // 5MB in bytes
// or
const fileChunk = availableMemory - 10000 // not wise to use up all availableMemory, tweak around figure to minus!
}
See also : Golang : Get hardware information such as disk, memory and CPU usage
By Adam Ng(黃武俊)
IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.
Advertisement
Tutorials
+6.1k Golang : Extract sub-strings
+6.1k Golang : Test input string for unicode example
+31.2k Golang : Example for ECDSA(Elliptic Curve Digital Signature Algorithm) package functions
+15k nginx: [emerg] unknown directive "ssl"
+5.8k Golang : Build new URL for named or registered route with Gorilla webtoolkit example
+12.1k Golang : Extract part of string with regular expression
+18.7k Golang : When to use public and private identifier(variable) and how to make the identifier public or private?
+19.9k Golang : How to get struct tag and use field name to retrieve data?
+7k Golang : Null and nil value
+28.1k Golang : Change a file last modified date and time
+13.1k Golang : Verify token from Google Authenticator App
+29k Golang : Save map/struct to JSON or XML file