Golang : How to split or chunking a file to smaller pieces?
Problem :
You have a big file and you want to split / chunk the file into smaller pieces. How to do that in Golang?
Solution :
- Calculate the size of each chunk that you want to split the file into.
- Read the file content into a buffer(created according to the calculated chunk size).
- Write the buffer content into individual pieces.
Here you go :
package main
import (
"fmt"
"io/ioutil"
"math"
"os"
"strconv"
)
func main() {
fileToBeChunked := "./somebigfile"
file, err := os.Open(fileToBeChunked)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer file.Close()
fileInfo, _ := file.Stat()
var fileSize int64 = fileInfo.Size()
const fileChunk = 1 * (1 << 20) // 1 MB, change this to your requirement
// calculate total number of parts the file will be chunked into
totalPartsNum := uint64(math.Ceil(float64(fileSize) / float64(fileChunk)))
fmt.Printf("Splitting to %d pieces.\n", totalPartsNum)
for i := uint64(0); i < totalPartsNum; i++ {
partSize := int(math.Min(fileChunk, float64(fileSize-int64(i*fileChunk))))
partBuffer := make([]byte, partSize)
file.Read(partBuffer)
// write to disk
fileName := "somebigfile_" + strconv.FormatUint(i, 10)
_, err := os.Create(fileName)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// write/save buffer to disk
ioutil.WriteFile(fileName, partBuffer, os.ModeAppend)
fmt.Println("Split to : ", fileName)
}
}
Sample output :
Splitting to 5 pieces.
Split to : somebigfile_1
Split to : somebigfile_2
Split to : somebigfile_3
Split to : somebigfile_4
References :
https://www.socketloop.com/tutorials/golang-convert-cast-int64-to-string
See also : Golang : Upload big file (larger than 100MB) to AWS S3 with multipart upload
By Adam Ng
IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.
Advertisement
Tutorials
+72k Golang : How to return HTTP status code?
+4.8k Golang : Launching your executable inside a console under Linux
+4.7k Unix/Linux : Get reboot history or check when was the last reboot date
+15.4k Golang : Merge video(OpenCV) and audio(PortAudio) into a mp4 file
+3.9k MariaDB/MySQL : Form select statement or search query with Chinese characters
+6k Golang : Get environment variable
+7.9k Golang : Go as a script or running go with shebang/hashbang style
+12.3k Golang : How to get year, month and day?
+5.7k Mac/Linux/Windows : Get CPU information from command line
+11k Golang : How to parse plain email text and process email header?
+17k Golang : How to get hour, minute, second from time?
+10k Golang : Find age or leap age from date of birth example