Summary
In high-performance applications, Input/Output (I/O) operations are often the silent performance killers. A common mistake in Go applications is reading or writing data in small chunks—sometimes as small as a single byte—directly to a file or network connection. This approach triggers a massive number of system calls, forcing the CPU to constantly switch contexts between user space and kernel space.
The solution is Buffered I/O. By wrapping your readers and writers with the bufio package, you introduce a memory buffer (typically 4KB) that acts as an intermediary. The application interacts with this fast memory buffer, and the actual expensive disk or network I/O only happens when the buffer is full (for writes) or empty (for reads). This simple architectural change can improve throughput by orders of magnitude.
The Hidden Cost: System Calls
To understand why unbuffered I/O is slow, you need to look at what happens under the hood. When your Go program reads from a file, it can't access the hardware directly. It must ask the operating system kernel to do it. This request is a system call.
A system call involves a context switch. The CPU saves the state of your program, switches to kernel mode, performs the operation (checking permissions, talking to drivers), and then switches back to user mode to resume your program. While modern CPUs are fast, doing this millions of times per second for individual bytes creates significant overhead. It's like driving to the grocery store every time you need a single egg, instead of buying a dozen at once.
Insight: The primary bottleneck in disk I/O isn't always the disk speed itself—it's often the latency introduced by these context switches. Minimizing the boundary crossings between user space and kernel space is a fundamental optimization strategy in systems programming.
Efficient Reading with bufio.Scanner
For reading text data, bufio.Scanner is the standard tool. It handles buffering automatically and provides a convenient API for reading data line-by-line or word-by-word without the overhead of unbuffered reads.
// Inefficient: Reading byte by byte (Don't do this)
// for {
// b := make([]byte, 1)
// _, err := file.Read(b) // System call every time!
// }
// Efficient: Using bufio.Scanner
file, err := os.Open("large_log_file.txt")
if err != nil {
log.Fatal(err)
}
defer file.Close()
// Wraps the file reader in a buffer
scanner := bufio.NewScanner(file)
// Scan() reads the next token (default is line by line)
// It reads large chunks from disk into memory, then serves lines from memory
for scanner.Scan() {
line := scanner.Text()
// Process line...
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
Efficient Writing with bufio.Writer
Writing data is where buffering often shows the most dramatic improvements. Without buffering, every Write() call hits the disk. With bufio.Writer, writes go into a memory buffer. The actual system call to write to disk only happens when that buffer fills up.
There is one critical catch: You must flush the buffer. If your program finishes while there is still data sitting in the buffer that hasn't filled it up yet, that data will be lost unless you explicitly call Flush().
file, err := os.Create("output.txt")
if err != nil {
log.Fatal(err)
}
defer file.Close()
// Create a buffered writer (default 4KB buffer)
writer := bufio.NewWriter(file)
for i := 0; i < 1000; i++ {
// These writes go to memory, not disk
fmt.Fprintln(writer, "Writing a log line to the buffer...")
}
// CRITICAL: Ensure any remaining data in buffer is written to disk
if err := writer.Flush(); err != nil {
log.Fatal(err)
}
When Not To Use Buffering
While buffered I/O is excellent for throughput, it introduces latency. If you are writing a real-time application where every piece of data needs to be persisted immediately (like a database transaction log or a heartbeat signal), buffering might delay that data. In those specific cases, unbuffered I/O or manually flushing after every critical write is the correct choice.