Optimize by pre-allocating slice

Allocations are, relatively speaking, expensive.

Appending new data to a slice requires re-allocating memory if the new size goes over current capacity of the slice.

Knowing that, if you know the final size of the slice, you can pre-allocate the required capacity and avoid re-allocations.

You don't need to know the exact size: knowing the upper bounds is just as good. Guesstimating is better than nothing.

Consider appending to an empty slice:

s := []byte("123456")
var d []byte
fmt.Printf("d: %p, len: %d, cap: %d\n", d, len(d), cap(d))
d = append(d, s...)
fmt.Printf("d: %p, len: %d, cap: %d\n", d, len(d), cap(d))
d = append(d, s...)
fmt.Printf("d: %p, len: %d, cap: %d\n", d, len(d), cap(d))
d = append(d, s...)
fmt.Printf("d: %p, len: %d, cap: %d\n", d, len(d), cap(d))

d: 0x0, len: 0, cap: 0
d: 0xc00007c018, len: 6, cap: 8
d: 0xc00007c040, len: 12, cap: 16
d: 0xc00007e040, len: 18, cap: 32

The results might vary depending on the the version of the compiler and runtime. In Go 1.12 appending 6 bytes to an empty slice 4 times required 32 allocation.

By printing the address of slice with %p we can see if it gets re-allocated.

As the cap shows, first allocation was for 8 bytes, it was then re-allocated to 16 bytes and finally to 32 bytes.

On each re-allocation, the content had to be copied, so we copied 6 + 6*2 + 6*3 bytes in total. To store 24 bytes, we wasted time copying 36 bytes.

Memory allocations and copying memory takes time.

Go runtime is smart in that it doubles the the capacity, anticipating that we might need more space in the future.

We can be smarter than that and pre-allocate the slice:

s := []byte("123456")
d := make([]byte, 0, len(s)*4)
fmt.Printf("d: %p, len: %d, cap: %d\n", d, len(d), cap(d))
d = append(d, s...)
fmt.Printf("d: %p, len: %d, cap: %d\n", d, len(d), cap(d))
d = append(d, s...)
fmt.Printf("d: %p, len: %d, cap: %d\n", d, len(d), cap(d))
d = append(d, s...)
d = append(d, s...)
fmt.Printf("d: %p, len: %d, cap: %d\n", d, len(d), cap(d))

d: 0xc0000140a0, len: 0, cap: 24
d: 0xc0000140a0, len: 6, cap: 24
d: 0xc0000140a0, len: 12, cap: 24
d: 0xc0000140a0, len: 24, cap: 24

Here we can see that we start with the right capacity from the beginning and at no time the runtime had to make additional allocation or memory copy.