Common mistakes when using golang’s sync.WaitGroup

This can be your WaitGroup crash course. Lets start step by step.

Read through the end or else you might end up learning something wrong and blame me for it 😝

0. Example Worker

Below is a simple example with a worker function to print a message. No concurrency 🍰.

// https://go.dev/play/p/vWL9zhQC1LT
package main

import (
 "fmt"
)

func worker(msg string) {
 fmt.Println("Worker", msg)
}

func main() {
 worker("w1")
 fmt.Println("exiting")
}

Output:

Worker w1
exiting

1. More workers 🐜🐜🐜

// https://go.dev/play/p/qE-UT_NM5Q3
package main

import (
 "fmt"
)

func worker(msg string) {
 fmt.Println("Worker", msg)
}

func main() {
 worker("w1")
 worker("w2")
 worker("w3")
 fmt.Println("exiting")
}

Output:

Worker w1
Worker w2
Worker w3
exiting

2. Sprinkle some concurrency 🧂

// https://go.dev/play/p/qE-UT_NM5Q3
package main

import (
 "fmt"
)

func worker(msg string) {
 fmt.Println("Worker", msg)
}

func main() {
 go worker("w1")
 go worker("w2")
 go worker("w3")
 fmt.Println("exiting")
}

Output:

Worker w1
Worker w2
Worker w3
exiting

or maybe

Worker w1
exiting

or

Worker w1
Worker w2
exiting

or even

Worker w2
Worker w1
exiting

Honestly we don’t know 🤷🏻‍♂️

3. Fix with a WaitGroup 🔧

To ensure we wait for all workers to complete. WaitGroup has a simple API.

• Add(int) adds delta, which may be negative, to the WaitGroup counter.
• Done() decrements the WaitGroup counter by one.
• Wait() blocks until the WaitGroup counter is zero.

// https://go.dev/play/p/r-4JT1upJQf
package main

import (
 "fmt"
 "sync"
)

var wg sync.WaitGroup

func worker(msg string) {
 wg.Add(1)
 defer wg.Done()
 fmt.Println("Worker", msg)
}

func main() {
 go worker("w1")
 go worker("w2")
 go worker("w3")

 fmt.Println("waiting")
 wg.Wait()
 fmt.Println("exiting")
}

Output:

waiting
Worker w1
Worker w2
Worker w3
exiting

or

Worker w1
waiting
Worker w2
Worker w3
exiting

or

Worker w2
waiting
Worker w1
Worker w3
exiting

or something else. But we have a guarantee. Guarantee that exiting is always printed after all lines starting with Worker have been printed.

4. Passing WaitGroup as argument

Often.. or mostly we don’t create global variables. We create function scoped variables. Lets see how our code changes when we pass WaitGroup to a function.

// https://go.dev/play/p/QoCJIbPkXkC
package main

import (
 "fmt"
 "sync"
)

func worker(msg string, wg sync.WaitGroup) {
 wg.Add(1)
 defer wg.Done()
 fmt.Println("Worker", msg)
}

func main() {
 var wg sync.WaitGroup
 go worker("w1", wg)
 go worker("w2", wg)
 go worker("w3", wg)

 fmt.Println("waiting")
 wg.Wait()
 fmt.Println("exiting")
}

Notice the wg declaration moving inside main and our worker now accepts the WaitGroup as second argument.

Output:

waiting
exiting

Not quite expected right !!

Reason: If you carefully notice the definition of WaitGroup methods carefully, you will see notice a pointer receiver. And WaitGroup is not an interface with an implementation using pointer receiver💡.

This means, if we pass the WaitGroup without a pointer, a copy gets created.

Also notice

5. Lets fix it 💪🏻

// https://go.dev/play/p/QoCJIbPkXkC
package main

import (
 "fmt"
 "sync"
)

// Notice the * here
func worker(msg string, wg *sync.WaitGroup) {
 wg.Add(1)
 defer wg.Done()
 fmt.Println("Worker", msg)
}

func main() {
 var wg sync.WaitGroup
 go worker("w1", &wg)
 go worker("w2", &wg)
 go worker("w3", &wg)

 fmt.Println("waiting")
 wg.Wait()
 fmt.Println("exiting")
}

Get Ready 🎇🎉🎆🎉🎇

Output:

waiting
Worker w1
Worker w2
Worker w3
exiting

Do you really get above output ?? Eh.. fooled by concurrency.

Run it enough times and you will get this

waiting
exiting

Or something else weird.

But wait why.. what’s the problem 😵

6. Enlightenment 🧠

Problem is not with the Waitgroup or the reference. Problem is in the ordering.

Notice the two statements wg.Add(1)and defer wg.Done() are written inside the go routine and not in the caller. So it is possible that the caller starts the go routine.. infact all go routines but none of them starts execution till we reach the line wg.Wait() . All go routines are waiting to execute their first statement wg.Add(1) while the main thread continues till end.

Wait !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

How the hell did our third example “2. Add some concurrency 🧂” work then 🙄

To answer that yourself, run it a couple times till you the magic happen yourself. And our third example is logically incorrect and doesn’t gurantee execution of all workers.

What’s the fix ?
It’s a common construct you’ll find in go code. Which looks something like this

// https://go.dev/play/p/QoCJIbPkXkC
package main

import (
 "fmt"
 "sync"
)

func worker(msg string) {
 fmt.Println("Worker", msg)
}

func main() {
 var wg sync.WaitGroup

 wg.Add(1)
 go func() {
  defer wg.Done()
  worker("w1")
 }()

 wg.Add(1)
 go func() {
  defer wg.Done()
  worker("w2")
 }()

 wg.Add(1)
 go func() {
  defer wg.Done()
  worker("w3")
 }()

 fmt.Println("waiting")
 wg.Wait()
 fmt.Println("exiting")
}

Get Ready 🎇🎉🎆🎉🎇

Output:

waiting
Worker w1
Worker w2
Worker w3
exiting

Notice the theme

 wg.Add(1)
 go func() {
  defer wg.Done()
  worker("w1")
 }()

wg.Add(1) appears before entering go routine

defer wg.Done() is the first statement in the go routine.

These 2 things ensure proper termination of our concurrent workers.

Also notice a couple more important things:

1. we no longer need to pass WaitGroup to the Worker
2. the caller has ownership of wg and takes care of cleanup
3. worker only contains core logic and doesn’t have to deal with any concurrency constructs

Phew 😅
That’s most of it.

7. Can we do better ?

There can be just different ways of expressing above code using loops. That’s how we see it in practice as well. Below is an example using a loop to spawn three workers.

// https://go.dev/play/p/0UiybqXphC0
package main

import (
 "fmt"
 "sync"
)

func worker(msg string) {
 fmt.Println("Worker", msg)
}

func main() {
 var wg sync.WaitGroup

 for i := 1; i <= 3; i++ {

  wg.Add(1)
  go func(seq int) {
   defer wg.Done()
   worker(fmt.Sprintf("w%d", seq))
  }(i)

 }

 fmt.Println("waiting")
 wg.Wait()
 fmt.Println("exiting")
}

Key takeaways

1. Avoid passing WaitGroup as argument
2. In case WaitGroup has to be passed as argument, do it by reference
3. Use the common WaitGroup usage syntax

wg.Add(1)         // 1 - Add 
go func() {       // 2 - The go routine
  defer wg.Done() // 3 - Done
  someFunc()      // 4 - Your Code
} ()              // 5 - Remember to call the goroutine

Concurrency is so tricky that still I’m not sure whether I’ve explained all above examples with enough and correct details. We all make mistakes and they help us learn. Have fun and AVOID using concurrent code in production till it’s really necessary.

Find me on Linkedin 👋🏻👋🏻.