Goroutine Leak Profiles
Go 1.27 includes new goroutine leak profiles.
以下正文同步自 Go Blog，版权归原站所有，已转换为易读排版。
The Go Blog
Goroutine Leak Profiles
Vlad Saioc
2 September 2026
Go’s concurrency features are powerful and easy to use, but
that same ease can sometimes lead even seasoned developers to make
mistakes.
Fortunately, the Go ecosystem comes equipped with useful tools for
debugging, e.g., the race detector,
but even existing tools may miss some concurrency bugs,
such as the topic of this article, the goroutine leak.
Goroutines synchronize or exchange information
via shared concurrency primitives, e.g., channels, locks, and wait groups.
While communicating, goroutines often block on these primitives,
as in, wait until some condition is met;
ubiquitous examples include waiting to acquire a held mutex,
or receive a message over a channel.
Goroutines can also block on operating system operations, like reading from a network socket or a file.
We may consider a goroutine leaked if it is blocked,
but the conditions needed to unblock it can never be met.
Over time, an accumulation of leaked goroutines degrades
performance through excessive memory usage (by the leaked
goroutines themselves or the memory they reference), as well as
CPU usage from the garbage collector, especially
if GOMEMLIMIT is in use.
Goroutine leaks can be notoriously difficult to detect.
In unit testing, the most significant breakthroughs include
the open-source library goleak,
which can instrument individual tests to signal any
un-terminated goroutines after the test wraps up as suspicious.
Similarly, Go 1.25 introduced the synctest package to
the standard library; it can significantly improve
the quality of unit tests in concurrent code by giving
Go developers more control over the ordering of concurrent events
in order to reliably test hard-to-reproduce scenarios.
Unfortunately, neither approach can check for goroutine leaks
in production systems, especially at larger scales,
which may behave in ways unaccounted for by tests.
Goroutine profiles are a rudimentary way to check for operations
that block too many goroutines, or analyze growth trends.
However, goroutine profiles cannot distinguish between
goroutines which are leaked, and those which are temporarily blocked
in high numbers by design, e.g., as caused by increased
traffic in a microservice.
Likewise, leaks which are low in number may slip by undetected for many years.
Go 1.27 introduces the goroutine leak profiler,
a flexible and lightweight mechanism for finding
goroutine leaks in running Go programs, including production systems.
Unlike previous approaches, which require human analysis,
this mechanism is precise and generates little-to-no false positives.
The trade-off is that it is limited to a subset of goroutine leaks:
goroutines permanently blocked on channels or primitives
in the sync package.
Luckily for us, this already covers a very large subset of goroutine leaks,
as we’ll see in our examples.
In the following sections, we showcase how to use the feature, followed by
some additional examples of detectable leaks, and a description of the
underlying implementation and trade-offs.
Example: concurrent workers
Consider a function that processes work items concurrently:
type result struct {
res workResult
err error
}
func processWorkItems(ws []workItem) ([]workResult, error) {
// Process work items in parallel, aggregating results in ch.
ch := make(chan result)
for _, w := range ws {
go func() {
res, err := processWorkItem(w)
ch
Because ch is an unbuffered channel, each worker goroutine blocks when sending
its result until the main goroutine receives from the channel.
If processWorkItems returns early due to an error, the receiving loop terminates,
and all remaining sender goroutines block forever.
This example is emblematic of a common mistake discovered in real Go programs,
including Uber production services.
Let’s see how we can find these leaks by using the
new goroutine leak profiler.
Debugging with the goroutine leak profiler
The profile is available through the
runtime/pprof package, as the
goroutineleak profile type, or by installing the profile handlers defined
by the net/http/pprof package.
If you already have net/http/pprof set up in your service,
then you don’t need to do anything else! The profile will be
automatically made available for collection at the /debug/pprof/goroutineleak
endpoint on whatever host and port the handlers are installed.
Let’s put our concurrency bug in context and set up the net/http/pprof package.
This way, you can try it yourself!
package main
import (
"errors"
"log"
"net/http"
_ "net/http/pprof"
"time"
)
type workItem int
type workResult int
func processWorkItem(w workItem) (workResult, error) {
time.Sleep(10 * time.Millisecond)
if w == 5 {
return 0, errors.New("simulated error")
}
return workResult(w * 2), nil
}
type result struct {
res workResult
err error
}
func processWorkItems(ws []workItem) ([]workResult, error) {
ch := make(chan result)
for _, w := range ws {
go func() {
res, err := processWorkItem(w)
ch
Build the program above, then run it:
$ go build -o leaky
$ ./leaky
Collecting the profile
It won’t take long for the program to start accumulating
leaks, which you can then view by using the web UI
at http://localhost:6060/debug/pprof.
Alternatively, you can collect the goroutine
leak profile using curl, and then examine it with go tool pprof:
$ curl http://localhost:6060/debug/pprof/goroutineleak > leak.prof
$ go tool pprof leak.prof
Type: goroutineleak
Time: 2026-03-01 13:19:49 UTC
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) list processWorkItems
Total: 116
ROUTINE ======================== main.processWorkItems.func1 in .../main.go
0 116 (flat, cum) 100% of Total
. . 31: go func() {
. . 32: res, err := processWorkItem(w)
. 116 33: ch
The profile reveals the goroutines leaked at
ch (line 33), pinpointing the culprit operation.
Notably, the longer the program is running, the larger the number of leaked
goroutines.
Addressing the leak
This leak can be simply fixed by giving ch a buffer:
ch := make(chan result, len(ws))
This allows all the work item goroutines to send a message without blocking
in the event of a premature return of processWorkItems.
We list more real-world examples in this section.
Implementation
This section is for those interested how leak detection
works under the hood of the goroutine leak profiler.
For details strictly pertaining to performance overhead and limitations,
skip ahead to this section.
Core concept
Let’s start with an initial observation: if a goroutine
is blocked over some concurrency primitive that no other goroutine has access to
(in this case, via a reference in memory), then it is obviously leaked.
This is already a strong lead, we can generalize it further into a definition
for when a goroutine is not leaked, a property we term as liveness.
We formally define liveness, an inductive property
as follows:
A goroutine is live if:
- it is not blocked by a concurrency primitive, or
- at least one concurrency primitive that blocks it is referenced
by another live goroutine.
In the trivial case, goroutines which are not blocked are obviously
not leaked.
In the inductive case, the underlying assumption is that
any goroutine which is not leaked may eventually use
concurrency primitives it references to unblock any
other goroutines blocked by those primitives.
To find all live goroutines, we start from the obviously live
unblocked goroutines and trace any references
they hold, i.e., through their local variables, to find
the concurrency primitives they have access to.
We then incrementally include any goroutines blocked over those
primitives as live, and repeat the process until no
additional live goroutines are discovered.
Fortunately for us, the Go runtime already computes memory reachability
through the garbage collector (GC),
so the next step is to adapt the GC to suit our purposes.
You can quickly compare the two GCs with the following diagrams:
A complete overhaul of the GC is not necessary.
The Go runtime uses a concurrent tri-color mark-and-sweep garbage collector,
(now with the Green Tea variant!),
so its MO already neatly aligns with our goals.
Only a few key changes are needed:
- In the initial phases, the regular GC marks all goroutines (and global variables)
as reachable, such that they would never be considered garbage,
i.e., they are mark roots.
We change it to instead only include unblocked goroutines,
since these are guaranteed to be live.
- This is followed by the marking phase, where the GC traces objects referenced
(transitively) by the mark roots, and marks them as usable memory.
Even though we do not modify this phase directly, the changes in step 1.
implicitly ensure that the GC only marks memory referenced by live goroutines.
- The marking phase is finalized by inspecting all the blocked
goroutines not included as mark roots in step 1.
If a goroutine is blocked by at least one concurrency
primitive that has been marked in step 2., it is added as a mark root,
and the GC resumes the marking phase from step 2.
This coincides with the inductive step in the definition
of liveness.
- Once all live goroutines have been discovered, any goroutine
which has not been added as a mark root has its status set to leaked.
- The marking phase then resumes one last time with all the leaked goroutines
added as mark roots, allowing the GC to mark all the memory it would have
marked during a regular run.
Once the GC cycle is complete, the goroutine leak profiler picks up
like in a regular goroutine profile, and filters for strictly
leaked goroutines.
Limitations
The examples above demonstrate the usefulness of goroutine leak profiles.
Nevertheless, the garbage collector has some limitations that may lead
it to miss leaks:
-
Memory overreach: if a concurrency primitive is
consistently reachable through global variables or runnable goroutines,
then goroutines blocking on it are never reported as leaked, even if
that concurrency primitive is never used in the future.
This can be alleviated by better regimenting access to
concurrency primitive references, and more clearly
delineating their lifecycle.
-
Non-standard blocking:
For the sake of correctness, goroutine leak detection is strictly limited
to Go first-class concurrency primitives, which includes:
channel send and receive operations (including over nil channels),
blocking select statements, i.e., with no default case, up to, and including
select statements with no cases, and members of the
sync package, specifically Mutex,
RWMutex, WaitGroup and Cond.
Goroutines blocked for any other reason, e.g.,
file and network IO, or direct system calls
are never considered as leaked.
This likewise applies for custom, user-defined concurrency,
e.g., spin locks, unless they rely on the primitives outlined above
for their underlying implementation.
-
Non-determinism: leaks can be detected only after
they have occurred, but cannot be otherwise predicted,
so reproducing and diagnosing leaks in flaky programs
continues to be a challenge.
For the best results, we encourage mixing approaches, by using
goroutine leak profiles at various layers, up to, and including production,
as well as comprehensive test suites instrumented with goleak and synctest.
Performance impact
Goroutine leak detection is carefully designed to minimize
performance impact, but there are, nevertheless, some costs.
While memory overhead is negligible, only limited to small additions
required for bookkeeping, goroutine leak detection can be slower
than the regular GC.
This is best illustrated through a pathological case we
call the “daisy-chain”:
In this leak-free example, runnable goroutine G₀ has a
reference to primitive P₁ which blocks G₁, and so on.
This implies that proving liveness for some Pᵢ₊₁,
requires proving liveness for Pᵢ, which introduces
two costs:
- The GC marking phase is effectively serialized relative to the
order in which goroutines can be scanned, as all the memory reachable
from some Pᵢ must be marked before Pᵢ
正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。