18. 06. 2019 MarinovMihail NetEye, Unified Monitoring

Go pprof – How to Understand Where There is Memory Retention

Recently I had to face a problem concerning a daemon written in Go that was consuming all available memory – causing critical and unpredictable situations for all the other components of the system.

The first alarm came from NetEye: There was something that was requesting more and more memory.

It was easy to understand which process it was: ‘top’ was quite clear. But why?

Go has a garbage collector that frees memory on the heap when it is no longer being referenced. In our case there was something that was forgetting to release the unused memory references, causing a memory retention – but what?

The Go pprof package allows you to enable sampled dumps of the heap, making the data available at a web-based endpoint over http. Once available, this dump can be analyzed to understand both the current allocated memory and the cumulative memory allocation.

The only thing you need to do is import net/http/pprof and start an http listener. The pprof package will register its endpoints right below the web server root at /debug/pprof.

import (
  "net/http"
  _ "net/http/pprof"
)
...
func main() {
  ...
  http.ListenAndServe("localhost:8080", nil)
}

After that you will be able to get heap data just by using a browser or curl, connecting via http://localhost:8080/debug/pprof/heap:

curl -s http://localhost:8080/debug/pprof/heap > /tmp/heap.out

Once you get data, you can analyze it using the pprof tool:

go tool pprof /tmp/heap.out

or by connecting to the web server:

go tool pprof http://localhost:8080/debug/pprof/heap

In order to analyze the currently used memory, just select the inuse indices (bytes or object counts):

  • inuse_space:   memory allocated but not yet released
  • inuse_objects:  objects allocated but not yet released

If you are interested in understanding the total amount of bytes or objects, use the alloc indices instead:

  • alloc_space:   the  total amount of memory allocated
  • alloc_objects : the  total number of objects allocated

Once I ran the pprof tool, it was easy to understand what exactly was happening with the daemon at fault.

go tool pprof --inuse_objects http://localhost:8080/debug/pprof/heap
  • flat: the memory allocated and held by this function
  • cum: the memory allocated by this function together with functions that it has called

It is also possible to see additional details for memory used in functions called further down the stack:

MarinovMihail

MarinovMihail

Developer at Würth Phoenix
“Hi guys! I’m Mihail and since the university years I has been fascinated by distributed systems and measurements on them. Now when I join the Neteye project I get the possibility to continue with this passion and this is great. My free time is completely dedicated to my wife and my daughters, I simply love them.”

Author

MarinovMihail

“Hi guys! I’m Mihail and since the university years I has been fascinated by distributed systems and measurements on them. Now when I join the Neteye project I get the possibility to continue with this passion and this is great. My free time is completely dedicated to my wife and my daughters, I simply love them.”

Leave a Reply

Your email address will not be published. Required fields are marked *

Archive