Add the option to configure memory ballast for Loki (#5081)

* Add memory ballast

* Add changelog entry

* Add documentation

Add additional information to Flag description

* Ensure that ballast_bytes can be configured from config file

Add documentation

* Add Operational scalability documentation related to memory ballast

* Restore config.ballast-bytes cli flag

* Move config.ballast-bytes cli flag declaration

Co-authored-by: Owen Diehl <ow.diehl@gmail.com>

CHANGELOG.md

@@ -1,5 +1,6 @@
 ## Main
 
+* [5081](https://github.com/grafana/loki/pull/5081) **SasSwart**: Add the option to configure memory ballast for Loki
 * [5085](https://github.com/grafana/loki/pull/5085) **aknuds1**: Upgrade Cortex to [e0807c4eb487](https://github.com/cortexproject/cortex/compare/4e9fc3a2b5ab..e0807c4eb487) and Prometheus to [692a54649ed7](https://github.com/prometheus/prometheus/compare/2a3d62ac8456..692a54649ed7)
 * [5067](https://github.com/grafana/loki/pull/5057) **cstyan**: Add a metric to Azure Blob Storage client to track total egress bytes
 * [4950](https://github.com/grafana/loki/pull/4950) **DylanGuedes**: Implement common instance addr/net interface

cmd/loki/main.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"os"
 	"reflect"
+	"runtime"
 
 	"github.com/go-kit/log/level"
 	"github.com/prometheus/common/version"
@@ -87,6 +88,12 @@ func main() {
 		}()
 	}
 
+	// Allocate a block of memory to reduce the frequency of garbage collection.
+	// The larger the ballast, the lower the garbage collection frequency.
+	// https://github.com/grafana/loki/issues/781
+	ballast := make([]byte, config.BallastBytes)
+	runtime.KeepAlive(ballast)
+
 	// Start Loki
 	t, err := loki.New(config.Config)
 	util_log.CheckFatal("initialising loki", err)

docs/sources/configuration/_index.md

@@ -93,6 +93,12 @@ Pass the `-config.expand-env` flag at the command line to enable this way of set
 # if true. If false, the OrgID will always be set to "fake".
 [auth_enabled: <boolean> | default = true]
 
+# The amount of virtual memory to reserve as a ballast in order to optimise
+# garbage collection. Larger ballasts result in fewer garbage collection passes, reducing CPU overhead at
+# the cost of heap size. The ballast will not consume physical memory, because it is never read from.
+# It will, however, distort metrics, because it is counted as live memory.
+[ballast_bytes: <int> | default = 0]
+
 # Configures the server of the launched module(s).
 [server: <server>]
 

docs/sources/operations/scalability.md

@@ -22,3 +22,8 @@ In order to run with the Query Scheduler, the frontend needs to be passed the sc
 It is not valid to start the querier with both a configured frontend and a scheduler address. 
 
 The query scheduler process itself can be started via the `-target=query-scheduler` option of the Loki Docker image. For instance, `docker run grafana/loki:latest -config.file=/cortex/config/cortex.yaml -target=query-scheduler -server.http-listen-port=8009 -server.grpc-listen-port=9009` starts the query scheduler listening on ports `8009` and `9009`.
+
+## Memory Ballast
+In compute constrained environments, garbage collection can become a significant factor. This can be optimised, at the expense of memory consumption, by configuring a memory ballast using the `ballast_bytes` configuration option.
+
+A larger memory ballast will mean that standard heap size volatility is less relatively significant, and therefore less likely to trigger garbage collection. This will result in lower compute overhead, but higher memory consumption, as the heap is cleaned less frequently.

pkg/loki/loki.go

@@ -53,9 +53,10 @@ import (
 
 // Config is the root config for Loki.
 type Config struct {
-	Target      flagext.StringSliceCSV `yaml:"target,omitempty"`
-	AuthEnabled bool                   `yaml:"auth_enabled,omitempty"`
-	HTTPPrefix  string                 `yaml:"http_prefix"`
+	Target       flagext.StringSliceCSV `yaml:"target,omitempty"`
+	AuthEnabled  bool                   `yaml:"auth_enabled,omitempty"`
+	HTTPPrefix   string                 `yaml:"http_prefix"`
+	BallastBytes int                    `yaml:"ballast_bytes"`
 
 	Common           common.Config            `yaml:"common,omitempty"`
 	Server           server.Config            `yaml:"server,omitempty"`
@@ -90,6 +91,8 @@ func (c *Config) RegisterFlags(f *flag.FlagSet) {
 		"The alias 'all' can be used in the list to load a number of core modules and will enable single-binary mode. "+
 		"The aliases 'read' and 'write' can be used to only run components related to the read path or write path, respectively.")
 	f.BoolVar(&c.AuthEnabled, "auth.enabled", true, "Set to false to disable auth.")
+	f.IntVar(&c.BallastBytes, "config.ballast-bytes", 0, "The amount of virtual memory to reserve as a ballast in order to optimise "+
+		"garbage collection. Larger ballasts result in fewer garbage collection passes, reducing compute overhead at the cost of memory usage.")
 
 	c.registerServerFlagsWithChangedDefaultValues(f)
 	c.Common.RegisterFlags(f)