Documentation for load testing Loki with xk6-loki (#5182)

* Add initial page for xk6-loki docs Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * added documentation for xk6-loki query scenario * Add simple installation and usage instructions Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add page about log generation with xk6-loki * Add documentation for k6 write scenario Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup: incorporate feedback from Karen Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup: incorporate feedback from Karen Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup: @ppcano's suggestions * fixup: @cyriltovena's suggestions * fixup: make read scenario more generic * Apply suggestions from code review Co-authored-by: Pepe Cano <pepe@loadimpact.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Use correct URL for git clone command Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Pepe Cano <pepe@loadimpact.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>
3 years ago · f0ef4662fe
parent 10b93cebe5
commit f0ef4662fe
5 changed files with 596 additions and 0 deletions
--- a/docs/sources/clients/_index.md
+++ b/docs/sources/clients/_index.md
@ -15,6 +15,8 @@ Grafana Loki supports the following official clients for sending logs:

 There are also a number of third-party clients, see [Unofficial clients](#unofficial-clients).

+The [xk6-loki extension](https://github.com/grafana/xk6-loki) permits [load testing Loki](k6/).
+
 ## Picking a client

 While all clients can be used simultaneously to cover multiple use cases, which
--- a/docs/sources/clients/k6/_index.md
+++ b/docs/sources/clients/k6/_index.md
@ -0,0 +1,106 @@
+---
+title: k6 load testing
+weight: 90
+---
+
+# k6 Loki extension load testing
+
+Grafana [k6](https://k6.io) is a modern load-testing tool.
+Its clean and approachable scripting [API](https://k6.io/docs/javascript-api/)
+works locally or in the cloud.
+Its configuration makes it flexible.
+
+The [xk6-loki extension](https://github.com/grafana/xk6-loki) permits pushing logs to and querying logs from a Loki instance.
+It acts as a Loki client, simulating real-world load to test the scalability,
+reliability, and performance of your Loki installation.
+
+## Before you begin
+
+k6 is written in Golang. [Download and install](https://go.dev/doc/install) a Go environment.
+
+## Installation
+
+`xk6-loki` is an extension to the k6 binary.
+Build a custom k6 binary that includes the `xk6-loki` extension.
+
+1. Install the `xk6` extension bundler:
+
+   ```bash
+   go install go.k6.io/xk6/cmd/xk6@latest
+   ```
+
+1. Check out the `grafana/xk6-loki` repository:
+
+   ```bash
+   git clone https://github.com/grafana/xk6-loki
+   cd xk6-loki
+   ```
+
+1. Build k6 with the extension:
+
+   ```bash
+   make k6
+   ```
+
+## Usage
+
+Use the custom-built k6 binary in the same way as a non-custom k6 binary:
+
+```bash
+./k6 run test.js
+```
+
+`test.js` is a Javascript load test.
+Refer to the [k6 documentation](https://k6.io/docs/) to get started.
+
+### Scripting API
+
+The custom-built k6 binary provides a Javascript `loki` module.
+
+Your Javascript load test imports the module: 
+
+```js
+import loki from 'k6/x/loki';
+```
+
+Classes of this module are:
+
+| class | description |
+| ----- | ----------- |
+| `Config` | configuration for the `Client` class |
+| `Client` | client for writing and reading logs from Loki |
+
+`Config` and `Client` must be called on the k6 init context (see
+[Test life cycle](https://k6.io/docs/using-k6/test-life-cycle/)) outside of the
+default function so the client is only configured once and shared between all
+VU iterations.
+
+The `Client` class exposes the following instance methods:
+
+| method | description |
+| ------ | ----------- |
+| `push()` | shortcut for `pushParameterized(5, 800*1024, 1024*1024)` |
+| `pushParameterized(streams, minSize, maxSize)` | execute push request ([POST /loki/api/v1/push]({{< relref "../../api/_index.md#post-lokiapiv1push" >}})) |
+| `instantQuery(query, limit)` | execute instant query  ([GET /loki/api/v1/query]({{< relref "../../api/_index.md#get-lokiapiv1query" >}})) |
+| `client.rangeQuery(query, duration, limit)` | execute range query  ([GET /loki/api/v1/query_range]({{< relref "../../api/_index.md#get-lokiapiv1query_range" >}})) |
+| `client.labelsQuery(duration)` | execute labels query  ([GET /loki/api/v1/labels]({{< relref "../../api/_index.md#get-lokiapiv1labels" >}})) |
+| `client.labelValuesQuery(label, duration)` | execute label values query  ([GET /loki/api/v1/label/\<name\>/values]({{< relref "../../api/_index.md#get-lokiapiv1labelnamevalues" >}})) |
+| `client.seriesQuery(matchers, duration)` | execute series query  ([GET /loki/api/v1/series]({{< relref "../../api/_index.md#series" >}})) |
+
+**Javascript load test example:**
+
+```js
+import loki from 'k6/x/loki';
+
+const timeout = 5000; // ms
+const conf = loki.Config("http://localhost:3100", timeout);
+const client = loki.Client(conf);
+
+export default () => {
+   client.pushParameterized(2, 512*1024, 1024*1024);
+};
+```
+
+Refer to
+[grafana/xk6-loki](https://github.com/grafana/xk6-loki#javascript-api)
+for the complete `k6/x/loki` module API reference.
--- a/docs/sources/clients/k6/log-generation.md
+++ b/docs/sources/clients/k6/log-generation.md
@ -0,0 +1,137 @@
+---
+title: Log generation
+weight: 10
+---
+# Log generation
+
+## Using `pushParameterized`
+
+Push logs to Loki with `pushParameterized`.
+This method generates batches of streams in a random fashion.
+This method requires three arguments:
+
+| name | description |
+| ---- | ----------- |
+| `streams` | number of streams per batch |
+| `minSize` | minimum batch size in bytes |
+| `maxSize` | maximum batch size in bytes |
+
+**Javascript example code fragment:**
+
+```javascript
+import loki from 'k6/x/loki';
+
+const KB = 1024;
+const MB = KB * KB;
+
+const conf = loki.Config("http://localhost:3100");
+const client = loki.Client(conf);
+
+export default () => {
+   client.pushParameterized(2, 500 * KB, 1 * MB);
+};
+```
+
+### Argument `streams`
+
+The first argument of the method is the desired amount of streams per batch.
+Instead of using a fixed amount of streams in each call, you can randomize the
+value to simulate a more realistic scenario.
+
+**Javascript example code fragment:**
+
+```javascript
+function randomInt(min, max) {
+  return Math.floor(Math.random() * (max - min + 1) + min);
+};
+
+export default () => {
+   let streams = randomInt(2, 8);
+   client.pushParameterized(streams, 500 * KB, 1 * MB);
+}
+```
+
+### Arguments `minSize` and `maxSize`
+
+The second and third argument of the method take the lower and upper bound of
+the batch size. The resulting batch size is a random value between the two
+arguments. This mimics the behaviour of a log client, such as Promtail or
+the Grafana Agent, where logs are buffered and pushed once a certain batch size
+is reached or after a certain size when no logs have been received.
+
+The batch size is not equal to the payload size, as the batch size only counts
+bytes of the raw logs. The payload may be compressed when Protobuf encoding
+is used.
+
+## Log format
+
+`xk6-loki` can emit log lines in six distinct formats. The label `format` of a stream defines the format of its log lines.
+
+* Apache common (`apache_common`)
+* Apache combined (`apache_combined`)
+* Apache error (`apache_error`)
+* [BSD syslog](https://datatracker.ietf.org/doc/html/rfc3164) (`rfc3164`)
+* [Syslog](https://datatracker.ietf.org/doc/html/rfc5424) (`rfc5424`)
+* JSON (`json`)
+
+Under the hood, the extension uses the library [flog](https://github.com/mingrammer/flog) for generating log lines.
+
+## Labels
+
+`xk6-loki` uses the following label names for generating streams:
+
+| name      | type     | cardinality     |
+| --------- | -------- | --------------- |
+| instance  | fixed    | 1 per k6 worker |
+| format    | fixed    | 6               |
+| os        | fixed    | 3               |
+| namespace | variable | >100            |
+| app       | variable | >100            |
+| pod       | variable | >100            |
+| language  | variable | >100            |
+| word      | variable | >100            |
+
+By default, variable labels are not used.
+However, you can specify the
+cardinality (quantity of distinct label values) using the `cardinality` argument
+in the `Config` constructor.
+
+**Javascript example code fragment:**
+```javascript
+import loki from 'k6/x/loki';
+
+const cardinality = {
+   "app": 1,
+   "namespace": 2,
+   "language": 2,
+   "pod": 5,
+};
+const conf = loki.Config("http://localhost:3100", 5000, 1.0, cardinality);
+const client = loki.Client(conf);
+```
+
+The total quantity of distinct streams is defined by the cartesian product of
+all label values. Keep in mind that high cardinality negatively impacts the performance of
+the Loki instance.
+
+## Payload encoding
+
+Loki accepts two kinds of push payload encodings: JSON and Protobuf.
+While JSON is easier for humans to read,
+Protobuf is optimized for performance
+and should be preferred when possible.
+
+To define the ratio of Protobuf to JSON requests, the client
+configuration accepts values of 0.0 to 1.0.
+0.0 means 100% JSON encoding, and 1.0 means 100% Protobuf encoding.
+
+The default value is 0.9.
+
+**Javascript example code fragment:**
+```javascript
+import loki from 'k6/x/loki';
+
+const ratio = 0.8; // 80% Protobuf, 20% JSON
+const conf = loki.Config("http://localhost:3100", 5000, ratio);
+const client = loki.Client(conf);
+```
--- a/docs/sources/clients/k6/query-scenario.md
+++ b/docs/sources/clients/k6/query-scenario.md
@ -0,0 +1,168 @@
+---
+title: Query testing
+weight: 30
+---
+# Query testing
+
+When designing a test scenario for load testing the read path of a Loki
+installation, it is important to know what types of queries you expect.
+
+## Supported query types
+
+Loki has 5 types of queries:
+
+* instant query
+* range query
+* labels query
+* label values query
+* series query
+
+In a real-world use-case, such as querying Loki using it as a Grafana
+datasource, all of these queries are used. Each of them has a different
+[API]({{< relref "../../api/_index.md" >}}) endpoint. The xk6-loki extension
+provides a [Javascript API](https://github.com/grafana/xk6-loki#javascript-api)
+for all these query types.
+
+
+### Instant query
+
+Instant queries can be executed using the function `instantQuery(query, limit)`
+on the `loki.Client` instance:
+
+**Javascript example code fragment:**
+
+```javascript
+export default () => {
+  client.instantQuery(`rate({app="my-app-name"} | logfmt | level="error" [5m])`);
+}
+```
+
+### Range query
+
+Range queries can be executed using the function `rangeQuery(query, duration, limit)`
+on the `loki.Client` instance:
+
+**Javascript example code fragment:**
+
+```javascript
+export default () => {
+  client.rangeQuery(`{app="my-app-name"} | logfmt | level="error"`, "15m");
+}
+```
+
+### Labels query
+
+Labels queries can be executed using the function `labelsQuery(duration)`
+on the `loki.Client` instance:
+
+**Javascript example code fragment:**
+
+```javascript
+export default () => {
+  client.labelsQuery("10m");
+}
+```
+
+### Label values query
+
+Label values queries can be executed using the function `labelValuesQuery(label, duration)`
+on the `loki.Client` instance:
+
+**Javascript example code fragment:**
+
+```javascript
+export default () => {
+  client.labelValuesQuery("app", "10m");
+}
+```
+
+### Series query
+
+Series queries can be executed using the function `seriesQuery(matcher, range)`
+on the `loki.Client` instance:
+
+**Javascript example code fragment:**
+
+```javascript
+export default () => {
+  client.seriesQuery(`match[]={app=~"loki-.*"}`, "10m");
+}
+```
+
+## Labels pool
+
+With the xk6-loki extension, you can use the field `labels` on the `Config`
+object. It contains label names and values that are generated in a reproducible
+manner. Use the same labels cardinality configuration for both `write` and
+`read` testing.
+
+**Javascript example code fragment:**
+
+```javascript
+const labelCardinality = {
+  "app": 5,
+  "namespace": 2,
+};
+const conf = new loki.Config(BASE_URL, 10000, 1.0, labelCardinality);
+const client = new loki.Client(conf);
+
+function randomChoice(items) {
+  return items[Math.floor(Math.random() * items.length)];
+}
+
+export default() {
+  let app = randomChoice(conf.labels.app);
+  let namespace = randomChoice(conf.labels.namespace);
+  client.rangeQuery(`{app="${app}", namespace="${namespace}"} | logfmt | level="error"`, "15m");
+}
+```
+
+Alternatively, you can define your own pool of label names and values,
+and then randomly select labels from your pool instead of a generated pool. 
+
+## Javascript example
+
+A more complex example of a read scenario can be found in xk6-loki repository.
+The test file
+[read-scenario.js](https://github.com/grafana/xk6-loki/blob/main/examples/read-scenario.js)
+can be resused and extended for your needs.
+
+It allows you to configure ratios for each type of query and the ratios of time
+ranges.
+
+**Javascript example:**
+
+```javascript
+const queryTypeRatioConfig = [
+  {
+    ratio: 0.1,
+    item: readLabels
+  },
+  {
+    ratio: 0.15,
+    item: readLabelValues
+  },
+  {
+    ratio: 0.05,
+    item: readSeries
+  },
+  {
+    ratio: 0.5,
+    item: readRange
+  },
+  {
+    ratio: 0.2,
+    item: readInstant
+  },
+];
+```
+
+This configuration would execute approximately
+
+- 10% labels requests
+- 15% label values requests
+- 5% requests for series
+- 50% range queries
+- 20% instant queries
+
+during a test run.
--- a/docs/sources/clients/k6/write-scenario.md
+++ b/docs/sources/clients/k6/write-scenario.md
@ -0,0 +1,183 @@
+---
+title: Write path testing
+weight: 20
+---
+# Write path load testing
+
+There are multiple considerations when
+load testing a Loki cluster's write path.
+
+The most important consideration is the setup of the target cluster.
+Keep these items in mind when writing your load test.
+
+- Deployment mode. The cluster might be deployed as
+a single-binary, as a simple scalable deployment, or as microservices
+
+- Quantity of component instances. This aids in predicting the need
+to horizontally scale the quantity of component instances.
+
+- Resource allocation such as CPU, memory, disk, and network.
+This aids in predicting the need to vertically scale the
+underlying hardware.
+
+These parameters can be adjusted in the load test:
+
+* The quantity of distinct labels and their cardinality
+
+    This will define how many active streams your load test will generate.
+    Start with only a few label values,
+    to keep the quantity of streams small enough,
+    such that it does not overwhelm your cluster.
+
+* The batch size the client sends
+
+    The batch size indirectly controls how many log lines per push request are
+    sent. The smaller the batch size, and the larger the quantity
+    of active streams you have,
+    the longer it takes before chunks are flushed.
+    Keeping lots of chunks
+    in the ingester increases memory consumption.
+
+* The number of virtual users (VUs)
+
+    VUs can be used to control the amount of parallelism with which logs should
+    be pushed. Every VU runs it's own loop of iterations.
+    Therfore, the number of VUs has the most impact on
+    the generated log throughput.
+    Since generating logs is CPU-intensive, there is a threshold above which
+    increasing the number VUs does not result in a higher amount of log data.
+    A rule of thumb is that the
+    most data can be generated when the number of VUs are set to 1-1.5 times
+    the quantity of CPU cores available on the k6 worker machine.
+    For example,
+    set the value in the range of 8 to 12 for an 8-core machine.
+
+* The way to run k6
+
+    k6 can be run locally, self-managed in a distributed way,
+    or it can be run highly-scalable within the k6 cloud.
+    Whereas running your k6 load test from a single (local
+    or remote) machine is easy to set up and fine for smaller Loki clusters,
+    the single machine does not load test large Loki installations,
+    because it cannot create the data to saturate the write path.
+    Therefore, it makes sense to run
+    the tests in the [k6 Cloud](https://k6.io/cloud/).
+
+## Metrics
+
+The extension collects two metrics that are printed in the
+[end-of-test summary](https://k6.io/docs/results-visualization/end-of-test-summary/) in addition to the built-in metrics.
+
+| name | description |
+| ---- | ----------- |
+| `loki_client_uncompressed_bytes` | the quantity of uncompressed log data pushed to Loki, in bytes |
+| `loki_client_lines` | the number of log lines pushed to Loki |
+
+## k6 value checks
+
+An HTTP request that successfully pushes logs to Loki
+responds with status `204 No Content`.
+The status code should be checked explicitly with a [k6 check](https://k6.io/docs/javascript-api/k6/check-val-sets-tags/).
+
+
+## Javascript example
+
+```javascript
+import { check, fail } from 'k6';
+import loki from 'k6/x/loki';
+
+/*
+ * Host name with port
+ * @constant {string}
+ */
+const HOST = "localhost:3100";
+
+/**
+ * Name of the Loki tenant
+ * passed as X-Scope-OrgID header to requests.
+ * If tenant is omitted, xk6-loki runs in multi-tenant mode,
+ * and every VU will use its own ID.
+ * @constant {string}
+ */
+const TENANT_ID = "my_org_id"
+
+/**
+ * URL used for push and query requests
+ * Path is automatically appended by the client
+ * @constant {string}
+ */
+const BASE_URL = `${TENANT_ID}@${HOST}`;
+
+/**
+ * Minimum amount of virtual users (VUs)
+ * @constant {number}
+ */
+const MIN_VUS = 1
+
+/**
+ * Maximum amount of virtual users (VUs)
+ * @constant {number}
+ */
+const MAX_VUS = 10;
+
+/**
+ * Constants for byte values
+ * @constant {number}
+ */
+const KB = 1024;
+const MB = KB * KB;
+
+/**
+ * Definition of test scenario
+ */
+export const options = {
+  thresholds: {
+    'http_req_failed': [{ threshold: 'rate<=0.01', abortOnFail: true }],
+  },
+  scenarios: {
+    write: {
+      executor: 'ramping-vus',
+      exec: 'write',
+      startVUs: MIN_VUS,
+      stages: [
+        { duration: '5m', target: MAX_VUS },
+        { duration: '30m', target: MAX_VUS },
+      ],
+      gracefulRampDown: '1m',
+    },
+  },
+};
+
+const labelCardinality = {
+  "app": 5,
+  "namespace": 1,
+};
+const timeout = 10000; // 10s
+const ratio = 0.9; // 90% Protobuf
+const conf = new loki.Config(BASE_URL, timeout, ratio, labelCardinality);
+const client = new loki.Client(conf);
+
+/**
+ * Entrypoint for write scenario
+ */
+export function write() {
+  let streams = randomInt(4, 8);
+  let res = client.pushParameterized(streams, 800 * KB, 1 * MB);
+  check(res,
+    {
+      'successful write': (res) => {
+        let success = res.status === 204;
+        if (!success) console.log(res.status, res.body);
+        return success;
+      },
+    }
+  );
+}
+
+/**
+ * Return a random integer between min and max including min and max
+ */
+function randomInt(min, max) {
+  return Math.floor(Math.random() * (max - min + 1) + min);
+}
+```