Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Hamish Forbes d0baa40df3 Only modify identical timestamps when increment-duplicate-timestamps enabled (#9437) ... When `-validation.increment-duplicate-timestamps` is enabled out of order log entries within the same push request can have their timestamps incorrectly mutated. The code is currently just looking for 2 consecutive entries where the timestamp is not increasing. It should only affect entries where the timestamp is a *duplicate* Test push request payload: ```json { "streams": [ { "stream": { "increment_duplicate_timestamp": "fixed|true|false" }, "values": [ [ "1683665111628000000", "1a: 1683665111628000000" ], [ "1683665111628000000", "1b: 1683665111628000000" ], [ "1683665111628000003", "3: 1683665111628000003" ], [ "1683665111628000002", "2: 1683665111628000002" ] ] } ] } ```` As only the first 2 of these timestamps are the same I would expect all but `1b` to have the timestamps recorded as-is and the logs returned by a query in the order: 1a,1b,2,3. Actual Results: ``` > logcli query --forward --limit 100 -z UTC -q --no-labels --since 60m -o jsonl '{increment_duplicate_timestamp="fixed"}' | jq { "line": "1a: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "1b: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628000001Z" } { "line": "2: 1683665111628000002", "timestamp": "2023-05-09T20:45:11.628000002Z" } { "line": "3: 1683665111628000003", "timestamp": "2023-05-09T20:45:11.628000003Z" } > logcli query --forward --limit 100 -z UTC -q --no-labels --since 60m -o jsonl '{increment_duplicate_timestamp="true"}' | jq { "line": "1a: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "1b: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628000001Z" } { "line": "3: 1683665111628000003", "timestamp": "2023-05-09T20:45:11.628000003Z" } { "line": "2: 1683665111628000002", "timestamp": "2023-05-09T20:45:11.628000004Z" } > logcli query --forward --limit 100 -z UTC -q --no-labels --since 60m -o jsonl '{increment_duplicate_timestamp="false"}' | jq { "line": "1a: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "1b: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "2: 1683665111628000002", "timestamp": "2023-05-09T20:45:11.628000002Z" } { "line": "3: 1683665111628000003", "timestamp": "2023-05-09T20:45:11.628000003Z" } ``` `true` and `false` were written to Loki 2.8.2 with the `-validation.increment-duplicate-timestamps` enabled or disabled `fixed` with this patch applied and the flag enabled **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`		2 years ago
..
clientpool	…
shardstreams	…
writefailures	LogPushFailures: Log the tenant id (#9819)	2 years ago
distributor.go	Only modify identical timestamps when increment-duplicate-timestamps enabled (#9437)	2 years ago
distributor_ring.go	fix function names in comments (#9415)	2 years ago
distributor_test.go	Only modify identical timestamps when increment-duplicate-timestamps enabled (#9437)	2 years ago
http.go	Failures logging: avoid logging generic errors (#9796)	2 years ago
http_test.go	…
ingestion_rate_strategy.go	Allow passing of context to query related limits functions (#8689)	2 years ago
ingestion_rate_strategy_test.go	…
instance_count.go	Distributor: Add AutoForget support (#8067)	2 years ago
instance_count_test.go	Distributor: Add AutoForget support (#8067)	2 years ago
limits.go	Allow passing of context to query related limits functions (#8689)	2 years ago
ratestore.go	Stream Sharding amortize push payloads across the observed rate. (#9414)	2 years ago
ratestore_metrics.go	…
ratestore_test.go	Upgrade `golangci-lint` and fix linting errors (#9601)	2 years ago
shard_tracker.go	…
shard_tracker_test.go	…
streamsharder_test.go	…
validator.go	Log push failures per-entry instead of in batches (#9720)	2 years ago
validator_test.go	Log push failures per-entry instead of in batches (#9720)	2 years ago