1045 Commits (a58aea1bf456280aa06fb2f2d09d2d005eefa6ed)

Author	SHA1	Message	Date
Arve Knudsen	a5790f332e	Chore: Use runtimeconfig from dskit (#4227) ... * Use runtimeconfig from dskit Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * runtimeconfig: Use cortex_ prefix for metrics Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Bryan Boreham	13a3df8826	Simplify Distributor.push (#4240) ... Instead of creating another goroutine to wait for the background `sendSamples()` to finish, give the channels it is sending on a buffer so that they can send without a listener. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 years ago
李国忠	6f78ffe33c	[fix] distributor: fix goroutine leak (#4238) ... Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>	4 years ago
Owen Diehl	79d53317dd	bumps per stream default rate limits (#4228) ... After some testing, this seems a more reasonable balance between protecting ingesters and good defaults out of the box	4 years ago
Arve Knudsen	cfb4fc1f55	Use flagext from dskit (#4225) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Nick Pillitteri	79a7bdbabc	Chore: Use services and modules from grafana/dskit (#4196) ... * Update Cortex vendor to bcf15611345da65e03604a08a0213aaaaac49e54 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Use services and modules packages from grafana/dskit Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Update promtail test to set default weaveworks server network type Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Punctuation Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Owen Diehl	668622c813	Refactor per stream rate limit (#4213) ... * uses new StreamRateLimiter * sets burst before limiter, uses requested ts * creates new non infinite rate limiters when configs change * time/rate testware * less indirection & protect rate limit access from race conditions * removes old comment * always recreate stream limiter after config change	4 years ago
So Koide	b72d8abfbc	Authc/z: Enable grpc_client_config to allow mTLS (#4176) ... * feat: enable gRPC client config * Update pkg/ingester/client/client.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	abcf4d083f	WAL replay discard metrics (#4212) ... * adds replay discard metrics * lint	4 years ago
Benoît Knecht	488a21bff7	pkg/storage/chunk/aws: Add s3.http.ca-file option (#4211) ... * pkg/storage/chunk/aws: Fix insecure-skip-verify documentation The documentation claimed that the `s3.http.insecure-skip-verify` option needed to be set to `false` in order to skip verifying certificates, but it needs to be set to `true`. Signed-off-by: Benoît Knecht <bknecht@protonmail.ch> * pkg/storage/chunk/aws: Add s3.http.ca-file option This option lets users set a custom CA file for connections to S3, in case they use a local S3 instance with an internal PKI. Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>	4 years ago
Owen Diehl	d42de7ee2b	fixes typo (#4199)	4 years ago
Callum Styan	757eeacb88	[ingester/stream]: Add a byte stream rate limit. (#4191) ... * Add a byte stream rate limit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Track discarded lines due to stream rate limit + use ByteSize for the config value. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address review comments Signed-off-by: Callum Styan <callumstyan@gmail.com> * Another round of review fixes. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix goimports issue. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix duplicate flag registration + stream rate limit test. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix stream rate/burst limit flags yaml naming. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix tests now that we have stream rate limit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * fix some comments Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Callum Styan	3ee8ed3b1d	Allow for loki-canary to generate a percentage of out of order log lines (#4126) ... * Allow for loki-canary to generate a percentage of log lines with out of order timestamps, percentage and out of order time range is all configurable. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Switch OoO min/max flags to time duration and properly calculate OoO percentage check. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Change some variable names + add comments to make it more obvious what various parts of the canary code are actually doing. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update pkg/canary/writer/writer.go * Add notes about usage of the out-of-order-percentage flag and the requirement to use pipeline stages. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Arve Knudsen	10c92a901e	Chore: Upgrade to latest Cortex (#4188) ... * Chore: Upgrade to latest Cortex Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix formatting Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Danny Kopping	ea735ab8c0	Write meaningful logs when remote-write is disabled or is misconfigured (#4141) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Owen Diehl	9cdb1295b5	adds loki_ingester_wal_replay_active metric and records this more acc… (#4193) ... * adds loki_ingester_wal_replay_active metric and records this more accurately * adds WAL to msg	4 years ago
Owen Diehl	ff2ced09ea	Cleanup/unordered writes ingester config (#4192) ... * handle replaying unordered WAL into ordered configs. * ingester limiter metric, testware, correctly replayed unordered WAL into ordered config	4 years ago
Sandeep Sukhani	29b07d190d	Retention speedup (#4172) ... * drop the multi table chunk entries from index when the whole table is out of retention * speed up cleaning of deleted series from index * mark partially deleted chunk for deletion when all its tables are processed	4 years ago
Owen Diehl	f17b189a3f	Promote/ruler api (#4132) ... * unmark the ruler api as experimental * deprecation notice for experimental prefix * removes vendor changes, adds -ruler.enable-api (without experimental prefix) on our ruler.Config	4 years ago
Owen Diehl	3f6f2fc717	remove unused code and reformat (#4155)	4 years ago
Owen Diehl	eb5643be26	initiate grpc health check always (#4181)	4 years ago
Sandeep Sukhani	6b775ac31c	allow setting tenant id for querying logs from loki (#4185)	4 years ago
Sandeep Sukhani	8ef3d5fee9	add a storage client for boltdb-shipper which would do all the object key management for storage operations (#4128) ... * add a storage client for boltdb-shipper which would do all the object key management for storage operations * changes suggested from PR review	4 years ago
Owen Diehl	6ac96478a7	safely close nonOverlapping iterators (#4160)	4 years ago
Owen Diehl	bd65883a50	uses more fleshed out cortex auth utility & adds new auth-ignored routes (#4153)	4 years ago
Owen Diehl	066f0bf3c1	Tenant/unordered (#4130) ... * makes unordered writes a tenant config * json tag annotation	4 years ago
Owen Diehl	92a4e57448	ingester.index-shards config (#4111) ... * consistent shard mapping test & better error msg * ingester.index-shards config * loki fails to start if the ingester & schema factors arent compatible. * lint * docs * update changelog	4 years ago
Owen Diehl	45693b7941	stream iterators account for unordered data (#4114)	4 years ago
Owen Diehl	572bbe15ec	Discard/ooo (#4104) ... * stream includes tenant id * expose out_of_order discarded metrics	4 years ago
Sandeep Sukhani	67fe2d68a3	fix finding tables which would have out of retention data (#4107)	4 years ago
Karsten Jeschkies	c3aaebe509	Support frontend V2 with query scheduler. (#4071) ... * Support Cortex Frontend V2. * Format code. * Start query scheduler in Docker compose. * Model query flow in sequence diagram. * Model calls to downstreamer. * Move diagram to docs. * Embed generated SVG. * Descripe calls. * Launch Cortex query scheduler. * Disable sharding for now. * Log which framework is used. * Configure querier to use scheduler. * Remove unwanted docs. * Enable query splitting again. * Document scheduler address. * Document new scheduler options for frontend as well. * Remove unused querier option. * Use new query-scheduler module in Loki. * Remove unused option. * Document `max_queriers_per_tenant`.	4 years ago
Travis Patterson	8a2081a39e	Distributor: Truncate rather than drop log lines (#4051) ... * Truncate rather than drop logs - When log lines are longer than maxLineSize, truncate them to maxLineSize rather than dropping them Fixes: #3747 * Review feedback: - Rename 'max_line_size_should_truncate' to 'max_line_size_truncate' - Add configurable truncated indicator - Add Truncated Line metric - Changed truncated metrics to have the name mutated_{samples,bytes}_total * Linter * Review Feedback - Make metric variable names match metric names - Add benchmarks - reduce truncation allocs by 100x * Linter * Remove truncation indicator	4 years ago
Cyril Tovena	1efeace644	Fixes instant queries in the frontend. (#4091) ... * Fixes instant queries in the frontend. This is a set of fixes for instant queries: - correctly propagate statistics. - correctly convert sampleStream to vector. - correctly marshal to json. - correctly encode the time for the request. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	10bd165996	Fixes unaligned shards between ingesters and storage. (#4087) ... * Fixes unaligned shards between ingesters and storage. Currently shards computation are not correctly aligned between ingester and storage causing bad metric results. - Fixed the hash caculation which was missing the metric name `logs`. - Revert some changes from 5f0e2453fe which was not keeping the alignement of shards. For example if the shard is 9 of 16 total, then it should also be 9 and 25 of 32. The previous commit was trying to use 10 and 11 of 32. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	138ca71dd5	add a method to check whether error is due to a missing object in storage while reading or deleting the object (#4081)	5 years ago
Danny Kopping	52da3844f9	Ruler: Adding group_key label to recording_rules_samples_queue_capacity metric (#4080) ... * Adding group_key label to recording_rules_samples_queue_capacity metric Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Cyril Tovena	ac6ace8dbd	Make `ParseExternalKey` from the chunk package alloc free. (#4075) ... ``` ~/go/src/github.com/grafana/loki main* ⇣ ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_ParseExternalKey-16 569 415 -27.03% benchmark old allocs new allocs delta Benchmark_ParseExternalKey-16 2 0 -100.00% benchmark old bytes new bytes delta Benchmark_ParseExternalKey-16 96 0 -100.00% ``` This is a very hot function in Loki being used for every chunk reference we get from the index. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	88c25965da	Inject the org-id if possible from the start. (#4068) ... This allows the http logger to include the org-id. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	19aec6e6a3	fix broken TestCachingStorageClientStoreQueries (#4074)	5 years ago
Sandeep Sukhani	d31f8676fd	Optimize index queries and caching (#4070) ... * Use RangeValuePrefix for building queries for Series IDs * optimize index store queries and caching by not doing broad queries	5 years ago
Cyril Tovena	32e55c3043	Fork cortex chunk storage into Loki. (#3842) ... The chunk package has been forked from this commit for future reference and licensing. 09fe17c0e6	5 years ago
Owen Diehl	56256bfd5b	Unordered chunks (#4001) ... * merge feature/unordered-replay * interoperable head chunks * memchunk block interop * retain ordered memchunk optimizations when possible * tests+bench for unordered chunk reads * reorder on chunk close * [wip] ingester stream unorderd * unordered writes default in testware config, fixes OOO bug & removes unused lastChunkTimestamp var * validity window is 1/2 of max age & fixes old transfer test * more consistent headblock checking/creation * more cohesive encoding tests * unordered stream test with validity bounds * compat - unordered * reinstates memchunk defaults when rebounding & updates storage test compatibility * lint * reorder across blocks doesnt overflow * respect chunk configs during rebounding when possible * only sync checks on ordered writes	5 years ago
Cyril Tovena	4c2feb32b5	Fixes test helper `newChunk` in the storage pkg. (#4056) ... I wanted to test that overlapping chunks are correctly re-ordered and bump into a rounding issue. Keeping both for future will avoid the same mistake. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Oleg Zaytsev	b4086dfe33	Upgrade Prometheus and Cortex (#4049) ... This bumps prometheus up to the changes from https://github.com/prometheus/prometheus/pull/9083 and Cortex to https://github.com/cortexproject/cortex/pull/4380 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	5 years ago
Danny Kopping	0c6b4dc8c9	Ruler: implementing write relabelling on recording rule samples (#4048) ... * Implementing write relabelling Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Cyril Tovena	3f3d10afcc	Add min_sharding_lookback limits to the frontends (#4047) ... * Add min_sharding_lookback limits to the frontends Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Karsten Jeschkies	17f4a73a24	Filter instant queries and shard them. (#3984) ... * Filter instan queries and shard them. * [WIP] Test instant query sharding * Trace casting error. * WRap LokiInstantRequest for Params interface. * Convert paras to request. * Convert vector. * Use proper query time stamp. * Assert number of calls. * Convert vector to multiple samples. * Format code. * Use type switch. * Format code. * Rename Matrix to proto conversion. * Remvoe comment on caching. * Update pkg/querier/queryrange/roundtrip_test.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove nil middlware. * Handle LokiInstantRequest correctly. * Assert reponse type. * Return LokiPromResponse. * Verify err in tests. * Set shards. * Correct result type. * Use limits middleware. * Use shars parameter in querier. * Return vector. * Unify params. * Change direction to FORWARD. Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Arve Knudsen	3e385464c6	Chore: Upgrade Prometheus and Cortex (#4017) ... * Chore: Upgrade Prometheus Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Upgrade Cortex Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Upgrade Cortex Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Undo unnecessary changes Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	5 years ago
Jack Baldry	e22e666ca4	fix: Restore /config endpoint and correct handlerFunc for buildinfo (#4027) ... As far as I can tell from reading the "next" documentation, the /loki/api/v1/status/buildinfo endpoint should return the build information. The /version endpoint is also listed further down in the documentation but from the code prior to this change, it looks like it should perform the same function as buildinfo endpoint so I have removed it. I've updated the documentation to reflect my changes. Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	5 years ago
Kaviraj	bfb24e6a4c	Ip matcher for LogQL (#3986) ... * IP matcher and filter via netaddr.IP package Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Adding IP filter into label filter gramer 1. ast tests 2. parse tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Label filters all test cases passes Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Rename ipFilter -> ipLabelFilter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * More tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove testlogs.txt Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add more cases to the benchmark Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks 1. support NEQ 2. reduce allocation. 3. Minor tweaks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make IP filter work as line filter 1. Add necessary interface implementation 2. More tests for parser and ast Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Bug with filter matcher Its wierd bug. Not handing string() method breaks the parser :( Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Split Line and Label filter to different struct. It got out of hands to handle both in same struct. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add basic doc for IP matcher Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add docs feedback from Karen * Label filter add patter error report * PR remarks 1. Indendation fix 2. Remove string() interface for ip label filte * Linter happy * PR remaks * Fix ip label matcher string() method if pattern is invalid * Fix label filter tests * Have local copy of `pattern` in label filter to use in string() * Rebase conflicts * ast.go fixes	5 years ago