1112 Commits (3c16658ee07e92b9271d2c42f93add85949eb3ae)

Author	SHA1	Message	Date
Dylan Guedes	3c16658ee0	Loki: Append loopback to ingester net interface default list (#4570) ... * Move LoopbackInterfaceName to its own package. - Create new package `util/net` and puts `LoopbackInterfaceName` there - Modify distributor_test to use `util/net` package * By default, append loopback to ingester interfaces. - Append the loopback net interface to the list of net interfaces used by the ingester ring. The append only occurs if the ingester net interfaces are default - Add CHANGELOG entry * Test ring net interface behavior. - Add tests to check the current net interface name resolution. The tests checks if the ingester ring net interface names are correctly copied to other rings	4 years ago
Trevor Whitney	ed025e0cd6	Change a few default config values and improve application of common storage config (#4543) ... * further config simplifications and better defaults * default ruler api to on * register defaults for storage configs in the common config (ie. signature version for S3) * enable ingester wal by default * disable chunk retries by default * allow common path prefix to include or omit trailing slash * allow partial storage config to be defined for overriding things like bucket names per component * fix tests * add changelog and upgrade docs * explain impact of wal change in upgrading.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * rename add WithPrefix to RegisterFlags * fix (frontend OR worker) and scheduler boot order when both are running (#4545) Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Sandeep Sukhani	b4bd96073c	validate default limits config with other configs at startup (#4573) ... * validate default limits config with other configs at startup This should also parse and set the matchers in default stream retention config * directly set matchers in stream retention instead of building the whole struct again	4 years ago
Dylan Guedes	331fa6d7b0	Docs: Add virtual targets docs (#4565) ... * Add CHANGELOG entry and updated modes diagram. * Update documentation with the new hybrid mode. - Update targets with new 'read' and 'write' options - Add new docker-compose yaml that exemplifies how to use new 'read' and 'write' targets * Add missing targets to configuration docs. - Missing ones: index-gateway, ingester-querier, query-scheduler	4 years ago
Cyril Tovena	165ff31e2d	Fixes the Series function to hanlde properly sharding. (#4563) ... Before they would try to match the injected label but this label is only used by the store and not returned in the chunks causing all chunks to be filtered out. Fixes #4547 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ea8e0987b3	Fixes an edge case in the batch chunk iterator. (#4550) ... * no-cache Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * found a bug to fix. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a edge case in the batch chunk iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	47336cdbe6	Fixes a panic in the labels API when no parameters are supplied. (#4554) ... * Fixes the a panic in the labels API when no parameters are supplied. We were supporting this before. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove bad test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	a27aec37cf	only exports tenant limits which differ from defaults and export defa… (#4542) ... * only exports tenant limits which differ from defaults and export defaults under `loki_overrides_defaults` metric. * docs & changelog	4 years ago
Ed Welch	6853c387ed	Loki: Apply the ingester ring config to all other rings (distributor, ruler, query-scheduler) (#4546) ... * apply the ingester ring config to all other rings in ApplyDynamicConfig * update the test, only verify ingester config wasn't manipulated by the step to apply the common memberlist config when join_members is defined.	4 years ago
Trevor Whitney	2864a4f728	enable virtual read write targets (#4498) ... Co-Authored-By: Mauro Stettler <mauro.stettler@gmail.com> Co-authored-by: Mauro Stettler <mauro.stettler@gmail.com>	4 years ago
李国忠	6fcd02bad4	[querier] s3: add getObject retry (#4453) ... * [querier] s3: add getObject retry * [querier] s3: add getObject retry * [querier] s3: add getObject retry #4452 * [querier] s3: add getObject retry * [querier] s3: add getObject retry #4453 * [querier] s3: add getObject retry * [querier] s3: add getObject retry #4453 * [querier] s3: add getObject retry #4453 * [querier] s3: add getObject retry #4453 * [querier] s3: add getObject retry Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Dylan Guedes	77ea11724b	Loki: Enable FIFO cache by default (#4519) ... * Set default FIFO cache with reasonable defaults. Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * Remove FIFO cache feature flag. - We noticed that we always want that behavior, so the logic is better implemented in the `ApplyDinamicConfig` method - Update documentation * Move FIFO cache default behavior. - Move FIFO cache default behavior implementation to ApplyDinamicConfig * Add tests for the new FIFO cache default behavior. - The tests tests the behavior of the four existing caches (query results, query index, write dedupe, and chunk results) - Add changelog entry - Fix lint issues regarding import order - Revert moving of EnableFifoCache check * Apply suggestions from code review - Change `EnableFifoCache` description to mention that it is only applied for chunk and query results cache Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Mauro Stettler <mauro.stettler@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
JordanRushing	93507a00ed	Add quick nil check in TenantLimits for runtime_config (#4531)	4 years ago
Owen Diehl	91eba4fb6b	correctly sets subservicesWatcher on scheduler (#4529)	4 years ago
JordanRushing	ca67292b54	Introduce `overrides-exporter` module to Loki (#4520) ... * Introduce overrides-exporter module from cortex into loki; add some test configs for local dev * Fix panic by adding server dependency to overrides-exporter module * Add tests for limits exporter * Remove dummy configs for testing * Add more limits to export * Add more limits; move mockTenantLimits to exporter_test.go * Introduce overrides-exporter module from cortex into loki; add some test configs for local dev * Fix panic by adding server dependency to overrides-exporter module * Add tests for limits exporter * Remove dummy configs for testing * Add more limits to export * Add more limits; move mockTenantLimits to exporter_test.go * Use reflection; add more limits to export * Fix import to satisfy linter * refactors TenantLimits for clarity & efficiency. Also fixes tenantLimits nil check * Fix breaking test with nil check in runtime_config Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	aec00c2507	safely checks read ring for potentially nil scheduler (#4525)	4 years ago
Trevor Whitney	6f28a5dc70	Apply better defaults when boltdb shipper is being used (#4508) ... * changes from PR review * rename object_store -> storage * add local/filesytem support to common object storage config * add better defaults for boltdb shipper * guard against missing schema configs in dynamic config application * fix alphabetical order of storage configs * return an error when more than one storage config is provided	4 years ago
Ed Welch	2b5f3000b6	Loki: Add a ring to the query scheduler to allow discovery via the ring as an alternative to DNS (#4424) ... * Fork the frontend and scheduler so we can add support for discovery via the ring. * Add a ring to the query scheduler which is then used by the frontend worker and querier workers to find the scheduler address as an alternative to using DNS * update some of the forked code to use dskit grpcclient and grpcutil since #4312 was merged after the original fork was created. * remove query scheduler dependency from querier Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * migrate logging package and make linter happy Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add SafeReadRing for instances when scheduler is not enabled on same instance * Doc changes from code review A few doc fixes Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * go mod tidy on go-kit dep * update changelog * remove ReadRing in favor of SafeReadRing Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Trevor Whitney	2427fab32d	Configuration: add a common config section for object storage (#4473) ... * add common config for object storage Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * wip Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add test to compactor config override Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * update changelog Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * changes from PR review * rename object_store -> storage * add local/filesytem support to common object storage config	4 years ago
Ed Welch	d8e6debaa5	Loki: Fix bug where items are returned to a sync.Pool incorrectly (#4518) ... * Make sure calls to the SampleIteratorWithClose Close() function are idempotent. * the linter hates my fun error messages	4 years ago
Garrett	a046d79261	make LogQL syntax scope from private to public (#4446) ... * make LogQL syntax scope from private to public The follow-up for https://github.com/grafana/loki/pull/3991, it makes using LogQL syntax outside possible, it also makes LogRange implements Expr interface. * make constructor also public * Revert "make constructor also public" This reverts commit ab9d0a942c.	4 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484) ... * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Owen Diehl	7832783b1c	always expoe loki_build_info (#4482)	4 years ago
Owen Diehl	3fb8f3d3c6	restores for state at seconds(now-forDuration) (#4479) ... * restores for state at seconds(now-forDuration) * bump ci	4 years ago
Mauro Stettler	4b59556342	Update cortex to newer version (#4478) ... * update cortex and thanos to newer version * pin k8s.io/apimachinery * fixes for newer libs * add missing method to mock wal data collector * fixing test by using different gokit level pkg * make check-mod	4 years ago
Dylan Guedes	6498c8cb13	Loki: Change default limits to common values (#4415) ... * Change some default limits to reasonable values. - These new limits protect users from overwhelming their cluster with ingestion load - The new limits are: * ingestion_rate_strategy: global <- currently defaults to local * max_global_streams_per_user: 5000 <- current default is no limit * max_query_length: 721h <- current default is no limit * max_query_parallelism: 32 <- current default is 14 * max_streams_per_user: 0 <- current default is 10000 * reject_old_samples: true <-- current default is false * reject_old_samples_max_age: 168h <-- current default is 336h * Remove parameters from examples - They are not necessary anymore because of the new default values * Add new entries to CHANGELOG and upgrading guide * Apply suggestions from code review Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> * Change changelog mark entry from '-' to '*' Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Dylan Guedes	d3d63e1778	Loki: Override distributor's default ring KV store (#4440) ... * Override default KV store for the compactor. - Currently, the default KVstore is consul. This changes it to inmemory to make it easier for new users to run the project without dependencies. - Refactor default flag value tests. Now, it will check for a populated map instead of checking for flags during iteration. * Move distributor flag override to RegisterFlags. * Add missing CHANGELOG and upgrading guide entries.	4 years ago
Trevor Whitney	7118dc3272	Change default values for two GRPC setting we have to set so the queriers can connect to a frontend or scheduler (#4435) ... * change gprc server defaults Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add docs and changelog Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Trevor Whitney	81d0f1d533	Add more tests around config parsing changes from common config PR (#4433) ... * add test to demonstrate bug Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * a totally empty yaml file causes weird behavior Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add default unmarshal test for test config wrapper Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add additional tests around default unmarshal to confirm bug is not there Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * clean up tests Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * check errors in test Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Kaviraj	2e1279d996	Fix return values of Matrix and Vector during query range in QueryShardingMiddleware (#4457) ... * Fix matrics `result` type being `null` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add test for vector type as well Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Arve Knudsen	cdde2cf050	Chore: Use dskit/grpcclient (#4312)	4 years ago
Dylan Guedes	ab10bc69d7	Loki: Change how push API checks for contentType (#4443) ... * Change how push API checks for contentType - Currently, the ContentType is expected to be exactly "application/json". That's a problem because although a "application/json; charset=utf-8" is considered a valid JSON content type, it will be seen as an error. * Use mime.ParseMediaType to parse contentType. - mime.ParseMediaType is more reliable than strings.Contains because if a wrong contentType (such as "application/jsonnn") is given, we still accept it - mime.ParseMediaType has support for contentType parameters. This way, valid contentTypes such as "application/json; charset=utf-8" will not be rejected * Add changelog entry. * Change body in test to avoid lint issues. - Without this change, the lint fails because gzipString would be called with always the same input. Full error: ``` `gzipString` - `source` always receives ``{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1570818238000000000", "fizzbuzz" ] ] }]}` ("{\"streams\": [{ \"stream\": { \"foo\": \"bar2\" }, \"values\": [ [ ...) ` (unparam) ```	4 years ago
Danny Kopping	d265e1d206	Do not clear HTTP client config (#4437) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Trevor Whitney	b07920cb16	auto-apply memberlist ring config when join_members provided (#4400) ... * auto-apply memberlist ring config when join_members provided Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * move "memberlist" to a const Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add comment for memberlist config auto-apply, update changelog Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add docs Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Owen Diehl	1921c3d428	tests checkpoints immediately and gives more of a time buffer (#4432)	4 years ago
Danny Kopping	b99be1d310	Ruler: Overwrite instead of merge remote-write headers (#4431) ... * Overwrite instead of merge remote-write headers Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Allowing for clearing of headers when empty map is given Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Danny Kopping	a961612436	Ruler: Refactoring remote-write config overrides (#4429) ... * Refactoring remote-write config overrides Replacing mergo.Merge because it cannot distinguish between a nil and an empty slice, which is needed to overwrite relabel configs Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Simplifying condition Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Merging headers instead of overwriting blindly Adding extra check for header smuggling Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Unexporting OverwriteMarshalingStringMap field, preferring constructor Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Owen Diehl	415d4b03fa	Safe per tenant overrides loading (#4421) ... * safe per tenant overrides loading * exports submap for OverwriteMarshalingStringMap	4 years ago
Garrett	4417340089	add on and ignoring clauses in binOpExpr (#4391) ... * add on and ignoring clauses in binOpExpr * add on/ignoring new operator shardmapper unit test * add doc for on and ignoring keywords * resolve comments * use a scoped error to replace mutate stepEvaluator via type switching * change VectorMatching Include from empty string array to nil * prohibit sharding when we're changing the label groupings such as on or ignoring * add unit test to cover multiple matches for labels error * fix expr.y format issue * resolve comments * reformat on and ignoring keywords doc * add sharding unit test for on and ignoring keywords * resolve comments: add a sentence for each example to help docs readers to understand the larger picture Co-authored-by: garrettlish <garrettlish@163.com>	4 years ago
Ed Welch	d5ec714ae0	Loki: Bug: frontend waiting on results which would never come (#4411) ... * We weren't also checking if the context was canceled while waiting for query results leading to goroutines waiting forever for results that would never come... * return error from context upstream if the context is canceled	4 years ago
Ed Welch	11e215d79f	Loki: Common Config (#4347) ... * refactor the pkg/util/cfg package to simplify and make it accessable outside of this package. create a pkg/cfg which handles building the Loki config, this is created separately from pkg/util (which is Apache 2 licensed) to maintain proper separations of AGPL and Apache 2 code Update the main.go for several of the apps with the refactoring * refactor how dynamic/common config is loaded Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Add common config logic for path prefix Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Missed a file, update common config with prefix property Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * fix test, remove loki from args Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Clean up test names Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * fix typo in filename, make linter happy Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Update pkg/loki/config_wrapper.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/loki/config_wrapper.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/loki/config_wrapper_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * remove temp files in test Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Callum Styan	25163470b0	Improve error message for stream rate limit. (#4207) ... * Improve error message for stream rate limit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Move stream rate limit error message to validation package. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Use ByteSize for number values in stream rate limit error so we can print human readable rate limits and log line byte lengths. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix lint issues. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Don't need to type assert twice. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Compare errors contents instead of doing error equals comparison directly since the rate limit error is an RPC error. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Don't call Limit on hot path just to log the value in errors. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Make coment about rate limit clearer. Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Trevor Whitney	de0c469161	default chunk target size to ~1MB~ 1.5MB (#4388) ... * default chunk target size to 1MB Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * bump up to 1.5MB Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Hiroki Sakamoto	502750deca	Add 'loki_ingester_memory_chunks' when recovery from wal (#4272)	4 years ago
JordanRushing	5e8a204480	Add recovery middleware to Ingester; re-add recovery middleware to Querier when not running in standalone mode (#4349) ... * Add recovery middleware to ingester; re-add recovery middleware to querier when not running in standalone mode * Re-add gRPC recovery middleware * Break out gRPC recovery middleware from setupAuthMiddleware() into new function setupGRPCRecoveryMiddleware()	4 years ago
Owen Diehl	fc0f98b802	removes deprecated duplicate per stream rate limit fields (#4367)	4 years ago
Danny Kopping	a1aaec1658	Create custom struct that eases json/yaml unmarshaling of complex relabel.Config struct (#4364) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Danny Kopping	8a546b31a9	Ruler: per-tenant WAL (#4344) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Danny Kopping	0f2f42de01	Runtime config: do not validate nil limits (#4251) ... * Do not validate nil limits If an overrides file has an entry for a tenant without any limits, this will cause a panic Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Warn when an empty limit is skipped Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Arve Knudsen	79b95d976a	Chore: Use dskit/limiter (#4327) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago