525 Commits (37d65027fe0cd3dc3fcb57eea7ff20fcbfdb2eec)

Author	SHA1	Message	Date
Sandeep Sukhani	b2d0481cbe	deletion: fix log deletion with line filters (#8151)	3 years ago
Owen Diehl	944b488bd2	logs more metadata when failing to recover head from WAL in TSDB (#8250)	3 years ago
Sandeep Sukhani	96a4fc1409	index-store: fix indexing of chunks overlapping multiple schemas (#8251)	3 years ago
Bryan Boreham	d2281a7b6e	Benchmark_store_SelectSample: stop accumulating results (#8242)	3 years ago
Bryan Boreham	321ee92fd2	stores/series: always return entries to pool (#8245)	3 years ago
Bryan Boreham	15b382294e	Remove dependency on thanos-io/thanos (#8227) ... **What this PR does / why we need it**: This avoids complications trying to match up versions between Prometheus and Thanos. **Special notes for your reviewer**: I don't know why the dependency on `github.com/efficientgo/e2e` has appeared in go.mod, but it doesn't seem to change what is in vendor. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - NA Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Ed Welch	c8760f31e2	Loki: Remove validation of matchers when performing label values lookup. (#8224) ... the curren validation fails on != style matchers when they are the only matchers present, requiring at least one `=` matcher. However for looking up label values in TSDB, by design it uses a `desired_label != ""` style query, as such it should be allowed to also include these as matchers. Additional matchers on this Label Value queries typically only exist with LBAC headers adding additional labels. Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Dimitar Dimitrov	6ea5a8e497	Update vendored dskit (#8155) ... **What this PR does / why we need it**: The Changelog diff between the two commits is 3e308a4944...7242706251 ``` * [CHANGE] Cache: Switch Memcached backend to use `github.com/grafana/gomemcache` repository instead of `github.com/bradfitz/gomemcache`. #248 * [ENHANCEMENT] Add cache, gate and stringsutil packages. #239 * [ENHANCEMENT] Cache: add Redis support. #245 * [ENHANCEMENT] Cache: Add the ability to use a custom memory allocator for cache results. #249 * [BUGFIX] Memberlist: fix crash when methods from `memberlist.Delegate` interface are called on `*memberlist.KV` before the service is fully started. #244 ``` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](../CONTRIBUTING.md) guide (**required**) - N/A Documentation added - N/A Tests updated - N/A `CHANGELOG.md` updated - N/A Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Owen Diehl	07487cd89d	fixes bug with queryIngesterWithin logic in asyncStore ingester stats… (#8145) ... Fixes a previous mistake in the logic calculating when to skip querying ingesters in the async store Statistics method. Notably `through.After` should be `through.Before` when skipping querying ingesters as that's when there's no overlap with the `query-ingesters-within` period: ```go // OLD CODE BELOW if a.queryIngestersWithin != 0 { // don't query ingesters if the query does not overlap with queryIngestersWithin. if !through.After(model.Now().Add(-a.queryIngestersWithin)) { // <----- should be through.Before return a.Store.Stats(ctx, userID, from, through, matchers...) } } ``` I discovered the problem while debugging querier OOMs during a boltdb-shipper -> tsdb migration and ultimately found this happened under the following circumstances: * Queries over high volumes of _recent_ data wouldn't query ingesters for index metadata * Without index metadata, we only checked storage metadata * There is a delay before we ship the index to storage, meaning we don't see it if ingesters are skipped * Calculating ideal shard factors without recent data for queries that only touch recent data _dramatically_ underestimates the desired shard factor * Queries aren't split enough and get scheduled onto too few querier replicas * They oom. We had some fun examples like a single replica trying to download 419,000 chunks/query To be clear, this is still a hypothesis, but a plausible one, especially after finding the boolean logic error this PR fixes. Instead of changing this one line, I took the opportunity to refactor this into a shared utility used by our other `GetChunkRefs` method which is already tested, ensuring the logic works as expected. I also added some more logging visibility into this code so we can understand what the difference is when querying statistics from storage vs ingesters. Finally, I added a helper to prepare our `Stats` objects to be logged which is now used in a few places.	3 years ago
Nils	e4fbd3c949	Loki: Add support for Azure Workload Identity authentication (#7540)	3 years ago
Sandeep Sukhani	dfff86f1cd	skip compacting tables that do not fit in start and end date of desired schema config (#8127) ... Before compacting tables, we look up the schema config for them. If we do not find a schema config for a table, we skip them. However, the schema lookup code was considering the start time of the schema and not the end time. This can cause the compactor to try compacting tables with the same prefix as one of the previous schemas, but the table interval falls into one of the subsequent schemas with a different config. This PR fixes the issue and simplifies the code by reusing existing schema lookup code. I have also updated test cases to verify the fix works.	3 years ago
Ashwanth	b3e038c7a8	config_wrapper: Add support for named stores (#7946) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: This PR adds support for named stores which allows users to define multiple object store configurations for a storage type. Named stores can be referred to from other sections of the config similar to how provider type is being used `i.e aws, gcs...`. Named store reference should be of the format: `<store_type>.<key_name>` Below is an example configuration with two named stores defined for aws `store-1, store-2`. Schema config is referring to one of them as `aws.store-1` ``` storage_config: aws: endpoint: s3://common-bucket region: us-east1 access_key_id: abc123 secret_access_key: def789 named_stores: aws: store-1: endpoint: s3://foo-bucket region: us-west1 access_key_id: 123abc secret_access_key: 789def store-2: endpoint: s3://bar-bucket region: us-west2 access_key_id: 456def secret_access_key: 789abc schema_config: configs: - from: "2020-07-31" index: period: 24h prefix: loki_prod_index_ object_store: aws.store-1 schema: v11 store: boltdb-shipper ``` **Which issue(s) this PR fixes**: Fixes #7276 **Special notes for your reviewer**: Configuration defined in common storage config has no effect on the named stores. Common storage config only overrides storageConfig if it's not explicitly set - same behavior as before. This should be a non-intrusive change. Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Ashwanth	dbf09246d5	Avoid failing tests on gRPC server shutdown errors (#8060)	3 years ago
Ashwanth	dc5cc4c494	store: write overlapping chunks to multiple stores (#7988) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: When a chunk falls on the period boundary it should be written to both the stores as it gets referred to by the indexes on both sides. But we currently skip the writes for subsequent periods if the chunk is found in the cache. This PR ensures that overlapping chunks get written to both the stores **Which issue(s) this PR fixes**: Fixes #7847 **Special notes for your reviewer**: Downside of this approach is that we end up writing the overlapping chunk multiple times, once from each instance in the replica set. But these duplicate writes would be infrequent given this occurs only on schema changes **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [X] Tests updated - [x] `CHANGELOG.md` updated Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Susana Ferreira	d086c237df	Add missing period_config root block to doc generator (#7981) ... **What this PR does / why we need it**: Update doc generator tool to validate if slice element is a root block. Add missing `period_config` root block to doc generator tool. **Special notes for your reviewer**: **Checklist** - [ ] Documentation updated	3 years ago
Danny Kopping	a4f306399a	Add store & cache download statistics (#7982)	3 years ago
Irina	10b869c82e	Move to fork github.com/grafana/gomemcache (#7936)	4 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916) ... **What this PR does / why we need it**: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](5550cd65ec/docs/sources/configuration/_index.md) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. **NOTE:** this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/83 **Special notes for your reviewer**: Files: * [docs/sources/configuration/index.template](5550cd65ec/docs/sources/configuration/index.template): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](c32e5d0acb/docs/sources/configuration/_index.md): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](5550cd65ec/tools/doc-generator) directory with documentation generation tool. **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	4 years ago
Robert Fratto	85392a9728	Update Prometheus dependency to latest release (v2.40.4) (#7826) ... Closes #7811, which is needed for Grafana Agent to update to v2.40 and add support for native histograms. I did not add support for native histograms to Loki, sorry :)	4 years ago
Kaviraj Kanagaraj	25f4dda811	fix(memcached): use `default` branch avoid writing to closed chan (#7833) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> **What this PR does / why we need it**: Follow up to https://github.com/grafana/loki/pull/7817 This PR uses `default` branch instead of `inputCh <- work` to make sure we are writing to closed `inputCh`. Gist is, `default` run [only when none of the branch is ready](https://go.dev/tour/concurrency/6). which makes more sense rather than to have `inputCh <- work` (writing to closed channel on the branch condition) These can be explained by these two tiny snippets. * [with `default`](https://go.dev/play/p/-FspbTZd20I) * [without `default`](https://go.dev/play/p/Ag4WznOaEq0) **Which issue(s) this PR fixes**: Fixes #NA **Special notes for your reviewer**: We already have test `TestMemcached_fetchKeysBatched` to catch this behaviour. In fact this test caught this probabilistic behaviour in some CI failures. [here ](https://drone.grafana.net/grafana/loki/17853/3/4) and[ here](https://drone.grafana.net/grafana/loki/17854/3/4) Thanks @DylanGuedes for catching this CI failures. **Checklist** Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Kaviraj Kanagaraj	4efdc131be	fix(memcached): panic on `send on closed channel`. (#7817)	4 years ago
Sandeep Sukhani	89d81020ce	fix lint issues from PR 7804 (#7814) ... **What this PR does / why we need it**: I had enabled auto-merge in PR #7804, but somehow it still merged the PR without all the checks passing. This PR fixes the failing lint and tests.	4 years ago
Sandeep Sukhani	1410808ee9	use grpc for communicating with compactor for query time filtering of data requested for deletion (#7804) ... **What this PR does / why we need it**: Add grpc support to compactor for getting delete requests and gen number for query time filtering. Since these requests are internal to Loki, it would be good to use grpc instead of HTTP same as all the internal requests we do in Loki. I have added a new config for accepting the grpc address of the compactor. I tried having just the existing config and detecting if it is a grpc server, but it was hard to do it reliably, considering the different deployment modes we support. I think it is safe to keep it the same and eventually deprecate the existing config. **Checklist** - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated	4 years ago
Sandeep Sukhani	d3e42959a5	fix bugs in processing delete requests with line filters and add proper tests (#7720) ... **What this PR does / why we need it**: It fixes the following issues: * Whole chunks being deleted with a line filter were skipped. * Chunks partially covered by delete requests with a line filter were not properly processed. The part that **was not covered** by the delete request was rewritten with a line filter, while the part that **was covered** by the delete request was completely deleted. The issue went unnoticed because it seems we continue to do query time filtering of data even after the delete requests are processed. **Which issue(s) this PR fixes**: Fixes #7590 **Special notes for your reviewer**: If multiple delete requests with a line filter touch the same chunk, then I am considering just the first request that marks it for deletion to keep this PR simple. I have left a to-do to see if we can make it work efficiently. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated	4 years ago
李国忠	0458dc8cbb	fix: Hide this error message when ctx timeout occurs in s3.getObject (#6360) ... **What this PR does / why we need it**: ``` level=error ts=2022-01-24T10:57:05.273404635Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: context canceled" level=error ts=2022-01-24T10:57:05.273434964Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: context canceled" level=warn ts=2022-01-24T10:57:05.273457211Z caller=grpc_logging.go:55 method=/logproto.Querier/Query duration=16.524312361s err="failed to get s3 object: RequestCanceled: request context canceled\ncaused by: context canceled" msg="gRPC\n" level=error ts=2022-01-24T10:57:05.273486368Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: context canceled" level=error ts=2022-01-24T10:57:05.273786914Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: RequestCanceled: request context canceled\ncaused by: context canceled" level=error ts=2022-01-24T10:57:05.274246531Z caller=batch.go:699 msg="error fetching chunks" err="failed to get s3 object: RequestCanceled: request context canceled\ncaused by: context canceled" level=warn ts=2022-01-24T10:57:05.274385302Z caller=grpc_logging.go:55 method=/logproto.Querier/Query duration=29.542213769s err="failed to get s3 object: RequestCanceled: request context canceled\ncaused by: context canceled" msg="gRPC\n" ``` When logql timeout, do not print these logs. Since we retry, we don't want to show intermittent timeouts, and we only care about the last error. **Which issue(s) this PR fixes**: Fixes [<5221>](https://github.com/grafana/loki/issues/5221)	4 years ago
李国忠	b3664ff255	[fix] cassandra: reconnection when k8s cassandra pods restart (#6725) ... **What this PR does / why we need it**: level=error ts=2022-07-20T04:07:11.881370946Z caller=flush.go:146 org_id=166256_8sxv2 msg="failed to flush user" err="store put chunk: gocql: no hosts available in the pool" The restart of cassandra （on k8s） caused the production accident that loki had to restart. It happened 4 times in our production environment in only half a year. I hope this problem can be fixed. The code of this PR is not particularly complete, but this PR seems to be available. If this PR does not merge, I also hope that other contributors in the loki community can propose another PR to fix this problem. The restart of cassandra that caused loki to be unavailable is already a conclusion recorded in our JIRA. **Which issue(s) this PR fixes**: Fixes #1821 #7140	4 years ago
Wu Zhiyuan	3bcc7ccd5a	use fifocache.duration instead of fifocache.interval in warn deprecated (#7609) ... **What this PR does / why we need it**: I configured fifo cache with `validity` option (which is [deprecated](b92f113cb0/docs/sources/configuration/_index.md?plain=1#L2004) in v2.6.0, `fifocache.duration` as an flag) and loki print that: ``` level=warn ts=2022-07-08T17:01:30.134251284Z caller=fifo_cache.go:125 msg="running with DEPRECATED flag fifocache.interval, use fifocache.ttl instead" cache=frontend.fifocache ``` However, there is no flag `fifocache.interval`. As #5148 changed, the deprecated flag should be `fifocache.duration`, not `fifocache.interval`.	4 years ago
Danny Kopping	d32f3f7b68	Querier: remove expensive & low-value debug log line (#7532) ... Removes a low-value debug log line that, in one recent example, can logs tens of thousands of times in a short span of time. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Dylan Guedes	3c792294d2	Loki: Fix index_set log message (#7538) ... As of now, this log message isn't correctly showing the list of files (instead it logs the value as "unsupported value type").	4 years ago
Periklis Tsirakidis	e0a7b28a61	Add single compactor http client for delete and gennumber clients (#7453)	4 years ago
Owen Diehl	6b4563cd53	Prevent IndexShipper from leaking mutex accesses. (#7548) ... # What Inverts control in the `index-shipper` so consumers don't manage accesses to mutexes. This prevents deadlocks and no longer requires consumers have internal knowledge of the `index-shipper` itself. This PR reverts a few previous attempts. Look at the last commit to see the net-additions. # Background: TSDB previously didn't use mmap, which meant we could freely pass around TSDBs as `[]byte`. Since they were immutable, we didn't need to know when an underlying index was cleaned up/closed; this coordination was unnecessary and the garbage collector would clean up dangling memory for us. After introducing TSDB, we needed to control access to avoid use-after-free bugs. A few attempts at controlling this became increasingly complex & error prone. Instead of requiring _consumers_ to prevent deadlocks on index-shipper internals, it makes more sense that references to the index-shipper's resources should not leak outside the `ForEach` functions that give temporary access to them. This _inverts control_ from the consumer (code which queries the indices) to the producer (the index-shipper). Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	4 years ago
Sandeep Sukhani	bb2927172a	tsdb post compaction cleanup (#7502) ... **What this PR does / why we need it**: Previously when we were reading the whole tsdb index in memory, there was not much cleanup required after we were done with the index. Now that we are mmaping the tsdb index, we need to close the mmap. I had missed calling `index.Close()` on the compacted-away multi-tenant index. This causes the compactor to retain the files on disk, leading to the compactor running away from disk space. This PR takes care of it. While doing the changes, I noticed that some of the tests failed due to `LabelValues` tsdb implementation returning strings valid only until the underlying byte slice stays valid. Since the underlying byte slice could become invalid when we close the tsdb, we should return a copy instead of yolostring. This can also happen on the query path when a sync closes the index before we get to return the response. This PR also fixes that.	4 years ago
Owen Diehl	70f6bf2e45	Index gw lock bugfix (#7497) ... While I'm still in favor of a longer refactor to address the complexity described in https://github.com/grafana/loki/pull/7490, I'm opening this PR as a shorter, more immediate alternative that should fix the deadlocks. This also reduces the number of total lock calls in many cases as we lock to act over entire groups rather than individual items.	4 years ago
Travis Patterson	b4ed595461	Remove groupcache (#7126) ... As discussed during the caching retro, this PR removes the experimental distributed cache based on Mailgun's fork of groupcache. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425)	4 years ago
Periklis Tsirakidis	4176de17d1	Add compactor delete client tls client config (#7426)	4 years ago
Owen Diehl	b59277b472	Revert "Revert "Update Thanos dependency to latest version"" (#7372) ... Reverts grafana/loki#7371 PR re re-enable Thanos dependency bump.	4 years ago
李国忠	435c055488	log panic full stack in boltdb recover block (#7410)	4 years ago
Sandeep Sukhani	18da26e668	use a channel to signal completion of index read operation for tsdb and boltdb shipper (#7381) ... **What this PR does / why we need it**: In PR #7367, I was relying on cancellation of context to detect if the index read operation has finished or not, but it is error prone since a cancelled or timed-out query could cancel the context while we are still reading the index. Since `tsdb` doesn't can't check the context for each read operation, we could still panic in this scenario. I am adding a channel where the consumer of the index would explicitly signal the completion of the read operation to the `indexshipper` **Checklist** - [x] Tests updated	4 years ago
Owen Diehl	8331785a23	Revert "Update Thanos dependency to latest version" (#7371) ... Reverts grafana/loki#7348 I expect we'll merge this again in the future, but we'll need some more time for downstream projects to be ready for this change.	4 years ago
Sandeep Sukhani	bbe68802d1	lock boltdb and tsdb index while queries are in process (#7367) ... **What this PR does / why we need it**: We noticed ingesters and index-gateways panicking when using `tsdb` as an index store with the latest build. This is caused by closing `mmap` while the index queries are still being processed. This PR takes care of it by locking the index until we are done querying the index. **Checklist** - [x] Tests updated	4 years ago
Owen Diehl	f4e2cf1993	TSDB explicit fingerprint (#7362) ... This PR does two things: * Includes the series fingerprint in TSDB WALs. Doing this ensures we recover fingerprint used in chunk addresses after restart even if loki is upgraded and the label hashing algorithm changes. * Removes much of the extra hashing in the TSDB write path + manager. Occasionally, the fingerprint may be different than the hash of labels. Historically this has happened in a few places: * When the `__name__="logs"` label was injected for non-tsdb indices (not applicable to TSDB) * When building a multitenant tsdb index and adding a synthetic `__loki_tenant__` label * In a previous bug(https://github.com/grafana/loki/pull/7355) when empty label values altered calculated fingerprints. Explicitly storing/retrieving the desired fingerprint throughout helps avoid calculating it in error.	4 years ago
Owen Diehl	289190ee26	removes unused tsdb files (#7353) ... These were introduced in an earlier revision of the TSDB work, but are unused now.	4 years ago
Robert Fratto	c657857e95	Update Thanos dependency to latest version (#7348)	4 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340) ... I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	4 years ago
Sandeep Sukhani	10d3fa0029	mmap tsdb index file while opening it for querying (#7338) ... **What this PR does / why we need it**: In PR #7321, we re-introduced mmap for reading tsdb index, but it does not cover all the paths where we open the index. This PR takes care of it. I have also removed the code for decompressing the index file since we always get a decompressed file from the `indexshipper`.	4 years ago
Danny Kopping	355175ff89	Upgrade to Go 1.19 (#7243) ... Upgrade Loki and all related components to Go 1.19.2.	4 years ago
Danny Kopping	6bf5b5d8c1	Use a line-buffered logger to deamplify write syscalls (#6954) ... We initialise a global logger in `pkg/util/log/log.go` and use it extensively throughout the Loki codebase. Every time we write a log message, a `write` syscall is invoked. Syscalls are problematic because they transition the process from userspace to kernelspace, which means: - a context-switch is incurred, which is inherently expensive ([1-2 microseconds](https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/)) - the goroutine executing the code is **blocked** - the underlying OS thread (_M_ in the go scheduler model) is **also blocked** - the goroutine has to be rescheduled once the syscall exits - the go scheduler may need to spawn additional OS threads if all are blocked in syscalls - which can also be expensive This change introduces the use of a line-buffered logger. It has a buffer of [256](https://gist.github.com/dannykopping/0704db32c0b08751d1d2494efaa734c2) entries, and once that buffer is filled it will flush to disk. However, a situation will arise in which that buffer remains somewhat empty for a period of time, so there is a periodic flush mechanism, configured to flush every 100ms. There is also a preallocated bytes slice of 10MB which is reused, to avoid excessive slice resizing & garbage collection. This does mean that we could lose up to 256 log messages in case of an ungraceful termination of the process, but this would need to be precisely timed within the 100ms flushes - in other words, the likelihood is low, and generally we shouldn't `kill -9` any Loki process.	4 years ago
Owen Diehl	2bfe626605	reintroduce mmap for tsdb indices (#7321) ... This reverts a [previous decision](b45efd4c2d (diff-e3f5966d58063d2852aa51899b5aae14e65bb05f29fec7302eb982c3d18c60ab)) which removed `mmap` support when loading TSDB indices. I did this primarily for simplicity, but since then have reneged on my original choice for two reasons: 1) Retention in Loki is variable and can be very long. Requiring we keep all loaded tsdb indices in memory creates a cost burden in terms of memory required to hold all queried indices at once. 2) A high number of tenants also exacerbates the memory claims which mmap helps alleviate.	4 years ago
Owen Diehl	2bd9c91564	Bugfix include cache generations on querier response (#7300) ... We check the cache number (incremented by deletes) after executing a query with the value returned by the queriers which executed it. If they’re different, we know we can’t safely cache. However, we’re never actually returning a number from the queriers. This means that as soon as this cache number is incremented by a delete, it’ll disable the results cache for that tenant completely. I suspect this has been a latent bug since we forked cortex (goes back to missing some code from this [PR](https://github.com/cortexproject/cortex/pull/2279) into cortex from @sandeepsukhani back in the day!). It just showed itself once we started using deletes.	4 years ago