688 Commits (2bc12fa2fb1ebf50f3c016b22ffa94deaeed65b9)

Author	SHA1	Message	Date
Dominik Schulz	182e17958a	Trivial spelling corrections and a small comment.	9 years ago
Fabian Reinartz	8fa18d564a	storage: enhance Querier interface usage ... This extracts Querier as an instantiateable and closeable object rather than just defining extending methods of the storage interface. This improves composability and allows abstracting query transactions, which can be useful for transaction-level caches, consistent data views, and encapsulating teardown.	9 years ago
beorn7	3f02e33e34	Re-add counting of evict chunk ops and decrementing NumMemChunks ... Also, modify test to expose the regression.	9 years ago
Julius Volz	ab870ac653	Clean up some doc comments	9 years ago
beorn7	719508752b	Re-add counting of evict chunk ops and decrementing NumMemChunks ... Also, modify test to expose the regression.	9 years ago
Julius Volz	cb02f017ee	Clean up some doc comments	9 years ago
Julius Volz	c212ef0326	Add Chunk.Utilization() methods ... When using the chunking code in other projects (both Weave Prism and ChronixDB ingester), you sometimes want to know how well you are utilizing your chunks when closing/storing them.	9 years ago
Julius Volz	c7932aa009	Remove gRPC leftovers in protobuf definitions	9 years ago
Maxim Ivanov	e6db9f8159	New fpsForLabelMatchers and seriesForLabelMatchers methods ... These more specific methods have replaced `metricForLabelMatchers` in cases where its `map[fingerprint]metric` result type was not necessary or was used as an intermediate step Avoids duplicated calls to `seriesForRange` from `QueryRange` and `QueryInstant` methods.	9 years ago
Brian Brazil	77605649a9	Add support for remote write relabelling. ... Switch back to a single remote writer, as we were only ever meant to have one and the relabel semantics are clearer that way.	9 years ago
Julius Volz	c9d4526428	Unpublish accidentally published series methods ... There were some more accidentally published methods of the memorySeries type which I didn't notice when reviewing https://github.com/prometheus/prometheus/pull/2011	9 years ago
Maxim Ivanov	4978a65495	Extract initial FP candidate build logic into candidateFPsForLabelMatchers method ... No functional changes otherwise	9 years ago
Maxim Ivanov	c048a0cde8	Add metrics to result after checking all matchers ... Should be marginally faster and somewhat more GC friendly	9 years ago
Maxim Ivanov	bedc0eda1f	Added BenchmarkQueryRange	9 years ago
Julius Volz	c25f0de5ae	Remove local.ZeroSample{,Pair}, use model definitions	9 years ago
Julius Volz	044ebce779	Review fixups.	9 years ago
Julius Volz	d30a3c7c0f	Fix accidental publishing of memorySeries.firstTime()	9 years ago
Julius Volz	ab80ced756	storage: separate chunk package, publish more names ... This is a followup to https://github.com/prometheus/prometheus/pull/2011. This publishes more of the methods and other names of the chunk code and moves the chunk code to its own package. There's some unavoidable ugliness: the chunk and chunkDesc metrics are used by both packages, so I had to move them to the chunk package. That isn't great, but I don't see how to do it better without a larger redesign of everything. Same for the evict requests and some other types.	9 years ago
beorn7	ca98382943	Avoid `defer` in seriesMap.get ... This is related to https://github.com/golang/go/issues/14939 . It's probably the only occurrence where it matters.	9 years ago
Matthew Campbell	67d76e3a5d	timeseries: store varbit encoded data into cassandra	9 years ago
Tom Wilkie	4520e12440	Add HTTP Basic Auth & TLS support to the generic write path. (#1957) ... * Add config, HTTP Basic Auth and TLS support to the generic write path. - Move generic write path configuration to the config file - Factor out config.TLSConfig -> tlf.Config translation - Support TLSConfig for generic remote storage - Rename Run to Start, and make it non-blocking. - Dedupe code in httputil for TLS config. - Make remote queue metrics global.	9 years ago
Julius Volz	c187308366	storage: Contextify storage interfaces. ... This is based on https://github.com/prometheus/prometheus/pull/1997. This adds contexts to the relevant Storage methods and already passes PromQL's new per-query context into the storage's query methods. The immediate motivation supporting multi-tenancy in Frankenstein, but this could also be used by Prometheus's normal local storage to support cancellations and timeouts at some point.	9 years ago
Maxim Ivanov	bdc53098fc	Avoid having contended mutexes on same cacheline ... CPUs have to serialise write access to a single cache line effectively reducing level of possible parallelism. Placing mutexes on different cache lines avoids this problem. Most gains will be seen on NUMA servers where CPU interconnect traffic is especially expensive Before: go test . -run none -bench BenchmarkFingerprintLocker BenchmarkFingerprintLockerParallel-4 2000000 932 ns/op BenchmarkFingerprintLockerSerial-4 30000000 49.6 ns/op After: go test . -run none -bench BenchmarkFingerprintLocker BenchmarkFingerprintLockerParallel-4 3000000 569 ns/op BenchmarkFingerprintLockerSerial-4 30000000 51.0 ns/op	9 years ago
Tom Wilkie	d83879210c	Switch back to protos over HTTP, instead of GRPC. ... My aim is to support the new grpc generic write path in Frankenstein. On the surface this seems easy - however I've hit a number of problems that make me think it might be better to not use grpc just yet. The explanation of the problems requires a little background. At weave, traffic to frankenstein need to go through a couple of services first, for SSL and to be authenticated. So traffic goes: internet -> frontend -> authfe -> frankenstein - The frontend is Nginx, and adds/removes SSL. Its done this way for legacy reasons, so the certs can be managed in one place, although eventually we imagine we'll merge it with authfe. All traffic from frontend is sent to authfe. - Authfe checks the auth tokens / cookie etc and then picks the service to forward the RPC to. - Frankenstein accepts the reads and does the right thing with them. First problem I hit was Nginx won't proxy http2 requests - it can accept them, but all calls downstream are http1 (see https://trac.nginx.org/nginx/ticket/923). This wasn't such a big deal, so it now looks like: internet --(grpc/http2)--> frontend --(grpc/http1)--> authfe --(grpc/http1)--> frankenstein Next problem was golang grpc server won't accept http1 requests (see https://groups.google.com/forum/#!topic/grpc-io/JnjCYGPMUms). It is possible to link a grpc server in with a normal go http mux, as long as the mux server is serving over SSL, as the golang http client & server won't do http2 over anything other than an SSL connection. This would require making all our service to service comms SSL. So I had a go a writing a grpc http1 server, and got pretty far. But is was a bit of a mess. So finally I thought I'd make a separate grpc frontend for this, running in parallel with the frontend/authfe combo on a different port - and first up I'd need a grpc reverse proxy. Ideally we'd have some nice, generic reverse proxy that only knew about a map from service names -> downstream service, and didn't need to decode & re-encode every request as it went through. It seems like this can't be done with golang's grpc library - see https://github.com/mwitkow/grpc-proxy/issues/1. And then I was surprised to find you can't do grpc from browsers! See http://www.grpc.io/faq/ - not important to us, but I'm starting to question why we decided to use grpc in the first place? It would seem we could have most of the benefits of grpc with protos over HTTP, and this wouldn't preclude moving to grpc when its a bit more mature? In fact, the grcp FAQ even admits as much: > Why is gRPC better than any binary blob over HTTP/2? > This is largely what gRPC is on the wire.	9 years ago
Tobias Schmidt	29ced0090f	Fix common english misspellings	9 years ago
Tobias Schmidt	e2c12dcdb5	Add missing error check in persistence test	9 years ago
Tobias Schmidt	8f3b62bfe4	Simplify struct initialization	9 years ago
Julius Volz	b24e5d63bc	Add noop local storage engine. ... This adds a flag -storage.local.engine which allows turning off local storage in Prometheus. Instead of adding if-conditions and nil checks to all parts of Prometheus that deal with Prometheus's local storage (including the web interface), disabling local storage simply means replacing the normal local storage with a noop version that throws samples away and returns empty query results. We also don't add the noop storage to the fanout appender to decrease internal overhead. Instead of returning empty results, an alternate behavior could be to return errors on any query that point out that the local storage is disabled. Not sure which one is more preferable, so I went with the empty result option for now.	9 years ago
Fabian Reinartz	22296dcb85	storage: clarify sample/matcher relation in docs	9 years ago
Fabian Reinartz	cc6f988a5e	storage: fix Querier interface documentation	9 years ago
Fabian Reinartz	7bd7e63f97	storage: fix struct alignment issue in test ... The uint64 `numCalls` ends up being not word-aligned on certain architectures, which makes atomic reads/writes panic.	9 years ago
nghialv	7655038184	fix typo	9 years ago
Tom Wilkie	d41d91388f	Update for new generic remote storage.	10 years ago
Tom Wilkie	a6931b71e8	Rationalise retrieval metrics so we have the state (success/failed) on both samples and batches, in a consistent fashion. ... Also, report total queue capacity of all queues, i.e. capacity * shards.	10 years ago
Tom Wilkie	ece12bff93	Shard/parrallelise samples by fingerprint in StorageQueueManager ... By splitting the single queue into multiple queues and flushing each individual queue in serially (and all queues in parallel), we can guarantee to preserve the order of timestampsin samples sent to downstream systems.	10 years ago
Julius Volz	aa3f2b7216	Generic write cleanups and changes. ... - fold metric name into labels - return initialization errors back to main - add snappy compression - better context handling - pre-allocation of labels - remove generic naming - other cleanups	10 years ago
Brian Brazil	36d2c4bd0b	Add generic write path using grpc. ... This uses a new proto format, with scope for multiple samples per timeseries in future. This will allow users to pump samples out to whatever they like without having to change the core Prometheus code. There's also an example receiver to save users figuring out the boilerplate themselves.	10 years ago
Dan Milstein	ec064c96f6	Add field names to table-driven test fixtures	10 years ago
Dan Milstein	ac8788aca6	Convert to table-driven test and inline helper func	10 years ago
Dan Milstein	f50f656a66	Fix double-delta unmarshaling to respect actual min header size ... Turns out its valid to have an overall chunk which is smaller than the full doubleDeltaHeaderBytes size -- if it has a single sample, it doesn't fill the whole header. Updated unmarshalling check to respect this.	10 years ago
Dan Milstein	b815956341	Catch errors when unmarshalling delta/doubleDelta encoded chunks ... This is (hopefully) a fix for #1653 Specifically, this makes it so that if the length for the stored delta/doubleDelta is somehow corrupted to be too small, the attempt to unmarshal will return an error. The current (broken) behavior is to return a malformed chunk, which can then lead to a panic when there is an attempt to read header values. The referenced issue proposed creating chunks with a minimum length -- I instead opted to just error on the attempt to unmarshal, since I'm not clear on how it could be safe to proceed when the length is incorrect/unknown. The issue also talked about possibly "quarantining series", but I don't know the surrounding code well enough to understand how to make that happen.	10 years ago
Matt Bostock	e618af5d0b	Storage: Add crash recovery metric 'started_dirty' ... ...to indicate when crash recovery was invoked during Prometheus startup. Fixes #1918.	10 years ago
Dan Milstein	764ceaa939	Add timeout to test, cap waiting at 1 second	10 years ago
Dan Milstein	007907b410	Fix one of the tests for a remote storage QueueManager ... Specifically, the TestSpawnNotMoreThanMaxConcurrentSendsGoroutines was failing on a fresh checkout of master. The test had a race condition -- it would only pass if one of the spawned goroutines happened to very quickly pull a set of samples off an internal queue. This patch rewrites the test so that it deterministically waits until all samples have been pulled off that queue. In case of errors, it also now reports on the difference between what it expected and what it found. I verified that, if the code under test is deliberately broken, the test successfully reports on that.	10 years ago
Julius Volz	3bfec97d46	Make the storage interface higher-level. ... See discussion in https://groups.google.com/forum/#!topic/prometheus-developers/bkuGbVlvQ9g The main idea is that the user of a storage shouldn't have to deal with fingerprints anymore, and should not need to do an individual preload call for each metric. The storage interface needs to be made more high-level to not expose these details. This also makes it easier to reuse the same storage interface for remote storages later, as fewer roundtrips are required and the fingerprint concept doesn't work well across the network. NOTE: this deliberately gets rid of a small optimization in the old query Analyzer, where we dedupe instants and ranges for the same series. This should have a minor impact, as most queries do not have multiple selectors loading the same series (and at the same offset).	10 years ago
beorn7	fc6737b7fb	storage: improve index lookups ... tl;dr: This is not a fundamental solution to the indexing problem (like tindex is) but it at least avoids utilizing the intersection problem to the greatest possible amount. In more detail: Imagine the following query: nicely:aggregating:rule{job="foo",env="prod"} While it uses a nicely aggregating recording rule (which might have a very low cardinality), Prometheus still intersects the low number of fingerprints for `{__name__="nicely:aggregating:rule"}` with the many thousands of fingerprints matching `{job="foo"}` and with the millions of fingerprints matching `{env="prod"}`. This totally innocuous query is dead slow if the Prometheus server has a lot of time series with the `{env="prod"}` label. Ironically, if you make the query more complicated, it becomes blazingly fast: nicely:aggregating:rule{job=~"foo",env=~"prod"} Why so? Because Prometheus only intersects with non-Equal matchers if there are no Equal matchers. That's good in this case because it retrieves the few fingerprints for `{__name__="nicely:aggregating:rule"}` and then starts right ahead to retrieve the metric for those FPs and checking individually if they match the other matchers. This change is generalizing the idea of when to stop intersecting FPs and go into "retrieve metrics and check them individually against remaining matchers" mode: - First, sort all matchers by "expected cardinality". Matchers matching the empty string are always worst (and never used for intersections). Equal matchers are in general consider best, but by using some crude heuristics, we declare some better than others (instance labels or anything that looks like a recording rule). - Then go through the matchers until we hit a threshold of remaining FPs in the intersection. This threshold is higher if we are already in the non-Equal matcher area as intersection is even more expensive here. - Once the threshold has been reached (or we have run out of matchers that do not match the empty string), start with "retrieve metrics and check them individually against remaining matchers". A beefy server at SoundCloud was spending 67% of its CPU time in index lookups (fingerprintsForLabelPairs), serving mostly a dashboard that is exclusively built with recording rules. With this change, it spends only 35% in fingerprintsForLabelPairs. The CPU usage dropped from 26 cores to 18 cores. The median latency for query_range dropped from 14s to 50ms(!). As expected, higher percentile latency didn't improve that much because the new approach is _occasionally_ running into the worst case while the old one was _systematically_ doing so. The 99th percentile latency is now about as high as the median before (14s) while it was almost twice as high before (26s).	10 years ago
Dmitry Vorobev	273e457da4	web: return status code and error message for config resource	10 years ago
beorn7	2a75b15328	Crash recovery: Fix an edge case. ... If the chunks of a series in the checkpoint are all older then the latest chunk on disk, the head chunk is persisted and therefore has to be declared closed. It would be great to have a test for this, but that would require more plumbing, subject of #447.	10 years ago
beorn7	064b57858e	Consistently use the `Seconds()` method for conversion of durations ... This also fixes one remaining case of recording integral numbers of seconds only for a metric, i.e. this will probably fix #1796.	10 years ago
Julius Volz	91401794fa	storage: Make MemorySeriesStorage a public type ... See https://twitter.com/fabxc/status/748032597876482048	10 years ago