prometheus

Commit Graph

Author	SHA1	Message	Date
Tom Wilkie	5169f990f9	Review feedback: add yaml struct tags, don't embed queue config. Also, rename QueueManageConfig to QueueConfig, for consistency with tags.	8 years ago
Tom Wilkie	454b661145	Make queue manager configurable.	8 years ago
Tom Wilkie	2dda5775e3	Initial port of remote storage to v2.	8 years ago
Tom Wilkie	e5d7bbfc3c	Remote writes: retry on recoverable errors. (#2552 ) * Remote writes: retry on recoverable errors. * Add comments * Review feedback * Comments * Review feedback * Final spelling misteak (I hope). Plus, record failed samples correctly.	8 years ago
Tom Wilkie	77cce900b8	Fix tests	8 years ago
Julius Volz	2f39dbc8b3	Rename StorageQueueManager -> QueueManager	8 years ago
Julius Volz	e9476b35d5	Re-add multiple remote writers Each remote write endpoint gets its own set of relabeling rules. This is based on the (yet-to-be-merged) https://github.com/prometheus/prometheus/pull/2419, which removes legacy remote write implementations.	8 years ago
Tom Wilkie	4520e12440	Add HTTP Basic Auth & TLS support to the generic write path. (#1957 ) * Add config, HTTP Basic Auth and TLS support to the generic write path. - Move generic write path configuration to the config file - Factor out config.TLSConfig -> tlf.Config translation - Support TLSConfig for generic remote storage - Rename Run to Start, and make it non-blocking. - Dedupe code in httputil for TLS config. - Make remote queue metrics global.	9 years ago
Fabian Reinartz	7bd7e63f97	storage: fix struct alignment issue in test The uint64 `numCalls` ends up being not word-aligned on certain architectures, which makes atomic reads/writes panic.	9 years ago
Tom Wilkie	ece12bff93	Shard/parrallelise samples by fingerprint in StorageQueueManager By splitting the single queue into multiple queues and flushing each individual queue in serially (and all queues in parallel), we can guarantee to preserve the order of timestampsin samples sent to downstream systems.	9 years ago
Dan Milstein	764ceaa939	Add timeout to test, cap waiting at 1 second	9 years ago
Dan Milstein	007907b410	Fix one of the tests for a remote storage QueueManager Specifically, the TestSpawnNotMoreThanMaxConcurrentSendsGoroutines was failing on a fresh checkout of master. The test had a race condition -- it would only pass if one of the spawned goroutines happened to very quickly pull a set of samples off an internal queue. This patch rewrites the test so that it deterministically waits until all samples have been pulled off that queue. In case of errors, it also now reports on the difference between what it expected and what it found. I verified that, if the code under test is deliberately broken, the test successfully reports on that.	9 years ago
Dmitry Vorobev	bd2a770015	storage/remote: Spawn not more than "maxConcurrentSends" goroutines.	9 years ago
Julius Volz	963ad82dcb	Fix "go vet" errors. I ignored all errors of the type "composite literal uses unkeyed fields". Most of them are wrong because of https://github.com/golang/go/issues/9171.	10 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	10 years ago
Julius Volz	593e565688	Allow writing to InfluxDB/OpenTSDB at the same time.	10 years ago
Julius Volz	61fb688dd9	Add experimental InfluxDB write support.	10 years ago
beorn7	be11cb2b07	Remove the sample ingestion channel. The one central sample ingestion channel has caused a variety of trouble. This commit removes it. Targets and rule evaluation call an Append method directly now. To incorporate multiple storage backends (like OpenTSDB), storage.Tee forks the Append into two different appenders. Note that the tsdb queue manager had its own queue anyway. It was a queue after a queue... Much queue, so overhead... Targets have their own little buffer (implemented as a channel) to avoid stalling during an http scrape. But a new scrape will only be started once the old one is fully ingested. The contraption of three pipelined ingesters was removed. A Target is an ingester itself now. Despite more logic in Target, things should be less confusing now. Also, remove lint and vet warnings in ast.go.	10 years ago
Bjoern Rabenstein	5859b74f1b	Clean up license issues. - Move CONTRIBUTORS.md to the more common AUTHORS. - Added the required NOTICE file. - Changed "Prometheus Team" to "The Prometheus Authors". - Reverted the erroneous changes to the Apache License.	11 years ago
Bjoern Rabenstein	b3ed9aa7a2	Clean up start-up and shut-down. Change-Id: Idff4bbb0a15a9f879bfbb3da5b1025179cab5e2c	11 years ago
Julius Volz	61d26e8445	Add optional sample replication to OpenTSDB. Prometheus needs long-term storage. Since we don't have enough resources to build our own timeseries storage from scratch ontop of Riak, Cassandra or a similar distributed datastore at the moment, we're planning on using OpenTSDB as long-term storage for Prometheus. It's data model is roughly compatible with that of Prometheus, with some caveats. As a first step, this adds write-only replication from Prometheus to OpenTSDB, with the following things worth noting: 1) I tried to keep the integration lightweight, meaning that anything related to OpenTSDB is isolated to its own package and only main knows about it (essentially it tees all samples to both the existing storage and TSDB). It's not touching the existing TieredStorage at all to avoid more complexity in that area. This might change in the future, especially if we decide to implement a read path for OpenTSDB through Prometheus as well. 2) Backpressure while sending to OpenTSDB is handled by simply dropping samples on the floor when the in-memory queue of samples destined for OpenTSDB runs full. Prometheus also only attempts to send samples once, rather than implementing a complex retry algorithm. Thus, replication to OpenTSDB is best-effort for now. If needed, this may be extended in the future. 3) Samples are sent in batches of limited size to OpenTSDB. The optimal batch size, timeout parameters, etc. may need to be adjusted in the future. 4) OpenTSDB has different rules for legal characters in tag (label) values. While Prometheus allows any characters in label values, OpenTSDB limits them to a to z, A to Z, 0 to 9, -, _, . and /. Currently any illegal characters in Prometheus label values are simply replaced by an underscore. Especially when integrating OpenTSDB with the read path in Prometheus, we'll need to reconsider this: either we'll need to introduce the same limitations for Prometheus labels or escape/encode illegal characters in OpenTSDB in such a way that they are fully decodable again when reading through Prometheus, so that corresponding timeseries in both systems match in their labelsets. Change-Id: I8394c9c55dbac3946a0fa497f566d5e6e2d600b5	12 years ago

19 Commits (a8887f46dcc4dc27b15b9d917ec729b5732d589c)