116 Commits (a391156dfba9cd2fed2fb960baea20175cbc06f9)

Author	SHA1	Message	Date
Chris Goller	42de0ae013	Use log.Logger interface for all discovery services	9 years ago
Conor Broderick	6766123f93	Replace regex with Secret type and remarshal config to hide secrets (#2775)	9 years ago
Shashank Varanasi	dea60bb553	Fix malformed uname string (#2727) ... * Fix malformed uname string * Make fix better * Reformat code for simplicity	9 years ago
Shashank Varanasi	61235fd851	Print system information (uname) at Prometheus startup (#2709) ... * Print uname on prom startup * Make uname file linux-only * Add missing license headers Add missing license headers * Print OS when uname is not available * Print only OS name when uname not available * Remove extra space, fix cmd/prometheus/main.go license header * Add fix for int8 and uint8 systems * Better formatting for build tags in cmd/prometheus/uname files * Remove newline	9 years ago
Matt Layher	283756c503	Initial commit of 'promtool check-metrics', promlint package (#2605)	9 years ago
beorn7	f20b84e816	flags: Improve doc strings for checkpoint flags	9 years ago
Goutham Veeramachaneni	f27ce34a13	Use Registerer to Register All Metrics ... * Made Metric a Gauge so that it can be registered.	9 years ago
Goutham Veeramachaneni	0d0c9d5440	Move Registerer to Config Struct in Notifier	9 years ago
beorn7	434ab2a6a3	storage: Evict chunks and calculate persistence pressure based on target heap size ... This is a fairly easy attempt to dynamically evict chunks based on the heap size. A target heap size has to be set as a command line flage, so that users can essentially say "utilize 4GiB of RAM, and please don't OOM". The -storage.local.max-chunks-to-persist and -storage.local.memory-chunks flags are deprecated by this change. Backwards compatibility is provided by ignoring -storage.local.max-chunks-to-persist and use -storage.local.memory-chunks to set the new -storage.local.target-heap-size to a reasonable (and conservative) value (both with a warning). This also makes the metrics intstrumentation more consistent (in naming and implementation) and cleans up a few quirks in the tests. Answers to anticipated comments: There is a chance that Go 1.9 will allow programs better control over the Go memory management. I don't expect those changes to be in contradiction with the approach here, but I do expect them to complement them and allow them to be more precise and controlled. In any case, once those Go changes are available, this code has to be revisted. One might be tempted to let the user specify an estimated value for the RSS usage, and then internall set a target heap size of a certain fraction of that. (In my experience, 2/3 is a fairly safe bet.) However, investigations have shown that RSS size and its relation to the heap size is really really complicated. It depends on so many factors that I wouldn't even start listing them in a commit description. It depends on many circumstances and not at least on the risk trade-off of each individual user between RAM utilization and probability of OOMing during a RAM usage peak. To not add even more to the confusion, we need to stick to the well-defined number we also use in the targeting here, the sum of the sizes of heap objects.	9 years ago
beorn7	96a303b348	storage: Use staleness delta as head chunk timeout ... Currently, if a series stops to exist, its head chunk will be kept open for an hour. That prevents it from being persisted. Which prevents it from being evicted. Which prevents the series from being archived. Most of the time, once no sample has been added to a series within the staleness limit, we can be pretty confident that this series will not receive samples anymore. The whole chain as described above can be started after 5m instead of 1h. In the relaxed case, this doesn't change a lot as the head chunk timeout is only checked during series maintenance, and usually, a series is only maintained every six hours. However, there is the typical scenario where a large service is deployed, the deoply turns out to be bad, and then it is deployed again within minutes, and quite quickly the number of time series has tripled. That's the point where the Prometheus server is stressed and switches (rightfully) into rushed mode. In that mode, time series are processed as quickly as possible, but all of that is in vein if all of those recently ended time series cannot be persisted yet for another hour. In that scenario, this change will help most, and it's exactly the scenario where help is most desperately needed.	9 years ago
beorn7	04ccf84559	main.go: Set GOGC to 40 by default ... Rationale: The default value for GOGC is 100, i.e. a garbage collected is initialized once as many heap space has been allocated as was in use after the last GC was done. This ratio doesn't make a lot of sense in Prometheus, as typically about 60% of the heap is allocated for long-lived memory chunks (most of which are around for many hours if not days). Thus, short-lived heap objects are accumulated for quite some time until they finally match the large amount of memory used by bulk memory chunks and a gigantic GC cyle is invoked. With GOGC=40, we are essentially reinstating "normal" GC behavior by acknowledging that about 60% of the heap are used for long-term bulk storage. The median Prometheus production server at SoundCloud runs a GC cycle every 90 seconds. With GOGC=40, a GC cycle is run every 35 seconds (which is still not very often). However, the effective RAM usage is now reduced by about 30%. If settings are updated to utilize more RAM, the time between GC cycles goes up again (as the heap size is larger with more long-lived memory chunks, but the frequency of creating short-lived heap objects does not change). On a quite busy large Prometheus server, the timing changed from one GC run every 20s to one GC run every 12s. In the former case (just changing GOGC, leave everything else as it is), the CPU usage increases by about 10% (on a mid-size referenc server from 8.1 to 8.9). If settings are adjusted, the CPU consumptions increases more drastically (from 8 cores to 13 cores on a large reference server), despite GCs happening more rarely, presumably because a 50% larger set of memory chunks is managed now. Having more memory chunks is good in many regards, and most servers are running out of memory long before they run out of CPU cycles, so the tradeoff is overwhelmingly positive in most cases. Power users can still set the GOGC environment variable as usual, as the implementation in this commit honors an explicitly set variable.	9 years ago
Julius Volz	8fda83ea12	Make rules only read local data	9 years ago
Julius Volz	406b65d0dc	Rename remote.Storage to remote.Writer	9 years ago
Julius Volz	02395a224d	[WIP] Remote Read	9 years ago
Goutham Veeramachaneni	f35816613e	Refactored Notifier to use Registerer ... * Brought metrics back into Notifier Notifier still implements a Collector. Check if that is needed.	9 years ago
Julius Volz	beb3c4b389	Remove legacy remote storage implementations ... This removes legacy support for specific remote storage systems in favor of only offering the generic remote write protocol. An example bridge application that translates from the generic protocol to each of those legacy backends is still provided at: documentation/examples/remote_storage/remote_storage_bridge See also https://github.com/prometheus/prometheus/issues/10 The next step in the plan is to re-add support for multiple remote storages.	9 years ago
Bartek Plotka	579e33f19a	Fixed style issues.	10 years ago
Bartek Plotka	d7febe97fa	Fixed regression in -alertmanager.url flag. Basic auth was ignored. ... - Included basic auth parsing while parsing to AlertmanagerConfig - Added test case Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	10 years ago
Erdem Agaoglu	054f8ebbfb	Increase default max-connections	10 years ago
Erdem Agaoglu	e487477a17	LimitListener to limit max number of connections ... This also drops tcp keep-alive in ListenAndServe but it's no longer necessary since we now close idle connections long before that.	10 years ago
Erdem Agaoglu	9986b28380	Set read-timeout for http.Server ... This also specifies a timeout for idle client connections, which may cause "too many open files" errors. See #2238	10 years ago
Fabian Reinartz	3fb4d1191b	config: rename AlertingConfig, resolve file paths	10 years ago
Fabian Reinartz	d4deb8bbf2	web: show discovered Alertmanagers in UI	10 years ago
Fabian Reinartz	f210d96497	notifier: use dynamic service discovery	10 years ago
Fabian Reinartz	200bbe1bad	config: extract SD and HTTPClient configurations	10 years ago
beorn7	5c41ca84e5	Catch negative staleness delta set on the command line	10 years ago
Brian Brazil	6bc29ba857	Fix regression from #1957, specify non-zero default timeout. (#2121) ... Fixes #2075	10 years ago
Julius Volz	ab80ced756	storage: separate chunk package, publish more names ... This is a followup to https://github.com/prometheus/prometheus/pull/2011. This publishes more of the methods and other names of the chunk code and moves the chunk code to its own package. There's some unavoidable ugliness: the chunk and chunkDesc metrics are used by both packages, so I had to move them to the chunk package. That isn't great, but I don't see how to do it better without a larger redesign of everything. Same for the evict requests and some other types.	10 years ago
Fabian Reinartz	57b358b82a	vendor: update govalidator (#2023) ... Fixes #2022	10 years ago
Matt Bostock	dd98766b32	cmd/prometheus/main.go: Fix typo in comment	10 years ago
Tom Wilkie	4520e12440	Add HTTP Basic Auth & TLS support to the generic write path. (#1957) ... * Add config, HTTP Basic Auth and TLS support to the generic write path. - Move generic write path configuration to the config file - Factor out config.TLSConfig -> tlf.Config translation - Support TLSConfig for generic remote storage - Rename Run to Start, and make it non-blocking. - Dedupe code in httputil for TLS config. - Make remote queue metrics global.	10 years ago
Julius Volz	c187308366	storage: Contextify storage interfaces. ... This is based on https://github.com/prometheus/prometheus/pull/1997. This adds contexts to the relevant Storage methods and already passes PromQL's new per-query context into the storage's query methods. The immediate motivation supporting multi-tenancy in Frankenstein, but this could also be used by Prometheus's normal local storage to support cancellations and timeouts at some point.	10 years ago
Julius Volz	ed5a0f0abe	promql: Allow per-query contexts. ... For Weaveworks' Frankenstein, we need to support multitenancy. In Frankenstein, we initially solved this without modifying the promql package at all: we constructed a new promql.Engine for every query and injected a storage implementation into that engine which would be primed to only collect data for a given user. This is problematic to upstream, however. Prometheus assumes that there is only one engine: the query concurrency gate is part of the engine, and the engine contains one central cancellable context to shut down all queries. Also, creating a new engine for every query seems like overkill. Thus, we want to be able to pass per-query contexts into a single engine. This change gets rid of the promql.Engine's built-in base context and allows passing in a per-query context instead. Central cancellation of all queries is still possible by deriving all passed-in contexts from one central one, but this is now the responsibility of the caller. The central query context is now created in main() and passed into the relevant components (web handler / API, rule manager). In a next step, the per-query context would have to be passed to the storage implementation, so that the storage can implement multi-tenancy or other features based on the contextual information.	10 years ago
Tom Wilkie	d83879210c	Switch back to protos over HTTP, instead of GRPC. ... My aim is to support the new grpc generic write path in Frankenstein. On the surface this seems easy - however I've hit a number of problems that make me think it might be better to not use grpc just yet. The explanation of the problems requires a little background. At weave, traffic to frankenstein need to go through a couple of services first, for SSL and to be authenticated. So traffic goes: internet -> frontend -> authfe -> frankenstein - The frontend is Nginx, and adds/removes SSL. Its done this way for legacy reasons, so the certs can be managed in one place, although eventually we imagine we'll merge it with authfe. All traffic from frontend is sent to authfe. - Authfe checks the auth tokens / cookie etc and then picks the service to forward the RPC to. - Frankenstein accepts the reads and does the right thing with them. First problem I hit was Nginx won't proxy http2 requests - it can accept them, but all calls downstream are http1 (see https://trac.nginx.org/nginx/ticket/923). This wasn't such a big deal, so it now looks like: internet --(grpc/http2)--> frontend --(grpc/http1)--> authfe --(grpc/http1)--> frankenstein Next problem was golang grpc server won't accept http1 requests (see https://groups.google.com/forum/#!topic/grpc-io/JnjCYGPMUms). It is possible to link a grpc server in with a normal go http mux, as long as the mux server is serving over SSL, as the golang http client & server won't do http2 over anything other than an SSL connection. This would require making all our service to service comms SSL. So I had a go a writing a grpc http1 server, and got pretty far. But is was a bit of a mess. So finally I thought I'd make a separate grpc frontend for this, running in parallel with the frontend/authfe combo on a different port - and first up I'd need a grpc reverse proxy. Ideally we'd have some nice, generic reverse proxy that only knew about a map from service names -> downstream service, and didn't need to decode & re-encode every request as it went through. It seems like this can't be done with golang's grpc library - see https://github.com/mwitkow/grpc-proxy/issues/1. And then I was surprised to find you can't do grpc from browsers! See http://www.grpc.io/faq/ - not important to us, but I'm starting to question why we decided to use grpc in the first place? It would seem we could have most of the benefits of grpc with protos over HTTP, and this wouldn't preclude moving to grpc when its a bit more mature? In fact, the grcp FAQ even admits as much: > Why is gRPC better than any binary blob over HTTP/2? > This is largely what gRPC is on the wire.	10 years ago
Tobias Schmidt	29ced0090f	Fix common english misspellings	10 years ago
Julius Volz	b24e5d63bc	Add noop local storage engine. ... This adds a flag -storage.local.engine which allows turning off local storage in Prometheus. Instead of adding if-conditions and nil checks to all parts of Prometheus that deal with Prometheus's local storage (including the web interface), disabling local storage simply means replacing the normal local storage with a noop version that throws samples away and returns empty query results. We also don't add the noop storage to the fanout appender to decrease internal overhead. Instead of returning empty results, an alternate behavior could be to return errors on any query that point out that the local storage is disabled. Not sure which one is more preferable, so I went with the empty result option for now.	10 years ago
Julius Volz	a88e950d1f	Mark remote write address flag as experimental.	10 years ago
Julius Volz	aa3f2b7216	Generic write cleanups and changes. ... - fold metric name into labels - return initialization errors back to main - add snappy compression - better context handling - pre-allocation of labels - remove generic naming - other cleanups	10 years ago
Brian Brazil	36d2c4bd0b	Add generic write path using grpc. ... This uses a new proto format, with scope for multiple samples per timeseries in future. This will allow users to pump samples out to whatever they like without having to change the core Prometheus code. There's also an example receiver to save users figuring out the boilerplate themselves.	10 years ago
Julius Volz	4a866c13be	Fix ApplyConfig() error handling ... Currently, Prometheus starts up without any error when there is an invalid rule file :-/	10 years ago
Dave Rawks	40b9666479	Error on non-flag commandline arguments ... - Added minor cmdline parsing logic change to bail on unconsumed arguments. Fixes #1821	10 years ago
Dave Rawks	00ea36cdbe	Error on non-flag commandline arguments ... - Added minor cmdline parsing logic change to bail on unconsumed arguments. Fixes #1821	10 years ago
beorn7	bf6201483c	Improve wording on log flag comment	10 years ago
beorn7	25385aafcb	Explicitly add logging flags to our custom flag set ... In https://github.com/prometheus/prometheus/pull/1782 , we moved to a custom flag set to avoid getting test flags into the main prometheus binary. However, that removed the logging flags, too. This commit updates the vendoring to a version of the log package that allows adding the log flags to our flag set explicitly.	10 years ago
Dmitry Vorobev	273e457da4	web: return status code and error message for config resource	10 years ago
Fabian Reinartz	59d26e8536	web: add -web.route-prefix flag ... Fixes #1191	10 years ago
Fabian Reinartz	8c24dfdb86	cmd/prometheus: use own flag set ... Fixes #1743	10 years ago
Fabian Reinartz	9baf120cd5	notifier: dispatch to multiple Alertmanagers ... This commit extends the notifier to dispatch alert batches to multiple Alertmanagers concurrently. It changes the `-alertmanager.url` flag to accept a comma separated list of URLs and/or to be set multiple times.	10 years ago
beorn7	99881ded63	Make the number of fingerprint mutexes configurable ... With a lot of series accessed in a short timeframe (by a query, a large scrape, checkpointing, ...), there is actually quite a significant amount of lock contention if something similar is running at the same time. In those cases, the number of locks needs to be increased. On the same front, as our fingerprints don't have a lot of entropy, I introduced some additional shuffling. With the current state, anly changes in the least singificant bits of a FP would matter.	10 years ago
beorn7	da8cb10b43	Partition the status tab into items in a dropdown ... I got feedback from different sources about rules and targets being too heavy in the status tab if their are lots of them. This change also allows for more fine-granular locking.	10 years ago