prometheus

Commit Graph

Author	SHA1	Message	Date
Fabian Reinartz	1f877f3d2a	Fix deadlock, structure target logging	10 years ago
Fabian Reinartz	59f1e722df	Return error on sample appending	10 years ago
beorn7	ec08c9a391	Rework the way to communicate backpressure (AKA suspended ingestion) This gives up on the idea to communicate throuh the Append() call (by either not returning as it is now or returning an error as suggested/explored elsewhere). Here I have added a Throttled() call, which has the advantage that it can be called before a whole _batch_ of Append()'s. Scrapes will happen completely or not at all. Same for rule group evaluations. That's a highly desired behavior (as discussed elsewhere). The code is even simpler now as the whole ingestion buffer could be removed. Logging of throttled mode has been streamlined and will create at most one message per minute.	10 years ago
beorn7	87ef24cd25	Add instrumentation and refactor things around "rushed mode"	10 years ago
beorn7	972d94433a	Introduce a hysteresis for "rushed mode" "Rushed mode" is formerly known as "degraded mode", which is changed with this commit, too. The name "degraded" was very misleading. Also, switch into rushed mode if we have too many chunks in memory and an at least reasonable amount of chunks to persist so that speeding up persisting chunks can help.	10 years ago
beorn7	cb117d8346	Add a series ops metric "purge_on_request" It counts series deletions triggered via the API.	10 years ago
beorn7	4221c7de5c	Improve handling of series file truncation If only very few chunks are to be truncated from a very large series file, the rewrite of the file is a lorge overhead. With this change, a certain ratio of the file has to be dropped to make it happen. While only causing disk overhead at about the same ratio (by default 10%), it will cut down I/O by a lot in above scenario.	10 years ago
Fabian Reinartz	e3b6ec9784	Switch to common/log	10 years ago
Julius Volz	af513468eb	Fix some dead code, missing error checks, shadowings. I applied https://medium.com/@jgautheron/quality-pipeline-for-go-projects-497e34d6567 and was greeted with a deluge of warnings, most of which were not applicable or really fixable realistically. These are some of the first ones I decided to fix.	10 years ago
beorn7	daeccdd0e9	Fix DropMetricsForFingerprints It now deletes the series file also for archived series. Also, fix a naming error in a doc comment.	10 years ago
Julius Volz	6774a73878	Fix error checking and logging around checkpointing.	10 years ago
Julius Volz	011faf9057	Fix typo in comment.	10 years ago
Fabian Reinartz	e061595352	Move COWMetric into storage/metric package	10 years ago
Fabian Reinartz	1535ef1457	Replace metric.SamplePair with model.SamplePair	11 years ago
Fabian Reinartz	c9d396f476	Replace metric.LabelPair with model.LabelPair	11 years ago
Fabian Reinartz	438e232c9b	Fix grouping of import blocks	11 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	11 years ago
Laurie Malau	20ad403587	Don't warn/increment metric upon equal timestamps during append. Perhaps it would be even better to still warn in case the sample value has changed but the timestamps are equal, but we don't have efficient access to the last value.	11 years ago
Julius Volz	517badc21d	Only do regex lookups when there was no equality match. For the label matching index-based preselection phase, don't do an OR between equality and non-equality matchers. Execute only one of the two (with equality matchers preferred when present). Fixes https://github.com/prometheus/prometheus/issues/924	11 years ago
beorn7	699946bf32	Fix chunk desc loading. If all samples in consecutive chunks have the same timestamp, the way we used to load chunks will fail. With this change, the persist watermark is used to load the right amount of chunkDescs from disk. This bug is a possible reason for the rare storage corruption we have observed.	11 years ago
beorn7	4203849c92	Test chunkDesc eviction and loading	11 years ago
beorn7	ff08f0b6fe	storage: ensure timestamp monotonicity within series. Fixes https://github.com/prometheus/prometheus/issues/481 While doing so, clean up and fix a few other things: - Fix `go vet` warnings (@fabxc to blame ;). - Fix a racey problem with unarchiving: Whenever we unarchive a series, we essentially want to do something with it. However, until we have done something with it, it appears like a series that is ready to be archived or even purged. So e.g. it would be ignored during checkpointing. With this fix, we always load the chunkDescs upon unarchiving. This is wasteful if we only want to add a new sample to an archived time series, but the (presumably more common) case where we access an archived time series in a query doesn't become more expensive. - The change above streamlined the getOrCreateSeries ond newMemorySeries flow. Also, the modTime is now always set correctly. - Fix the leveldb-backed implementation of KeyValueStore.Delete. It had the wrong behavior of still returning true, nil if a non-existing key has been passed in.	11 years ago
Fabian Reinartz	6bfb4549a6	storage: add LastSamplePairForFingerprint method	11 years ago
Fabian Reinartz	dc7d27ab9a	retrieval: add honor label handling and parametrized querying. This commit adds the honor_labels and params arguments to the scrape config. This allows to specify query parameters used by the scrapers and handling scraped labels with precedence.	11 years ago
Fabian Reinartz	5b91ea9b36	storage: improve label matching and allow unset matching. Matching of empty labels now also matches metrics where the label was not explicitly set to the empty string.	11 years ago
Fabian Reinartz	b105e26f4d	storage: remove global flags	11 years ago
Fabian Reinartz	5c6c0e2faa	Add storage method to delete time series	11 years ago
Fabian Reinartz	aff01e29c3	Limit retrievable samples to retention window. The storage does not delete data immediately after the retention period. We don't want to retrieve this data as it causes artifacts.	11 years ago
Fabian Reinartz	6e319532cf	Read from indexing queue during crash recovery. Change #704 introduced a regression that started reading the queue only after potential crash recovery. When more than the queue capacity was indexed, Prometheus deadlocked.	11 years ago
beorn7	3b9ab546e6	Add metrics to count inconsistencies and fp collisions.	11 years ago
Fabian Reinartz	112a778922	Align int64s for atomic operations	11 years ago
beorn7	3b9c421a69	Weed out all the [Gg]et* method names. The only exception is getNumChunksToPersist to avoid naming the struct member numChunksToPersist in a weird way.	11 years ago
Julius Volz	267fd34156	Switch Prometheus to use github.com/prometheus/log. This change is conceptually very simple, although the diff is large. It switches logging from "github.com/golang/glog" to "github.com/prometheus/log", while not actually changing any log messages. V(1)-style logging has been changed to be log.Debug*().	11 years ago
beorn7	81b190bf45	Remove locking from series iterator. Cache chunk iterators.	11 years ago
Fabian Reinartz	d8440d75f1	Do not start storage processing before Start() is called.	11 years ago
beorn7	2235cec175	Handle fingerprint collisions.	11 years ago
beorn7	9820e5fe99	Use FastFingerprint where appropriate.	11 years ago
beorn7	c5fa0b90c3	Fix the case where a series in memory has 0 chunks, but chunks on disk. This is actually completely normal for a freshly unarchived series. Test added to expose.	11 years ago
beorn7	3035b8bfdd	Adaptively reduce the wait time for memory series maintenance. This will make in-memory series maintenance the faster the more chunks are waiting for persistence.	11 years ago
beorn7	6a21f73898	Fixes after review.	11 years ago
beorn7	51d35f4481	Instrument series maintenance durations.	11 years ago
beorn7	12ae6e9203	Increase resilience of the storage against data corruption - step 4. Step 4: Add a configurable sync'ing of series files after modification.	11 years ago
beorn7	11bd9ce1bd	Increase resilience of the storage against data corruption - step 3. Step 3: Remember the mtime of series files and make use of it to detect series files that are not the one the checkpoint thinks they are.	11 years ago
beorn7	e25cca823c	Increase resilience of the storage against data corruption - step 2. Step 2: Add a flag -storage.local.pedantic-checks to check every series file. Also, remove countPersistedHeadChunks channel, which is unused.	11 years ago
beorn7	3d8d8928be	Increase resilience of the storage against data corruption - step 1. Step 1: Admit the problem by turning the various "panic"s into logged errors, followed by marking the persistence as dirty.	11 years ago
beorn7	da7c0461c6	Rename persist queue len/cap to num/max chunks to persist. Remove deprecated flag storage.incoming-samples-queue-capacity.	11 years ago
beorn7	1d8fc7d56f	Change minor things after code review.	11 years ago
beorn7	be11cb2b07	Remove the sample ingestion channel. The one central sample ingestion channel has caused a variety of trouble. This commit removes it. Targets and rule evaluation call an Append method directly now. To incorporate multiple storage backends (like OpenTSDB), storage.Tee forks the Append into two different appenders. Note that the tsdb queue manager had its own queue anyway. It was a queue after a queue... Much queue, so overhead... Targets have their own little buffer (implemented as a channel) to avoid stalling during an http scrape. But a new scrape will only be started once the old one is fully ingested. The contraption of three pipelined ingesters was removed. A Target is an ingester itself now. Despite more logic in Target, things should be less confusing now. Also, remove lint and vet warnings in ast.go.	11 years ago
beorn7	0056eaeb4f	Redesign series maintenance and chunk persistence.	11 years ago
beorn7	5bea942d8e	Improve various things around chunk encoding. A number of mostly minor things: - Rename chunk type -> chunk encoding. - After all, do not carry around the chunk encoding to all parts of the system, but just have one place where the encoding for new chunks is set based on the flag. The new approach has caveats as well, but the polution of so many method signatures is worse. - Use the default chunk encoding for new chunks of existing series. (Previously, only new _series_ would get chunks with the default encoding.) - Use an enum for chunk encoding. (But keep the version number for the flag, for reasons discussed previously.) - Add encoding() to the chunk interface (so that a chunk knows its own encoding - no need to have that in a different top-level function). - Got rid of newFollowUpChunk (which would keep the existing encoding for all chunks of a time series). Now only use newChunk(), which will create a chunk encoding according to the flag. - Simplified transcodeAndAdd. - Reordered methods of deltaEncodedChunk and doubleDeltaEncoded chunk to match the order in the chunk interface. - Only transcode if the chunk is not yet half full. If more than half full, add a new chunk instead.	11 years ago

1 2 3

124 Commits (1f877f3d2abc7d87e7cbfadeb7ca8c24ce8dc029)