logger.Info("Experimental start timestamp zero ingestion enabled. OpenMetrics 1.0 parsing will parse <metric>_created metrics as ST instead of normal sample. Changed default scrape_protocols to prefer PrometheusProto format.","global.scrape_protocols",fmt.Sprintf("%v",config.DefaultGlobalConfig.ScrapeProtocols))
case"st-storage":
// TODO(bwplotka): Implement ST Storage as per PROM-60 and document this hidden feature flag.
logger.Info("Experimental start timestamp storage enabled. OpenMetrics 1.0 parsing will parse <metric>_created metrics as ST instead of normal sample. Changed default scrape_protocols to prefer PrometheusProto format.","global.scrape_protocols",fmt.Sprintf("%v",config.DefaultGlobalConfig.ScrapeProtocols))
logger.Info("Experimental start timestamp storage enabled. OpenMetrics 1.0 parsing will parse <metric>_created metrics as ST instead of normal sample. Changed default scrape_protocols to prefer PrometheusProto format. Changed chunk encoding type to XOR2.","global.scrape_protocols",fmt.Sprintf("%v",config.DefaultGlobalConfig.ScrapeProtocols))
case"delayed-compaction":
c.tsdb.EnableDelayedCompaction=true
logger.Info("Experimental delayed compaction is enabled.")
@ -603,7 +602,7 @@ func main() {
a.Flag("scrape.discovery-reload-interval","Interval used by scrape manager to throttle target groups updates.").
| <codeclass="text-nowrap">--query.timeout</code> | Maximum time a query may take before being aborted. Use with server mode only. | `2m` |
| <codeclass="text-nowrap">--query.max-concurrency</code> | Maximum number of queries executed concurrently. Use with server mode only. | `20` |
| <codeclass="text-nowrap">--query.max-samples</code> | Maximum number of samples a single query can load into memory. Note that queries will fail if they try to load more samples than this into memory, so this also limits the number of samples a query can return. Use with server mode only. | `50000000` |
| <codeclass="text-nowrap">--enable-feature</code><codeclass="text-nowrap">...<codeclass="text-nowrap"> | Comma separated feature names to enable. Valid options: exemplar-storage, expand-external-labels, memory-snapshot-on-shutdown, promql-per-step-stats, promql-experimental-functions, extra-scrape-metrics, auto-gomaxprocs, created-timestamp-zero-ingestion, concurrent-rule-eval, delayed-compaction, old-ui, otlp-deltatocumulative, promql-duration-expr, use-uncached-io, promql-extended-range-selectors, promql-binop-fill-modifiers. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details. | |
| <codeclass="text-nowrap">--enable-feature</code><codeclass="text-nowrap">...<codeclass="text-nowrap"> | Comma separated feature names to enable. Valid options: exemplar-storage, expand-external-labels, memory-snapshot-on-shutdown, promql-per-step-stats, promql-experimental-functions, extra-scrape-metrics, auto-gomaxprocs, created-timestamp-zero-ingestion, st-storage, concurrent-rule-eval, delayed-compaction, old-ui, otlp-deltatocumulative, promql-duration-expr, use-uncached-io, promql-extended-range-selectors, promql-binop-fill-modifiers. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details. | |
| <codeclass="text-nowrap">--agent</code> | Run Prometheus in 'Agent mode'. | |
| <codeclass="text-nowrap">--log.level</code> | Only log messages with the given severity or above. One of: [debug, info, warn, error] | `info` |
| <codeclass="text-nowrap">--log.format</code> | Output format of log messages. One of: [logfmt, json] | `logfmt` |
@ -77,6 +77,35 @@ Therefore, when `created-timestamp-zero-ingestion` is enabled Prometheus changes
Besides enabling this feature in Prometheus, start timestamps need to be exposed by the application being scraped.
## Start timestamp (ST) native storage
`--enable-feature=st-storage`
> WARNING: This is a highly experimental and risky setting.
> * The new SamplesV2 WAL records cannot be replayed with Prometheus versions that do not support them.
> * This feature uses XOR2 encoded chunks, which cannot be read by older Prometheus versions that do not support the encoding.
> * XOR2 encoding is new, meaning downstream tools and LTS systems might now support it yet (e.g. Thanos sidecar uploaded blocks).
> See [PROM-60](https://github.com/prometheus/proposals/pull/60) for the full
design proposal.
Enables the storage of start timestamps (ST) natively per sample, instead of injecting synthetic 0 valued samples (as `created-timestamp-zero-ingestion` does).
Native storage of start timestamps preserves the exact ST values without adding extra samples.
Currently, native start timestamp storage is only supported for float samples; support for histograms will be added in the future.
Additionally, start timestamp values are not yet used by the PromQL engine for queries.
Currently, Prometheus supports start timestamps on:
* `PrometheusProto`
* `OpenMetrics1.0.0`
`PrometheusProto` is recommended.
Besides enabling this feature in Prometheus, start timestamps need to be exposed by the application being scraped.
Enabling this feature flag automatically enables the xor2-encoding flag.
Carrie Edwards
2 months ago