411 Commits (ce407d3276292fb5b225867a66617903f1a808f8)

Author	SHA1	Message	Date
Marco Pracucci	ce407d3276	Add ingestion rate global limit support (#1486) ... * Added ingestion rate global limit support Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed code comments Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated config doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated changelog Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated doc Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Cyril Tovena	33f70b70ce	Add duplicates info and remove timing informations. (#1496) ... * Add duplicates info and remove timing informations. Timing information in the chunkenc iterator are too eating too much resources. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	7c4906786d	pre-fetch the next chunk in parallel to processing the current chunk such that we aren't waiting for fetches from GCS and spend all our CPU time decompressing and processing (#1495) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Owen Diehl	a1d090e161	Fix/deadlock frontend queue (#1490) ... * buffers splitby interval channels to prevent deadlocks * improves codec merge test requiring merging from multiple intervals * bound concurrency by min(parallelism,jobs) * approximates goroutine test	5 years ago
Owen Diehl	7cc93b2694	unifies reverse iterators (#1489)	5 years ago
Cyril Tovena	589e53d844	Fixes response json encoding and add regression test. (#1488) ... Also align stream labels alphabetical order with querier. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	be010f049d	Close iterator properly and check nil before releasing buffers. (#1480) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	4153740a12	Loki Query Frontend (#1442) ... * Adds frontend to Loki. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sneaky bug in entries sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tweak jsonnet deployments and add a way to lint/fmt. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tests and rollback change in loghttp package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a flaky test that might run one more goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes windows build. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tracing in the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test stream to proto conversion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flappy retry test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove err shadowing in stopQueryFrontend as it was confusing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor grpc message size in the libsonnet config file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Don't check auth header for GRPC TransferChunks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Query frontend (#3) * frontend codec merging optimizations * codec benchmarks * removes unused bounds code in queryrange ordering * [wip] splitby uses channels instead of sub batching intervals * splitBy channel limit test * single allocation for merging entries from a single stream * skip merging loki responses when limit is already hit * removes checks for unlimited queries in queryrange * removes splitByInterval{,.interval} spans * removes interval_batch_size from jsonnet lib * moves benchmark utils to own file * renames markers -> entries * priority queue comments * Removes unused logRequest. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Sets the cache interval to the same split interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing import libsonnet for the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Frontend should not be a cluster IP. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Owen Diehl	0b72ddb6a6	positions.ignore-corruptions (#1472) ... * positions.ignore-corruptions * semantic change: s/ignore-corruptions/ignore-invalid-yaml/ * Update pkg/promtail/positions/positions.go Co-Authored-By: Robert Fratto <robertfratto@gmail.com>	5 years ago
Robert Fratto	52931a1bc5	pkg/querier: don't query all ingesters (#1473) ... This commit uses ReplicationSet.Do for communicating with the ingesters from the queriers. This mimics the behavior of Cortex. An extra_query_delay flag has been added to match the flag present in Cortex's querier config. Fixes #1447.	5 years ago
Nathan Zender	282d77b690	Using strict parsing for yaml configs (#1433)	5 years ago
Cyril Tovena	10b5bcab72	Validates limit parameter. (#1470) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	dafb9d8425	Enables Series API in loki (#1419) ... * helpers for parsing /series requests * logproto.SeriesResponse impl * passes unparsed matchers in protobuf for simplicity * ingester support for series queries * labels map coercion * fixes label match loghttp test * removes labelmatcher protobuf adapters * series api impl * tests matchers for dispatching series queries * series looked up serially for /series api * splits series api async code and remove unnecessary type assertion * adds series handle to /api/prom/series and includes docs * compatibility changes w/ upstream merge	6 years ago
Cyril Tovena	fb76416b27	Add decompression tracing instrumentation. (#1445) ... * Add decompression tracing instrumentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix time conversion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use span logger so we also get a log line. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve reusability and add fetched chunks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	768bf914e0	Changes the encoding used for the benchmark in the store. (#1448) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Peter Štibraný	4381b28c80	pkg/chunkenc: BenchmarkRead focuses on reading chunks (#1423) ... Previously BenchmarkRead asked for every single line from chunks to be returned. This causes a lot of unnnecessary allocations, which dominate the benchmark. Instead of counting bytes when reading, we now count size when generating data for logging speed. Another test was added to show that these two approaches are comparable. This change makes BenchmarkRead to report real time needed to decode chunks: name old time/op new time/op delta Read/none-4 86.2ms ± 0% 33.2ms ± 0% ~ (p=1.000 n=1+1) Read/gzip-4 255ms ± 0% 194ms ± 0% ~ (p=1.000 n=1+1) Read/lz4-4 121ms ± 0% 64ms ± 0% ~ (p=1.000 n=1+1) Read/snappy-4 119ms ± 0% 67ms ± 0% ~ (p=1.000 n=1+1) name old alloc/op new alloc/op delta Read/none-4 134MB ± 0% 0MB ± 0% ~ (p=1.000 n=1+1) Read/gzip-4 135MB ± 0% 0MB ± 0% ~ (p=1.000 n=1+1) Read/lz4-4 136MB ± 0% 1MB ± 0% ~ (p=1.000 n=1+1) Read/snappy-4 135MB ± 0% 0MB ± 0% ~ (p=1.000 n=1+1) name old allocs/op new allocs/op delta Read/none-4 491k ± 0% 3k ± 0% ~ (p=1.000 n=1+1) Read/gzip-4 491k ± 0% 3k ± 0% ~ (p=1.000 n=1+1) Read/lz4-4 491k ± 0% 3k ± 0% ~ (p=1.000 n=1+1) Read/snappy-4 491k ± 0% 3k ± 0% ~ (p=1.000 n=1+1) Decompression speed is now also more correct (esp. None is much higher than LZ4/Snappy) Before (/s) Now (/s) None 1.1 GB (n=1) 4.0 GB (n=1) None 1.5 GB (n=9) 3.8 GB (n=38) None 1.6 GB (n=13) Gzip 516 MB (n=1) 640 MB (n=1) Gzip 509 MB (n=3) 664 MB (n=4) Gzip 514 MB (n=4) 649 MB (n=6) LZ4 1.1 GB (n=1) 1.7 GB (n=1) LZ4 1.1 GB (n=9) 1.9 GB (n=15) Snappy 1.1 GB (n=1) 2.0 GB (n=1) Snappy 1.1 GB (n=9) 1.8 GB (n=16) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
wphan	d627cda848	Support microsecond timestamp format (#1426) ... Adds microsecond timestamp support. https://github.com/grafana/loki/issues/1407	6 years ago
Peter Štibraný	a2c2a90907	Lint: comments ... Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	06007c0ad1	Renamed LZ4 -> Lz4 to make lint happy. ... Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	fb143fec02	Undo small change. ... Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	9979321c2a	Don't pool LZ4 readers that read big blocks. ... Such readers must have allocated twice as many bytes as the block size, and we don't want to pool them and keep that memory allocated. This can happen when reading chunks that were compressed with high buffer size. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	b0a7cf8f9d	Change default LZ4 buffer size to 64k. ... Added more LZ4 buffer size options for testing. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Robert Fratto	874d016bd7	promtail: add support for passing through journal entries as JSON ... This commit adds an extra configuration field (`json`) to the journal target configuration schema. When set to true, messages from the journal target will be passed through as JSON, containing all of the fields from the entry as keys and their values as the value. For backwards compatibility, the value for `json` defaults to false. Closes #1288.	6 years ago
Peter Štibraný	dca244f7aa	pkg/ingester: added sync period flags ... SyncPeriod can be used to tell ingesters to cut chunks early (before they are full) to synchronize ingesters so that they put the same data into the new chunks. If they do, they generate fully equal chunks and then they don't need to store them. To avoid very small chunks, additional parameter is added: -ingester.sync-min-utilization. When set to eg. 0.5, chunk must be at least 50% full (utilization) in order to be cut early for sync purposes. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	93b85f92fa	pkg/ingester: Added possibility to disable transfers. (#1425) ... Previously, with -ingester.max-transfer-retries set to zero, transfers would never stop. Now zero means don't transfer at all. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Cyril Tovena	949126e75b	Add a line limit in case a chunk is corrupted to avoid OOM. ... Also when running out of memory we should just return an error and not keep going. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Owen Diehl	ccf9630c58	asserts encodings are nominally the same in chunks during roundtrips	6 years ago
Owen Diehl	245fe8e022	roundtrip v2 encoding tests	6 years ago
Cyril Tovena	549250c30d	Sets the chunk encoding correctly when creating chunk from bytes. ... Previously only the reader and writer for the encoding was set, on subsequent serialization the format would be missing, this only happens if the chunk is created from a chunk transfer. Because it uses NewBytesChunk to load and also write. I've also added a regression tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	7654c27c12	Adds configurable compression algorithms for chunks (#1411) ... * Adds L4Z encoding. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds encoding benchmarks Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds snappy encoding. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds chunk size test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds snappy v2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmarks Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove chunkenc Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update lz4 to latest master version. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use temporary buffer in serialise method to avoid allocations when doing string -> byte conversion. It also makes code little more readable. We pool those buffers for reuse. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Added gzip -1 for comparison. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Initialize reader and buffered reader lazily. This helps with reader/buffered reader reuse. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Don't keep entries, extracted generateData function (mostly to get more understandable profile) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Improve test and benchmark to cover all encodings. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds support for a new chunk format with encoding info. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Ingesters now support encoding config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add support for no compression. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add docs Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove default Gzip for ByteChunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes none, snappyv2 and gzip-1 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move log test lines to testdata and add supported encoding stringer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Peter Štibraný	4fea713b1e	pkg/promtail/client: missing URL in client returns error (#1416) ... Previously it would just panic, because URL.Host is used to construct logger. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	f6be636e13	RangeQuery benchmark optimizations (#1413) ... * Added some duplicate streams. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Instead of using sort.Sort(), we keep tuples sorted already. Also just return first element if there is only one tuple. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Don't do time comparison twice, compare nanoseconds instead. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Sebastian Widmer	f0f6f24926	pkg/promtail: IETF Syslog (RFC5424) Support (#1275) ... * IETF syslog (rfc5254) support * Upgrade go-syslog to correctly released v2.0.1 * Incorporate feedback from new linter * Update documentation with relevant RFC sections. * Incorporate feedback from review * Use context to shutdown server. * Use util.Backoff from cortex * Do not embed mutex into TestLabeledClient * Use strings.HasPrefix * Better naming of connectionsClosed -> openConnections * Incorporate further feedback from review * Callback instead of channel in ParseStream. Removes one running Goroutine per connection. * Improve parse error log level. * Move mutex above field it protects. * Use backoff.Ongoing() * Switch to "nontransparent" parser * Further improvements to the documentation	6 years ago
Peter Štibraný	3aff4dfe2f	pkg/cfg: print help only when requested, and print it on stdout (#1396) ... When invalid parameter or value is provided, error is printed to stderr, together with message indicating how to get list of all parameters. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Ed Welch	f9b25968f7	change the chunk size histogram to allow for bigger buckets (#1409) ... * change the chunk size histogram to allow for bigger buckets Signed-off-by: Edward Welch <edward.welch@grafana.com> * changing buckets again Signed-off-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Edward Welch	591905ed64	forgot to register the new metric ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Ed Welch	5aa82ba199	allow configuring a target chunk size in compressed bytes (#1406) ... * allow configuring a target chunk size in compressed bytes Signed-off-by: Edward Welch <edward.welch@grafana.com> * addressing feedback Signed-off-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Peter Štibraný	7bb77ecdd8	Convert string to bytes once only when doing string filtering. (#1405) ... This makes it slightly faster: name old time/op new time/op delta ContainsFilter-4 18.2ns ± 2% 9.8ns ± 1% -46.20% (p=0.000 n=10+8) name old alloc/op new alloc/op delta ContainsFilter-4 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta ContainsFilter-4 0.00 0.00 ~ (all equal) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Cyril Tovena	1fc0ffde8a	Logql benchmark and performance improvement. (#1371) ... * Add logql benchmark test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes store benchmark limits. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add LogQL improvements. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add back logging in the storage benchmark Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Edward Welch	ea5eec2abf	respond to label queries at `loki/api/v1/labels` instead of `loki/api/v1/label`	6 years ago
Owen Diehl	5227f5bee0	application/x-www-form-urlencoded support (#1381) ... * application/x-www-form-urlencoded support * handler func instead of unnecessary struct	6 years ago
Peter Štibraný	dd96fa146c	pkg/ingester: check that ingester is in LEAVING state when transferring chunks and claiming tokens. Required when using memberlist client. (#1300) ... * On chunks transfer, verify that source ingester is LEAVING This is important when using gossiping ring. If source ingester is not leaving yet, than claiming its tokens will fail (merge function will not move ownership to the target ingester), and target ingester will end up with no tokens in the ring. * Register ring to prometheus. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Update check using latest version from Cortex. There is no direct usage of logging, which is consistent with the rest of this method. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Move import to group of imports with third-party packages. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	e0ec61f986	pkg/ingester: handle labels mapping to the same fast fingerprint. (#1247) ... * pkg/ingester: handle labels mapping to the same fast fingerprint. Uses slightly adapted fpMapper code from Cortex. Fixes issue #898 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * empty commit to force new build Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed empty lines in imports. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Added test that pushes entries concurrently. To be run with -race. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Stream now keeps both original and mapped fingerprint Mapped fingerprint is used in streams map. Original ("rawFP") is preserved when saving chunks. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Vendoring Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Reverted previous commit (keep the test, with updated checks). Preserving raw fingerprints could lead to data loss when doing chunk deduplication while storing chunks. We don't want that. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Cyril Tovena	f7ee1c753c	Supports same duration format in LogQL as Prometheus ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Daniel D'Abate	df89859369	HTTP API: Support duration and float formats for step parameter (#1355) ... * Support duration and float formats for step parameter in http api * Support Prometheus duration format and add missing tests * Fixes linter Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	b34c2b0bd3	Move to jsoniter instead of default json package (#1249) ... * Move to jsoniter instead of default json package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes jsoniter on the logcli output Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Peter Štibraný	49d0d0f0da	Makefile changes to allow easy builds with or without vendoring. Also fixes version bug for both cases. (#1095) ... * Extracted -mod=vendor into a variable * Use ?= to set variable only if not set yet. This enables "MOD_VENDOR= make all" use from command line * Enable/disable vendoring with VENDOR=true or false. Enabled by default * Make go generate use -mod=vendor depending on VENDOR flag * Use GOMOD to specify module version to use. GOMOD can be set to empty string, "vendor", "readonly" and in Go 1.14 to "mod". If not defined, defaults to "vendor". Non empty values are passed to -mod parameter to Go tools, and --modules-download-mode parameter for golangci-cli linting tool. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix versioning info when not using vendoring. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated build image to use Golintci-cli v1.21.0 and Go 1.13.4. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated build image to version 0.8.0. Changs: Golintci-cli v1.21.0, Go 1.13.4. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Renamed MOD_VENDOR to MOD_FLAG to better describe its function. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix version info in all command line tools. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use loki-build-image:0.8.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Cyril Tovena	27f17bc1de	Fixes linter warning from the yacc file. (#1328) ... * Fixes linter warning from the yacc file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves sed to work on POSIX Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Julien Pivotto	eb7b495565	Metrics: use Namespace everywhere when declaring metrics (#1320) ... Code linting to make things consistent. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
Peter Štibraný	125cfbfd5b	pkg/promtail/client: Handle fingerprint hash collisions (#1254) ... * Handle fingerprint hash collisions Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Instead of using fingerprints, use label strings as map keys. Label strings are properly sorted key=value pairs. This solution produces more garbage, but doesn't reinvent its own hashing, and keeps code simple. This seems to be a good tradeoff for promtail. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago