185 Commits (2959184a42398277ff916206235b844a8f7be5d7)

Author	SHA1	Message	Date
David Robertson	3503f42741	Easy type hints in synapse.logging.opentracing (#12894)	3 years ago
Shay	cde8af9a49	Add config flags to allow for cache auto-tuning (#12701)	3 years ago
David Robertson	2607b3e181	Update mypy to 0.950 and fix complaints (#12650)	3 years ago
Richard van der Hoff	ae01a7edd3	Update type annotations for compatiblity with prometheus_client 0.14 (#12389) ... Principally, `prometheus_client.REGISTRY.register` now requires its argument to extend `prometheus_client.Collector`. Additionally, `Gauge.set` is now annotated so that passing `Optional[int]` causes an error.	3 years ago
David Robertson	4ae956c8bb	Use version string helper from matrix-common (#11979) ... * Require latest matrix-common * Use the common function	3 years ago
reivilibre	41818cda1f	Fix type errors introduced by new annotations in the Prometheus Client library. (#11832) ... Co-authored-by: David Robertson <davidr@element.io>	3 years ago
Patrick Cloke	c072c0b829	Fix mypy for platforms without epoll support. (#11771)	3 years ago
Richard van der Hoff	6a78ede569	Improve `reactor_tick_time` metric (#11724) ... The existing implementation of the `python_twisted_reactor_tick_time` metric is pretty useless, because it *only* measures the time taken to execute timed calls and callbacks from threads. That neglects everything that happens off the back of I/O, which is obviously quite a lot for us. To improve this, I've hooked into a different place in the reactor - in particular, where it calls `epoll`. That call is the only place it should wait for something to happen - the rest of the loop *should* be quick. I've also removed `python_twisted_reactor_pending_calls`, because I don't believe anyone ever looks at it, and it's a nuisance to populate.	3 years ago
Richard van der Hoff	20c6d85c6e	Simplify GC prometheus metrics (#11723) ... Rather than hooking into the reactor loop, just add a timed task that runs every 100 ms to do the garbage collection. Part 1 of a quest to simplify the reactor monkey-patching.	3 years ago
Patrick Cloke	10a88ba91c	Use auto_attribs/native type hints for attrs classes. (#11692)	3 years ago
Sean Quah	84fac0f814	Add type annotations to `synapse.metrics` (#10847)	3 years ago
Erik Johnston	82d2168a15	Add metrics to the threadpools (#11178)	3 years ago
David Robertson	1bfd141205	Type hints for the remaining two files in `synapse.http`. (#11164) ... * Teach MyPy that the sentinel context is False This means that if `ctx: LoggingContextOrSentinel` then `bool(ctx)` narrows us to `ctx:LoggingContext`, which is a really neat find! * Annotate RequestMetrics - Raise errors for sentry if we use the sentinel context - Ensure we don't raise an error and carry on, but not recording stats - Include stack trace in the error case to lower Sean's blood pressure * Make mypy pass for synapse.http.request_metrics * Make synapse.http.connectproxyclient pass mypy Co-authored-by: reivilibre <oliverw@matrix.org>	3 years ago
David Robertson	797ee7812d	Relax `ignore-missing-imports` for modules that have stubs now and update mypy (#11006) ... Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	3 years ago
Jonathan de Jong	95e47b2e78	[pyupgrade] `synapse/` (#10348) ... This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744	3 years ago
Jonathan de Jong	bf72d10dbf	Use inline type hints in various other places (in `synapse/`) (#10380)	3 years ago
Richard van der Hoff	b2557cbf42	opentracing: use a consistent name for background processes (#10135) ... ... otherwise we tend to get a namespace clash between the bg process and the functions that it calls.	4 years ago
Richard van der Hoff	ed53bf314f	Set opentracing priority before setting other tags (#10092) ... ... because tags on spans which aren't being sampled get thrown away.	4 years ago
Erik Johnston	8771b1337d	Export jemalloc stats to prometheus when used (#9882)	4 years ago
Erik Johnston	1fb9a2d0bf	Limit how often GC happens by time. (#9902) ... Synapse can be quite memory intensive, and unless care is taken to tune the GC thresholds it can end up thrashing, causing noticable performance problems for large servers. We fix this by limiting how often we GC a given generation, regardless of current counts/thresholds. This does not help with the reverse problem where the thresholds are set too high, but that should only happen in situations where they've been manually configured. Adds a `gc_min_seconds_between` config option to override the defaults. Fixes #9890.	4 years ago
Richard van der Hoff	5d281c10dd	Stop BackgroundProcessLoggingContext making new prometheus timeseries (#9854) ... This undoes part of b076bc276e.	4 years ago
Patrick Cloke	b076bc276e	Always use the name as the log ID. (#9829) ... As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).	4 years ago
Jonathan de Jong	4b965c862d	Remove redundant "coding: utf-8" lines (#9786) ... Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	4 years ago
Patrick Cloke	48d44ab142	Record more information into structured logs. (#9654) ... Records additional request information into the structured logs, e.g. the requester, IP address, etc.	4 years ago
Andrew Morgan	0d87c6bd12	Don't report anything from GaugeBucketCollector metrics until data is present (#8926) ... This PR modifies `GaugeBucketCollector` to only report data once it has been updated, rather than initially reporting a value of 0. Fixes zero values being reported for some metrics on startup until a background job to update the metric's value runs later.	4 years ago
Patrick Cloke	33a02f0f52	Fix additional type hints from Twisted upgrade. (#9518)	4 years ago
Eric Eastwood	0a00b7ff14	Update black, and run auto formatting over the codebase (#9381) ... - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](80d6dc9783/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version	4 years ago
Patrick Cloke	1619802228	Various clean-ups to the logging context code (#8935)	4 years ago
David Teller	f14428b25c	Allow spam-checker modules to be provide async methods. (#8890) ... Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.	4 years ago
Erik Johnston	427ede619f	Add metrics for tracking 3PID /requestToken requests. (#8712) ... The main use case is to see how many requests are being made, and how many are second/third/etc attempts. If there are large number of retries then that likely indicates a delivery problem.	4 years ago
Erik Johnston	2b7c180879	Start fewer opentracing spans (#8640) ... #8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously. This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary. I don't really know how much this will help.	4 years ago
Patrick Cloke	34a5696f93	Fix typos and spelling errors. (#8639)	4 years ago
Erik Johnston	1fcdbeb3ab	Start an opentracing span for background processes. (#8567) ... This should reduce the number of `There was no active span` errors we see. Fixes #8510.	4 years ago
Richard van der Hoff	6d2d42f8fb	Rewrite BucketCollector ... This was a bit unweildy for what I wanted: in particular, I wanted to assign each measurement straight into a bucket, rather than storing an intermediate Counter which didn't do any bucketing at all. I've replaced it with something that is hopefully a bit easier to use. (I'm not entirely sure what the difference between a HistogramMetricFamily and a GaugeHistogramMetricFamily is, but given our counters can go down as well as up the latter *sounds* more accurate?)	4 years ago
Richard van der Hoff	1c8ca2c543	Fix _exposition.py to stop stripping samples ... Our hacked-up `_exposition.py` was stripping out some samples it shouldn't have been. Put them back in, to more closely match the upstream `exposition.py`.	4 years ago
Richard van der Hoff	ceafb5a1c6	Drop support for ancient prometheus_client (#8426) ... Drop compatibility hacks for prometheus-client pre 0.4.0. Debian stretch and Fedora 31 both have newer versions, so hopefully this will be ok.	4 years ago
Patrick Cloke	aec294ee0d	Use slots in attrs classes where possible (#8296) ... slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.	4 years ago
Patrick Cloke	c619253db8	Stop sub-classing object (#8249)	4 years ago
Patrick Cloke	d89692ea84	Convert runWithConnection to async. (#8121)	4 years ago
Patrick Cloke	c36228c403	Convert run_as_background_process inner function to async. (#8032)	4 years ago
Richard van der Hoff	8ca39bd2c3	Improve stacktraces from exceptions in background processes (#7808) ... use `Failure()` to fish out the real exception.	4 years ago
Erik Johnston	a99658074d	Add some metrics for inbound and outbound federation processing times (#7755)	4 years ago
Christian Svensson	8bbe87f42d	Set Content-Length for Metrics requests (#7730) ... HTTP requires the response to contain a Content-Length header unless chunked encoding is being used. Prometheus metrics endpoint did not set this, causing software such as prometheus-proxy to not be able to scrape synapse for metrics. Signed-off-by: Christian Svensson <blue@cmd.nu>	4 years ago
Patrick Cloke	bd6dc17221	Replace iteritems/itervalues/iterkeys with native versions. (#7692)	5 years ago
Erik Johnston	f5353eff21	Make inflight background metrics more efficient. (#7597) ... Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	5 years ago
Ivan Shapovalov	ac481a738e	synapse.metrics: implement detailed memory usage reporting on PyPy (#7536) ... PyPy's gc.get_stats() returns an object containing detailed allocator statistics which could be beneficial to collect as metrics. Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>	5 years ago
Amber Brown	7cb8b4bc67	Allow configuration of Synapse's cache without using synctl or environment variables (#6391)	5 years ago
Richard van der Hoff	8c75667ad7	Add prometheus metrics for the number of active pushers (#7103)	5 years ago
Patrick Cloke	509e381afa	Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957) ... Ensure good comprehension hygiene using flake8-comprehensions.	5 years ago
Amber Brown	864f144543	Fix up some typechecking (#6150) ... * type checking fixes * changelog	5 years ago