355 Commits (a18828c12961ce254660e7641335428ec0655868)

Author	SHA1	Message	Date
Matthew Hodgson	ab9f844aaf	Add federation_domain_whitelist option (#2820) ... Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network	7 years ago
Matthew Hodgson	8fe253f19b	fix PR nitpicking	7 years ago
Matthew Hodgson	447f4f0d5f	rewrite based on PR feedback: ... * [ ] split config options into allowed_local_3pids and registrations_require_3pid * [ ] simplify and comment logic for picking registration flows * [ ] fix docstring and move check_3pid_allowed into a new util module * [ ] use check_3pid_allowed everywhere @erikjohnston PTAL	7 years ago
Richard van der Hoff	d57765fc8a	Fix bugs in block metrics ... ... which I introduced in #2785	7 years ago
Erik Johnston	be0dfcd4a2	Do logcontexts correctly	7 years ago
Erik Johnston	1432f7ccd5	Move test stuff to tests	7 years ago
Erik Johnston	2f18a2647b	Make all fields private	7 years ago
Erik Johnston	dc519602ac	Ensure we registerProducer isn't called twice	7 years ago
Erik Johnston	17b54389fe	Fix _notify_empty typo	7 years ago
Erik Johnston	28b338ed9b	Move definition of paused_producer to __init__	7 years ago
Erik Johnston	a177325b49	Fix comments	7 years ago
Erik Johnston	bc67e7d260	Add decent impl of a FileConsumer ... Twisted core doesn't have a general purpose one, so we need to write one ourselves. Features: - All writing happens in background thread - Supports both push and pull producers - Push producers get paused if the consumer falls behind	7 years ago
Richard van der Hoff	3d12d97415	Track DB scheduling delay per-request ... For each request, track the amount of time spent waiting for a db connection. This entails adding it to the LoggingContext and we may as well add metrics for it while we are passing.	7 years ago
Richard van der Hoff	6324b65f08	Track db txn time in millisecs ... ... to reduce the amount of floating-point foo we do.	7 years ago
Richard van der Hoff	44a498418c	Optimise LoggingContext creation and copying ... It turns out that the only thing we use the __dict__ of LoggingContext for is `request`, and given we create lots of LoggingContexts and then copy them every time we do a db transaction or log line, using the __dict__ seems a bit redundant. Let's try to optimise things by making the request attribute explicit.	7 years ago
Richard van der Hoff	39f4e29d01	Reorganise request and block metrics ... In order to circumvent the number of duplicate foo:count metrics increasing without bounds, it's time for a rearrangement. The following are all deprecated, and replaced with synapse_util_metrics_block_count: synapse_util_metrics_block_timer:count synapse_util_metrics_block_ru_utime:count synapse_util_metrics_block_ru_stime:count synapse_util_metrics_block_db_txn_count:count synapse_util_metrics_block_db_txn_duration:count The following are all deprecated, and replaced with synapse_http_server_response_count: synapse_http_server_requests synapse_http_server_response_time:count synapse_http_server_response_ru_utime:count synapse_http_server_response_ru_stime:count synapse_http_server_response_db_txn_count:count synapse_http_server_response_db_txn_duration:count The following are renamed (the old metrics are kept for now, but deprecated): synapse_util_metrics_block_timer:total -> synapse_util_metrics_block_time_seconds synapse_util_metrics_block_ru_utime:total -> synapse_util_metrics_block_ru_utime_seconds synapse_util_metrics_block_ru_stime:total -> synapse_util_metrics_block_ru_stime_seconds synapse_util_metrics_block_db_txn_count:total -> synapse_util_metrics_block_db_txn_count synapse_util_metrics_block_db_txn_duration:total -> synapse_util_metrics_block_db_txn_duration_seconds synapse_http_server_response_time:total -> synapse_http_server_response_time_seconds synapse_http_server_response_ru_utime:total -> synapse_http_server_response_ru_utime_seconds synapse_http_server_response_ru_stime:total -> synapse_http_server_response_ru_stime_seconds synapse_http_server_response_db_txn_count:total -> synapse_http_server_response_db_txn_count synapse_http_server_response_db_txn_duration:total synapse_http_server_response_db_txn_duration_seconds	7 years ago
Richard van der Hoff	b2cd6accf5	Remove __PreservingContextDeferred too	7 years ago
Richard van der Hoff	7e6fa29cb5	Remove preserve_context_over_{fn, deferred} ... Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.	7 years ago
Richard van der Hoff	bf993db11c	Logging and logcontext fixes for Limiter ... Add some logging to the Limiter in a similar spirit to the Linearizer, to help debug issues. Also fix a logcontext leak. Also refactor slightly to avoid throwing exceptions.	7 years ago
Erik Johnston	2a7e9faeec	Do logcontexts outside ResponseCache	7 years ago
Erik Johnston	33a9026cdf	Add logging and fix log contexts for publicRooms	7 years ago
Richard van der Hoff	0be99858f3	fix vars named `l` ... E741 says "do not use variables named ‘l’, ‘O’, or ‘I’".	7 years ago
Richard van der Hoff	eaaabc6c4f	replace 'except:' with 'except Exception:' ... what could possibly go wrong	7 years ago
Richard van der Hoff	2e9f5ea31a	Fix logcontext handling for persist_events ... * don't use preserve_context_over_deferred, which is known broken. * remove a redundant preserve_fn. * add/improve some comments	7 years ago
Richard van der Hoff	f30c4ed2bc	logformatter: fix AttributeError ... make sure we have the relevant fields before we try to log them.	7 years ago
Richard van der Hoff	4fad8efbfb	Fix stackoverflow and logcontexts from linearizer ... 1. make it not blow out the stack when there are more than 50 things waiting for a lock. Fixes https://github.com/matrix-org/synapse/issues/2505. 2. Make it not mess up the log contexts.	7 years ago
Richard van der Hoff	3cc852d339	Fancy logformatter to format exceptions better ... This is a bit of an experimental change at this point; the idea is to see if it helps us track down where our stack overflows are coming from by logging the stack when the exception was caught and turned into a Failure. (We'll also need edf2704420). If we deploy this, we'll be able to enable it via the log config yaml.	7 years ago
Richard van der Hoff	148428ce76	Fix logcontext handling for concurrently_execute ... Avoid preserve_context_over_deferred, which is broken.	7 years ago
David Baker	60c78666ab	pep8	7 years ago
David Baker	8ad5f34908	pep8	7 years ago
David Baker	9fd086e506	unnecessary parens	7 years ago
David Baker	0b03a97708	Add module_loader.py	7 years ago
Erik Johnston	495f075b41	Increase default cache factor size.	8 years ago
Erik Johnston	b5e8d529e6	Define CACHE_SIZE_FACTOR once	8 years ago
Erik Johnston	c72058bcc6	Use an ExpiringCache for storing registration sessions ... This is because pruning them was a significant performance drain on matrix.org	8 years ago
Erik Johnston	efc2b7db95	Rewrite conditional	8 years ago
Erik Johnston	eed59dcc1e	Fix has_any_entity_changed ... Occaisonally has_any_entity_changed would throw the error: "Set changed size during iteration" when taking the max of the `sorteddict`. While its uncertain how that happens, its quite inefficient to iterate over the entire dict anyway so we change to using the more traditional `bisect_*` functions.	8 years ago
Erik Johnston	304880d185	Add stream change cache	8 years ago
Erik Johnston	bd7bb5df71	Pull out if statement from for loop	8 years ago
Erik Johnston	e3417a06e2	Update list cache to handle one arg case ... We update the normal cache descriptors to handle caches with a single argument specially so that the key wasn't a 1-tuple. We need to update the cache list to be aware of this.	8 years ago
Erik Johnston	bbfe4e996c	Make get_state_groups_from_groups faster. ... Most of the time was spent copying a dict to filter out sentinel values that indicated that keys did not exist in the dict. The sentinel values were added to ensure that we cached the non-existence of keys. By updating DictionaryCache to keep track of which keys were known to not exist itself we can remove a dictionary copy.	8 years ago
Erik Johnston	ffad4fe35b	Don't update event cache hit ratio from get_joined_users ... Otherwise the hit ration of plain get_events gets completely skewed by calls to get_joined_users* functions.	8 years ago
Erik Johnston	d2d8ed4884	Optimise caches with single key	8 years ago
Richard van der Hoff	2e996271fe	Instantiate DeferredTimedOutError correctly ... Call `super` correctly, so that we correctly initialise the `errcode` field. Fixes https://github.com/matrix-org/synapse/issues/2179.	8 years ago
Erik Johnston	d9aa645f86	Reduce size of joined_user cache ... The _get_joined_users_from_context cache stores a mapping from user_id to avatar_url and display_name. Instead of storing those in a dict, store them in a namedtuple as that uses much less memory. We also try converting the string to ascii to further reduce the size.	8 years ago
Erik Johnston	efab1dadde	Remove DEBUG_CACHES	8 years ago
Erik Johnston	119cb9bbcf	Reduce cache size by not storing deferreds ... Currently the cache descriptors store deferreds rather than raw values, this is a simple way of triggering only one database hit and sharing the result if two callers attempt to get the same value. However, there are a few caches that simply store a mapping from string to string (or int). These caches can have a large number of entries, under the assumption that each entry is small. However, the size of a deferred (specifically the size of ObservableDeferred) is signigicantly larger than that of the raw value, 2kb vs 32b. This PR therefore changes the cache descriptors to store the raw values rather than the deferreds. As a side effect cached storage function now either return a deferred or the actual value, as the cached list decriptor already does. This is fine as we always end up just yield'ing on the returned value eventually, which handles that case correctly.	8 years ago
Erik Johnston	d134d0935e	Only intern ascii strings	8 years ago
Richard van der Hoff	e2eebf1696	Fix fixme in preserve_fn ... `preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a fixme therein.	8 years ago
Erik Johnston	4d17add8de	Remove unused instance variable	8 years ago