337 Commits (084afbb6a06f2661bed503bf49b0291ce999c6c1)

Author	SHA1	Message	Date
Richard van der Hoff	b2cd6accf5	Remove __PreservingContextDeferred too	7 years ago
Richard van der Hoff	7e6fa29cb5	Remove preserve_context_over_{fn, deferred} ... Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.	7 years ago
Richard van der Hoff	bf993db11c	Logging and logcontext fixes for Limiter ... Add some logging to the Limiter in a similar spirit to the Linearizer, to help debug issues. Also fix a logcontext leak. Also refactor slightly to avoid throwing exceptions.	7 years ago
Erik Johnston	2a7e9faeec	Do logcontexts outside ResponseCache	7 years ago
Erik Johnston	33a9026cdf	Add logging and fix log contexts for publicRooms	7 years ago
Richard van der Hoff	0be99858f3	fix vars named `l` ... E741 says "do not use variables named ‘l’, ‘O’, or ‘I’".	7 years ago
Richard van der Hoff	eaaabc6c4f	replace 'except:' with 'except Exception:' ... what could possibly go wrong	7 years ago
Richard van der Hoff	2e9f5ea31a	Fix logcontext handling for persist_events ... * don't use preserve_context_over_deferred, which is known broken. * remove a redundant preserve_fn. * add/improve some comments	7 years ago
Richard van der Hoff	f30c4ed2bc	logformatter: fix AttributeError ... make sure we have the relevant fields before we try to log them.	7 years ago
Richard van der Hoff	4fad8efbfb	Fix stackoverflow and logcontexts from linearizer ... 1. make it not blow out the stack when there are more than 50 things waiting for a lock. Fixes https://github.com/matrix-org/synapse/issues/2505. 2. Make it not mess up the log contexts.	7 years ago
Richard van der Hoff	3cc852d339	Fancy logformatter to format exceptions better ... This is a bit of an experimental change at this point; the idea is to see if it helps us track down where our stack overflows are coming from by logging the stack when the exception was caught and turned into a Failure. (We'll also need edf2704420). If we deploy this, we'll be able to enable it via the log config yaml.	7 years ago
Richard van der Hoff	148428ce76	Fix logcontext handling for concurrently_execute ... Avoid preserve_context_over_deferred, which is broken.	7 years ago
David Baker	60c78666ab	pep8	7 years ago
David Baker	8ad5f34908	pep8	7 years ago
David Baker	9fd086e506	unnecessary parens	7 years ago
David Baker	0b03a97708	Add module_loader.py	7 years ago
Erik Johnston	495f075b41	Increase default cache factor size.	7 years ago
Erik Johnston	b5e8d529e6	Define CACHE_SIZE_FACTOR once	7 years ago
Erik Johnston	c72058bcc6	Use an ExpiringCache for storing registration sessions ... This is because pruning them was a significant performance drain on matrix.org	8 years ago
Erik Johnston	efc2b7db95	Rewrite conditional	8 years ago
Erik Johnston	eed59dcc1e	Fix has_any_entity_changed ... Occaisonally has_any_entity_changed would throw the error: "Set changed size during iteration" when taking the max of the `sorteddict`. While its uncertain how that happens, its quite inefficient to iterate over the entire dict anyway so we change to using the more traditional `bisect_*` functions.	8 years ago
Erik Johnston	304880d185	Add stream change cache	8 years ago
Erik Johnston	bd7bb5df71	Pull out if statement from for loop	8 years ago
Erik Johnston	e3417a06e2	Update list cache to handle one arg case ... We update the normal cache descriptors to handle caches with a single argument specially so that the key wasn't a 1-tuple. We need to update the cache list to be aware of this.	8 years ago
Erik Johnston	bbfe4e996c	Make get_state_groups_from_groups faster. ... Most of the time was spent copying a dict to filter out sentinel values that indicated that keys did not exist in the dict. The sentinel values were added to ensure that we cached the non-existence of keys. By updating DictionaryCache to keep track of which keys were known to not exist itself we can remove a dictionary copy.	8 years ago
Erik Johnston	ffad4fe35b	Don't update event cache hit ratio from get_joined_users ... Otherwise the hit ration of plain get_events gets completely skewed by calls to get_joined_users* functions.	8 years ago
Erik Johnston	d2d8ed4884	Optimise caches with single key	8 years ago
Richard van der Hoff	2e996271fe	Instantiate DeferredTimedOutError correctly ... Call `super` correctly, so that we correctly initialise the `errcode` field. Fixes https://github.com/matrix-org/synapse/issues/2179.	8 years ago
Erik Johnston	d9aa645f86	Reduce size of joined_user cache ... The _get_joined_users_from_context cache stores a mapping from user_id to avatar_url and display_name. Instead of storing those in a dict, store them in a namedtuple as that uses much less memory. We also try converting the string to ascii to further reduce the size.	8 years ago
Erik Johnston	efab1dadde	Remove DEBUG_CACHES	8 years ago
Erik Johnston	119cb9bbcf	Reduce cache size by not storing deferreds ... Currently the cache descriptors store deferreds rather than raw values, this is a simple way of triggering only one database hit and sharing the result if two callers attempt to get the same value. However, there are a few caches that simply store a mapping from string to string (or int). These caches can have a large number of entries, under the assumption that each entry is small. However, the size of a deferred (specifically the size of ObservableDeferred) is signigicantly larger than that of the raw value, 2kb vs 32b. This PR therefore changes the cache descriptors to store the raw values rather than the deferreds. As a side effect cached storage function now either return a deferred or the actual value, as the cached list decriptor already does. This is fine as we always end up just yield'ing on the returned value eventually, which handles that case correctly.	8 years ago
Erik Johnston	d134d0935e	Only intern ascii strings	8 years ago
Richard van der Hoff	e2eebf1696	Fix fixme in preserve_fn ... `preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a fixme therein.	8 years ago
Erik Johnston	4d17add8de	Remove unused instance variable	8 years ago
Erik Johnston	5b5b171f3e	Docs	8 years ago
Erik Johnston	b282fe7170	Revert log context change	8 years ago
Erik Johnston	6194a64ae9	Doc new instance variables	8 years ago
Erik Johnston	014fee93b3	Manually calculate cache key as getcallargs is expensive ... This is because getcallargs recomputes the getargspec, amongst other things, which we don't need to do as its already been done	8 years ago
Erik Johnston	86780a8bc3	Don't convert to deferreds when not necessary	8 years ago
Richard van der Hoff	f9b4bb05e0	Fix the logcontext handling in the cache wrappers (#2077) ... The cache wrappers had a habit of leaking the logcontext into the reactor while the lookup function was running, and then not restoring it correctly when the lookup function had completed. It's all the fault of `preserve_context_over_{fn,deferred}` which are basically a bit broken.	8 years ago
Richard van der Hoff	5a16cb4bf0	Ignore backoff history for invites, aliases, and roomdirs ... Add a param to the federation client which lets us ignore historical backoff data for federation queries, and set it for a handful of operations.	8 years ago
Richard van der Hoff	b88a323ffb	Fix time_bound_deferred to throw the right exception ... Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred would throw a CancelledError when the deferred timed out, which was rather confusing.	8 years ago
Richard van der Hoff	4bd597d9fc	push federation retry limiter down to matrixfederationclient ... rather than having to instrument everywhere we make a federation call, make the MatrixFederationHttpClient manage the retry limiter.	8 years ago
Richard van der Hoff	19b9366d73	Fix a couple of logcontext leaks ... Use preserve_fn to correctly manage the logcontexts around things we don't want to yield on.	8 years ago
Richard van der Hoff	95f21c7a66	Fix caching of remote servers' signature keys ... The `@cached` decorator on `KeyStore._get_server_verify_key` was missing its `num_args` parameter, which meant that it was returning the wrong key for any server which had more than one recorded key. By way of a fix, change the default for `num_args` to be *all* arguments. To implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.	8 years ago
Richard van der Hoff	f40c2db05a	Stop preserve_fn leaking context into the reactor ... Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the reactor, and add a test for it. Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do the same thing but had its own, different, set of bugs.	8 years ago
Richard van der Hoff	d2d146a314	Logcontext docs	8 years ago
Richard van der Hoff	29ed09e80a	Fix assertion to stop transaction queue getting wedged ... ... and update some docstrings to correctly reflect the types being used. get_new_device_msgs_for_remote can return a long under some circumstances, which was being stored in last_device_list_stream_id_by_dest, and was then upsetting things on the next loop.	8 years ago
Richard van der Hoff	b5d1c68beb	Implement reset_context_after_deferred ... to correctly reset the context when we fire off a deferred we aren't going to wait for.	8 years ago
David Baker	73a5f06652	Support registration / login with phone number ... Changes from https://github.com/matrix-org/synapse/pull/1971	8 years ago