176 Commits (5dd73d029eff32668b3ca69b7fb8529fc7c58745)

Author	SHA1	Message	Date
Patrick Cloke	4e874ed593	Remove unnecessary maybeDeferred calls (#8044)	5 years ago
Patrick Cloke	38e1fac886	Fix some spelling mistakes / typos. (#7811)	5 years ago
Dagfinn Ilmari Mannsåker	a3f11567d9	Replace all remaining six usage with native Python 3 equivalents (#7704)	5 years ago
Patrick Cloke	bd6dc17221	Replace iteritems/itervalues/iterkeys with native versions. (#7692)	5 years ago
Erik Johnston	eefc6b3a0d	Don't apply cache factor to event cache. (#7578) ... This is already correctly done when we instansiate the cache, but wasn't when it got reloaded (which always happens at least once on startup).	5 years ago
Richard van der Hoff	d4676910c9	remove miscellaneous PY2 code	5 years ago
Amber Brown	7cb8b4bc67	Allow configuration of Synapse's cache without using synctl or environment variables (#6391)	5 years ago
Erik Johnston	f9073893af	Speed up fetching device lists changes in sync. ... Currently we copy `users_who_share_room` needlessly about three times, which is expensive when the set is large (which it can easily be).	5 years ago
Richard van der Hoff	13683a3a22	Extend StreamChangeCache to support multiple entities per stream ID (#7303) ... First some background: StreamChangeCache is used to keep track of what "entities" have changed since a given stream ID. So for example, we might use it to keep track of when the last to-device message for a given user was received [1], and hence whether we need to pull any to-device messages from the database on a sync [2]. Now, it turns out that StreamChangeCache didn't support more than one thing being changed at a given stream_id (this was part of the problem with #7206). However, it's entirely valid to send to-device messages to more than one user at a time. As it turns out, this did in fact work, because *some* methods of StreamChangeCache coped ok with having multiple things changing on the same stream ID, and it seems we never actually use the methods which don't work on the stream change caches where we allow multiple changes at the same stream ID. But that feels horribly fragile, hence: let's update StreamChangeCache to properly support this, and add some typing and some more tests while we're at it. [1]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L301 [2]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L47-L51	5 years ago
Richard van der Hoff	0f8f02bc39	On catchup, process each row with its own stream id (#7286) ... Other parts of the code (such as the StreamChangeCache) assume that there will not be multiple changes with the same stream id. This code was introduced in #7024, and I hope this fixes #7206.	5 years ago
Erik Johnston	ed630ea17c	Reduce amount of logging at INFO level. (#6862) ... A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>	5 years ago
Hubert Chathi	cb2db17994	look up cross-signing keys from the DB in bulk (#6486)	5 years ago
Erik Johnston	f166a8d1f5	Remove SnapshotCache in favour of ResponseCache	5 years ago
V02460	affcc2cc36	Fix LruCache callback deduplication (#6213)	5 years ago
Andrew Morgan	54fef094b3	Remove usage of deprecated logger.warn method from codebase (#6271) ... Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.	5 years ago
Erik Johnston	e6c7e239ef	Update docstring	5 years ago
Erik Johnston	d0d8a22c13	Quick fix to ensure cache descriptors always return deferreds	5 years ago
Amber Brown	864f144543	Fix up some typechecking (#6150) ... * type checking fixes * changelog	5 years ago
Erik Johnston	17e1e80726	Retry well-known lookup before expiry. ... This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.	6 years ago
Richard van der Hoff	618bd1ee76	Fix some error cases in the caching layer. (#5749) ... There was some inconsistent behaviour in the caching layer around how exceptions were handled - particularly synchronously-thrown ones. This seems to be most easily handled by pushing the creation of ObservableDeferreds down from CacheDescriptor to the Cache.	6 years ago
Richard van der Hoff	418635e68a	Add a prometheus metric for active cache lookups. (#5750) ... * Add a prometheus metric for active cache lookups. * changelog	6 years ago
Amber Brown	4806651744	Replace returnValue with return (#5736)	6 years ago
Amber Brown	463b072b12	Move logging utilities out of the side drawer of util/ and into logging/ (#5606)	6 years ago
Andrew Morgan	ef8c62758c	Prevent multiple upgrades on the same room at once (#5051) ... Closes #4583 Does slightly less than #5045, which prevented a room from being upgraded multiple times, one after another. This PR still allows that, but just prevents two from happening at the same time. Mostly just to mitigate the fact that servers are slow and it can take a moment for the room upgrade to actually complete. We don't want people sending another request to upgrade the room when really they just thought the first didn't go through.	6 years ago
Amber Brown	32e7c9e7f2	Run Black. (#5482)	6 years ago
Richard van der Hoff	bc5f6e1797	Add a caching layer to .well-known responses (#4516)	6 years ago
Amber Brown	e1728dfcbe	Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068)	6 years ago
Erik Johnston	4f3e3ac192	Correctly match 'dict.pop' api	6 years ago
Erik Johnston	8ea887856c	Don't update eviction metrics on explicit removal	6 years ago
Richard van der Hoff	7ee94fc1ba	Log which cache is throwing exceptions	7 years ago
Erik Johnston	3baf6e1667	Fix ExpiringCache.__len__ to be accurate ... It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch. (This appears to have been a longstanding bug, but one which has been made more of a problem by #3932 and #3933). (This was originally done by Erik as part of #3933. I'm cherry-picking it because really it's a fix in its own right)	7 years ago
Erik Johnston	19dc676d1a	Fix ExpiringCache.__len__ to be accurate ... It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch.	7 years ago
Erik Johnston	fdd1a62e8d	Add a five minute cache to get_destination_retry_timings ... Hopefully helps with #3931	7 years ago
Erik Johnston	79eded1ae4	Make ExpiringCache slightly more performant	7 years ago
Erik Johnston	8601c24287	Fix some instances of ExpiringCache not expiring cache items ... ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.	7 years ago
Amber Brown	b37c472419	Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)	7 years ago
Richard van der Hoff	a8cbce0ced	fix invalidation	7 years ago
Richard van der Hoff	f102c05856	Rewrite cache list decorator ... Because it was complicated and annoyed me. I suspect this will be more efficient too.	7 years ago
Richard van der Hoff	03751a6420	Fix some looping_call calls which were broken in #3604 ... It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.	7 years ago
Richard van der Hoff	667fba68f3	Run things as background processes ... This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).	7 years ago
Erik Johnston	b2aa05a8d6	Use efficient .intersection	7 years ago
Erik Johnston	547b1355d3	Fix perf regression in PR #3530 ... The get_entities_changed function was changed to return all changed entities since the given stream position, rather than only those changed from a given list of entities. This resulted in the function incorrectly returning large numbers of entities that, for example, caused large increases in database usage.	7 years ago
Erik Johnston	77b692e65d	Don't return unknown entities in get_entities_changed ... The stream cache keeps track of all entities that have changed since a particular stream position, so get_entities_changed does not need to return unknown entites when given a larger stream position. This makes it consistent with the behaviour of has_entity_changed.	7 years ago
Richard van der Hoff	fa5c2bc082	Reduce set building in get_entities_changed ... This line shows up as about 5% of cpu time on a synchrotron: not_known_entities = set(entities) - set(self._entity_to_key) Presumably the problem here is that _entity_to_key can be largeish, and building a set for its keys every time this function is called is slow. Here we rewrite the logic to avoid building so many sets.	7 years ago
Amber Brown	49af402019	run isort	7 years ago
Amber Brown	72d2143ea8	Revert "Revert "Try to not use as much CPU in the StreamChangeCache"" (#3454)	7 years ago
Matthew Hodgson	8057489b26	Revert "Try to not use as much CPU in the StreamChangeCache"	7 years ago
Amber Brown	1202508067	fixes	7 years ago
Amber Brown	bd3d329c88	fixes	7 years ago
Amber Brown	abfe4b2957	try and make loading items from the cache faster	7 years ago