480 Commits (1ccafb0c5e3e6c9c64ce231163e7c73f05b910fa)

Author	SHA1	Message	Date
Erik Johnston	4f3e3ac192	Correctly match 'dict.pop' api	6 years ago
Erik Johnston	8ea887856c	Don't update eviction metrics on explicit removal	6 years ago
Richard van der Hoff	4a15a3e4d5	Include eventid in log lines when processing incoming federation transactions (#3959) ... when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.	6 years ago
Richard van der Hoff	7ee94fc1ba	Log which cache is throwing exceptions	6 years ago
Erik Johnston	3baf6e1667	Fix ExpiringCache.__len__ to be accurate ... It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch. (This appears to have been a longstanding bug, but one which has been made more of a problem by #3932 and #3933). (This was originally done by Erik as part of #3933. I'm cherry-picking it because really it's a fix in its own right)	6 years ago
Erik Johnston	19dc676d1a	Fix ExpiringCache.__len__ to be accurate ... It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch.	6 years ago
Erik Johnston	fdd1a62e8d	Add a five minute cache to get_destination_retry_timings ... Hopefully helps with #3931	6 years ago
Erik Johnston	79eded1ae4	Make ExpiringCache slightly more performant	6 years ago
Erik Johnston	8601c24287	Fix some instances of ExpiringCache not expiring cache items ... ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.	6 years ago
Richard van der Hoff	642199570c	Improve the logging when handling a federation transaction (#3904) ... Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.	6 years ago
Erik Johnston	9407bcf37a	Replace custom DeferredTimeoutError with defer.TimeoutError	6 years ago
Erik Johnston	6c48aa0256	Run canceller first to allow it to generate correct error	6 years ago
Erik Johnston	a334e1cace	Update to use new timeout function everywhere. ... The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.	6 years ago
Erik Johnston	24efb2a70d	Fix timeout function ... Turns out deferred.cancel sometimes throws, so we do that last to ensure that we always do resolve the new deferred.	6 years ago
Erik Johnston	fcfe7a850d	Add an awful secondary timeout to fix wedged requests ... This is an attempt to mitigate #3842 by adding yet-another-timeout	6 years ago
Erik Johnston	0a81038ea0	Add in flight real time metrics for Measure blocks	6 years ago
Erik Johnston	9e05c8d309	Change the manhole SSH key to have more bits ... Newer versions of openssh client refuse to connect to the old key due to its length.	6 years ago
Richard van der Hoff	be6527325a	Fix exceptions when a connection is closed before we read the headers ... This fixes bugs introduced in #3700, by making sure that we behave sanely when an incoming connection is closed before the headers are read.	6 years ago
Richard van der Hoff	55e6bdf287	Robustness fix for logcontext filter ... Make the logcontext filter not explode if it somehow ends up with a logcontext of None, since that infinite-loops the whole logging system.	6 years ago
Amber Brown	324525f40c	Port over enough to get some sytests running on Python 3 (#3668)	6 years ago
Amber Brown	b37c472419	Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)	6 years ago
Richard van der Hoff	638d35ef08	Fix linearizer cancellation on twisted < 18.7 ... Turns out that cancellation of inlineDeferreds didn't really work properly until Twisted 18.7. This commit refactors Linearizer.queue to avoid inlineCallbacks.	6 years ago
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569)	6 years ago
Richard van der Hoff	a8cbce0ced	fix invalidation	6 years ago
Richard van der Hoff	f102c05856	Rewrite cache list decorator ... Because it was complicated and annoyed me. I suspect this will be more efficient too.	6 years ago
Richard van der Hoff	03751a6420	Fix some looping_call calls which were broken in #3604 ... It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.	6 years ago
Richard van der Hoff	3d6df84658	Test and fix support for cancellation in Linearizer	6 years ago
Richard van der Hoff	7c712f95bb	Combine Limiter and Linearizer ... Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.	6 years ago
Richard van der Hoff	8462c26485	Improvements to the Limiter ... * give them names, to improve logging * use a deque rather than a list for efficiency	6 years ago
Richard van der Hoff	d7275eecf3	Add a sleep to the Limiter to fix stack overflows. ... Fixes #3570	6 years ago
Amber Brown	95ccb6e2ec	Don't spew errors because we can't save metrics (#3563)	6 years ago
Richard van der Hoff	8c69b735e3	Make Distributor run its processes as a background process ... This is more involved than it might otherwise be, because the current implementation just drops its logcontexts and runs everything in the sentinel context. It turns out that we aren't actually using a bunch of the functionality here (notably suppress_failures and the fact that Distributor.fire returns a deferred), so the easiest way to fix this is actually by simplifying a bunch of code.	6 years ago
Richard van der Hoff	667fba68f3	Run things as background processes ... This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).	6 years ago
Erik Johnston	b2aa05a8d6	Use efficient .intersection	6 years ago
Erik Johnston	547b1355d3	Fix perf regression in PR #3530 ... The get_entities_changed function was changed to return all changed entities since the given stream position, rather than only those changed from a given list of entities. This resulted in the function incorrectly returning large numbers of entities that, for example, caused large increases in database usage.	6 years ago
Richard van der Hoff	33b40d0a25	Make FederationRateLimiter queue requests properly ... popitem removes the *most recent* item by default [1]. We want the oldest. Fixes #3524 [1]: https://docs.python.org/2/library/collections.html#collections.OrderedDict.popitem	6 years ago
Erik Johnston	77b692e65d	Don't return unknown entities in get_entities_changed ... The stream cache keeps track of all entities that have changed since a particular stream position, so get_entities_changed does not need to return unknown entites when given a larger stream position. This makes it consistent with the behaviour of has_entity_changed.	6 years ago
Richard van der Hoff	fa5c2bc082	Reduce set building in get_entities_changed ... This line shows up as about 5% of cpu time on a synchrotron: not_known_entities = set(entities) - set(self._entity_to_key) Presumably the problem here is that _entity_to_key can be largeish, and building a set for its keys every time this function is called is slow. Here we rewrite the logic to avoid building so many sets.	6 years ago
Richard van der Hoff	c3c29aa196	Attempt to include db threads in cpu usage stats (#3496) ... Let's try to include time spent in the DB threads in the per-request/block cpu usage metrics.	6 years ago
Richard van der Hoff	55370331da	Refactor logcontext resource usage tracking (#3501) ... Factor out the resource usage tracking out to a separate object, which can be passed around and copied independently of the logcontext itself.	6 years ago
Amber Brown	49af402019	run isort	6 years ago
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	7 years ago
Amber Brown	72d2143ea8	Revert "Revert "Try to not use as much CPU in the StreamChangeCache"" (#3454)	7 years ago
Matthew Hodgson	8057489b26	Revert "Try to not use as much CPU in the StreamChangeCache"	7 years ago
Amber Brown	1202508067	fixes	7 years ago
Amber Brown	bd3d329c88	fixes	7 years ago
Amber Brown	abfe4b2957	try and make loading items from the cache faster	7 years ago
Amber Brown	07cad26d65	Remove all global reactor imports & pass it around explicitly (#3424)	7 years ago
Richard van der Hoff	43e02c409d	Disable partial state group caching for wildcard lookups ... When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.	7 years ago
Richard van der Hoff	0495fe0035	Indirect evt_count updates via method call ... so that we can stub it for the sentinel and not have a billion failing UTs	7 years ago