508 Commits (14f13babb00d64009b11ef822ebe6fafe044eebd)

Author	SHA1	Message	Date
Richard van der Hoff	3d6df84658	Test and fix support for cancellation in Linearizer	6 years ago
Richard van der Hoff	7c712f95bb	Combine Limiter and Linearizer ... Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.	6 years ago
Richard van der Hoff	8462c26485	Improvements to the Limiter ... * give them names, to improve logging * use a deque rather than a list for efficiency	6 years ago
Richard van der Hoff	d7275eecf3	Add a sleep to the Limiter to fix stack overflows. ... Fixes #3570	6 years ago
Amber Brown	95ccb6e2ec	Don't spew errors because we can't save metrics (#3563)	6 years ago
Richard van der Hoff	8c69b735e3	Make Distributor run its processes as a background process ... This is more involved than it might otherwise be, because the current implementation just drops its logcontexts and runs everything in the sentinel context. It turns out that we aren't actually using a bunch of the functionality here (notably suppress_failures and the fact that Distributor.fire returns a deferred), so the easiest way to fix this is actually by simplifying a bunch of code.	6 years ago
Richard van der Hoff	667fba68f3	Run things as background processes ... This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).	6 years ago
Erik Johnston	b2aa05a8d6	Use efficient .intersection	6 years ago
Erik Johnston	547b1355d3	Fix perf regression in PR #3530 ... The get_entities_changed function was changed to return all changed entities since the given stream position, rather than only those changed from a given list of entities. This resulted in the function incorrectly returning large numbers of entities that, for example, caused large increases in database usage.	6 years ago
Richard van der Hoff	33b40d0a25	Make FederationRateLimiter queue requests properly ... popitem removes the *most recent* item by default [1]. We want the oldest. Fixes #3524 [1]: https://docs.python.org/2/library/collections.html#collections.OrderedDict.popitem	6 years ago
Erik Johnston	77b692e65d	Don't return unknown entities in get_entities_changed ... The stream cache keeps track of all entities that have changed since a particular stream position, so get_entities_changed does not need to return unknown entites when given a larger stream position. This makes it consistent with the behaviour of has_entity_changed.	6 years ago
Richard van der Hoff	fa5c2bc082	Reduce set building in get_entities_changed ... This line shows up as about 5% of cpu time on a synchrotron: not_known_entities = set(entities) - set(self._entity_to_key) Presumably the problem here is that _entity_to_key can be largeish, and building a set for its keys every time this function is called is slow. Here we rewrite the logic to avoid building so many sets.	6 years ago
Richard van der Hoff	c3c29aa196	Attempt to include db threads in cpu usage stats (#3496) ... Let's try to include time spent in the DB threads in the per-request/block cpu usage metrics.	6 years ago
Richard van der Hoff	55370331da	Refactor logcontext resource usage tracking (#3501) ... Factor out the resource usage tracking out to a separate object, which can be passed around and copied independently of the logcontext itself.	6 years ago
Amber Brown	49af402019	run isort	6 years ago
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	7 years ago
Amber Brown	72d2143ea8	Revert "Revert "Try to not use as much CPU in the StreamChangeCache"" (#3454)	7 years ago
Matthew Hodgson	8057489b26	Revert "Try to not use as much CPU in the StreamChangeCache"	7 years ago
Amber Brown	1202508067	fixes	7 years ago
Amber Brown	bd3d329c88	fixes	7 years ago
Amber Brown	abfe4b2957	try and make loading items from the cache faster	7 years ago
Amber Brown	07cad26d65	Remove all global reactor imports & pass it around explicitly (#3424)	7 years ago
Richard van der Hoff	43e02c409d	Disable partial state group caching for wildcard lookups ... When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.	7 years ago
Richard van der Hoff	0495fe0035	Indirect evt_count updates via method call ... so that we can stub it for the sentinel and not have a billion failing UTs	7 years ago
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	7 years ago
Richard van der Hoff	b088aafcae	Log number of events fetched from DB ... When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]	7 years ago
Amber Brown	a61738b316	Remove run_on_reactor (#3395)	7 years ago
Amber Brown	f7869f8f8b	Port to sortedcontainers (with tests!) (#3332)	7 years ago
Erik Johnston	042eedfa2b	Add hacky cache factor override system	7 years ago
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	7 years ago
Adrian Tschira	7873cde526	pep8	7 years ago
Amber Brown	57ad76fa4a	fix up tests	7 years ago
Amber Brown	3ef5cd74a6	update to more consistently use seconds in any metrics or logging	7 years ago
Amber Brown	357c74a50f	add comment about why unreg	7 years ago
Adrian Tschira	4ee4450d66	fix recursion error	7 years ago
Adrian Tschira	dd068ca979	remaining isintance fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	7 years ago
Amber Brown	53cc2cde1f	cleanup	7 years ago
Amber Brown	071206304d	cleanup pep8 errors	7 years ago
Amber Brown	85ba83eb51	fixes	7 years ago
Erik Johnston	7948ecf234	Comment	7 years ago
Erik Johnston	020377a550	Fix logcontext resource usage tracking	7 years ago
Amber Brown	df9f72d9e5	replacing portions	7 years ago
Adrian Tschira	45b55e23d3	Add batch_iter to utils ... There's a frequent idiom I noticed where an iterable is split up into a number of chunks/batches. Unfortunately that method does not work with iterators like dict.keys() in python3. This implementation works with iterators. Signed-off-by: Adrian Tschira <nota@notafile.com>	7 years ago
Adrian Tschira	73cbdef5f7	fix py3 intern and remove unnecessary py3 encode ... Signed-off-by: Adrian Tschira <nota@notafile.com>	7 years ago
Richard van der Hoff	a7fe62f0cb	Fix logcontext leaks in rate limiter	7 years ago
Richard van der Hoff	f22e7cda2c	Fix a class of logcontext leaks ... So, it turns out that if you have a first `Deferred` `D1`, you can add a callback which returns another `Deferred` `D2`, and `D2` must then complete before any further callbacks on `D1` will execute (and later callbacks on `D1` get the *result* of `D2` rather than `D2` itself). So, `D1` might have `called=True` (as in, it has started running its callbacks), but any new callbacks added to `D1` won't get run until `D2` completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield` will 'block'. In conclusion: some of our assumptions in `logcontext` were invalid. We need to make sure that we don't optimise out the logcontext juggling when this situation happens. Fortunately, it is easy to detect by checking `D1.paused`.	7 years ago
Richard van der Hoff	e482f8cd85	Fix incorrect reference to StringIO ... This was introduced in 4f2f5171	7 years ago
Adrian Tschira	e9143b6593	more bytes strings ... Signed-off-by: Adrian Tschira <nota@notafile.com>	7 years ago
Adrian Tschira	d82b6ea9e6	Move more xrange to six ... plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>	7 years ago
Adrian Tschira	4f2f5171b7	replace stringIO imports	7 years ago