256 Commits (8cced497640525ff1bed7011509b88306c7e7ea3)

Author	SHA1	Message	Date
Richard van der Hoff	71a1abb8a1	Stop the master relaying USER_SYNC for other workers (#7318) ... Long story short: if we're handling presence on the current worker, we shouldn't be sending USER_SYNC commands over replication. In an attempt to figure out what is going on here, I ended up refactoring some bits of the presencehandler code, so the first 4 commits here are non-functional refactors to move this code slightly closer to sanity. (There's still plenty to do here :/). Suggest reviewing individual commits. Fixes (I hope) #7257.	5 years ago
Erik Johnston	fdb1344716	Remove concept of a non-limited stream. (#7011)	5 years ago
Erik Johnston	1f773eec91	Port PresenceHandler to async/await (#6991)	5 years ago
Patrick Cloke	509e381afa	Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957) ... Ensure good comprehension hygiene using flake8-comprehensions.	5 years ago
Richard van der Hoff	ab4b4ee6a7	Fix an error which was thrown by the PresenceHandler _on_shutdown handler. (#6640)	5 years ago
Erik Johnston	2284eb3a53	Add database config class (#6513) ... This encapsulates config for a given database and is the way to get new connections.	5 years ago
Erik Johnston	5859a5c569	Fix presence timeouts when synchrotron restarts. (#6212) ... * Fix presence timeouts when synchrotron restarts. Handling timeouts would fail if there was an external process that had timed out, e.g. a synchrotron restarting. This was due to a couple of variable name typoes. Fixes #3715.	5 years ago
Richard van der Hoff	a139420a3c	Fix races in room stats (and other) updates. (#6187) ... Hopefully this will fix the occasional failures we were seeing in the room directory. The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8. We can solve this easily by only processing up to the stream_id where we know all events have been persisted.	5 years ago
Matthew Hodgson	8401bcd206	fix typo	5 years ago
Andrew Morgan	4548d1f87e	Remove unnecessary parentheses around return statements (#5931) ... Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)	5 years ago
Amber Brown	97a8b4caf7	Move some timeout checking logs to DEBUG #5785	5 years ago
Amber Brown	4806651744	Replace returnValue with return (#5736)	5 years ago
Amber Brown	463b072b12	Move logging utilities out of the side drawer of util/ and into logging/ (#5606)	5 years ago
Erik Johnston	915280f1ed	Fixup comment	5 years ago
Erik Johnston	8181e290a9	Fix sync tightloop bug. ... If, for some reason, presence updates take a while to persist then it can trigger clients to tightloop calling `/sync` due to the presence handler returning updates but not advancing the stream token. Fixes #5503.	6 years ago
Amber Brown	32e7c9e7f2	Run Black. (#5482)	6 years ago
Richard van der Hoff	71063a69b8	Fix missing logcontext for PresenceHandler.on_shutdown. (#5369) ... Fixes some warnings, and a scary-looking stacktrace when sytest kills the process.	6 years ago
Erik Johnston	75538813fc	Fix background updates to handle redactions/rejections (#5352) ... * Fix background updates to handle redactions/rejections In background updates based on current state delta stream we need to handle that we may not have all the events (or at least that `get_events` may raise an exception).	6 years ago
Richard van der Hoff	5726378ece	Fix "db txn 'update_presence' from sentinel context" log messages (#5275) ... Fixes #4414.	6 years ago
Richard van der Hoff	bd0d45ca69	Fix infinite loop in presence handler ... Fixes #5102	6 years ago
Andrew Morgan	4a4d5c4fd6	Fix grammar and document get_current_users_in_room (#4998)	6 years ago
Neil Johnson	e8419554ff	Remove presence lists (#4989) ... Remove presence list support as per MSC 1819	6 years ago
Erik Johnston	4e5f0f7ca0	Use an assert	6 years ago
Erik Johnston	40e56997bc	Review comments	6 years ago
Erik Johnston	197fae1639	Use event streams to calculate presence ... Primarily this fixes a bug in the handling of remote users joining a room where the server sent out the presence for all local users in the room to all servers in the room. We also change to using the state delta stream, rather than the distributor, as it will make it easier to split processing out of the master process (as well as being more flexible). Finally, when sending presence states to newly joined servers we filter out old presence states to reduce the number sent. Initially we filter out states that are offline and have a last active more than a week ago, though this can be changed down the line. Fixes #3962	6 years ago
Richard van der Hoff	856c83f5f8	Avoid rebuilding Edu objects in worker mode (#4770) ... In worker mode, on the federation sender, when we receive an edu for sending over the replication socket, it is parsed into an Edu object. There is no point extracting the contents of it so that we can then immediately build another Edu.	6 years ago
Amber Brown	c334ca67bb	Integrate presence from hotfixes (#3694)	6 years ago
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423)	6 years ago
Amber Brown	b37c472419	Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)	6 years ago
Amber Brown	49af402019	run isort	6 years ago
Amber Brown	07cad26d65	Remove all global reactor imports & pass it around explicitly (#3424)	7 years ago
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	7 years ago
Amber Brown	53cc2cde1f	cleanup	7 years ago
Amber Brown	8f5a688d42	cleanups, self-registration	7 years ago
Richard van der Hoff	8810685df9	Stub out ServerNoticesSender on the workers ... ... and have the sync endpoints call it directly rather than obsure indirection via PresenceHandler	7 years ago
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	7 years ago
Amber Brown	df9f72d9e5	replacing portions	7 years ago
Adrian Tschira	933bf2dd35	replace some iteritems with six ... Signed-off-by: Adrian Tschira <nota@notafile.com>	7 years ago
Richard van der Hoff	d43d480d86	Remove unused `update_external_syncs` ... This method isn't used anywhere. Burninate it.	7 years ago
Richard van der Hoff	9255a6cb17	Improve exception handling for background processes ... There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.	7 years ago
Erik Johnston	265b993b8a	Split replication layer into two	7 years ago
Erik Johnston	c3f79c9da5	Split out edu/query registration to a separate class	7 years ago
Matthew Hodgson	34c3d0a386	typo	7 years ago
Richard van der Hoff	eaaabc6c4f	replace 'except:' with 'except Exception:' ... what could possibly go wrong	7 years ago
Erik Johnston	ec5c4499f4	Make presence use cached users/hosts in room	8 years ago
Erik Johnston	c7ddb5ef7a	Reuse get_interested_parties	8 years ago
Erik Johnston	414522aed5	Move get_interested_parties	8 years ago
Erik Johnston	2be8a281d2	Comments	8 years ago
Erik Johnston	6308ac45b0	Move get_interested_remotes back to presence handler	8 years ago
Erik Johnston	29574fd5b3	Reduce federation presence replication traffic ... This is mainly done by moving the calculation of where to send presence updates from the presence handler to the transaction queue, so we only need to send the presence event (and not the destinations) across the replication connection. Before we were duplicating by sending the full state across once per destination.	8 years ago