45 Commits (b39b02c26e9550c629c027a06b25d64e94fe8e5e)

Author	SHA1	Message	Date
Mathieu Velten	6f68e32bfb	to_device updates could be dropped when consuming the replication stream (#15349) ... Co-authored-by: reivilibre <oliverw@matrix.org>	2 years ago
Patrick Cloke	afb216c202	Remove no-op send_command for Redis replication. (#15274) ... With Redis commands do not need to be re-issued by the main process (they fan-out to all processes at once) and thus it is no longer necessary to worry about them reflecting recursively forever.	2 years ago
Erik Johnston	b2357a898c	Fix bug where 5s delays would occasionally happen. (#15150) ... This only affects deployments using workers.	2 years ago
Patrick Cloke	42aea0d8af	Add final type hint to tests.unittest. (#15072) ... Adds a return type to HomeServerTestCase.make_homeserver and deal with any variables which are no longer Any.	2 years ago
Patrick Cloke	156cd88eef	Add missing type hints to tests.replication. (#14987)	2 years ago
Erik Johnston	65d0386693	Always notify replication when a stream advances (#14877) ... This ensures that all other workers are told about stream updates in a timely manner, without having to remember to manually poke replication.	2 years ago
Erik Johnston	316590d1ea	Fix bug in `wait_for_stream_position` (#14856) ... We were incorrectly checking if the *local* token had been advanced, rather than the token for the remote instance. In practice, I don't think this has caused any bugs due to where we use `wait_for_stream_position`, as critically we don't use it on instances that also write to the given streams (and so the local token will lag behind all remote tokens).	2 years ago
reivilibre	9e82caac45	Faster remote room joins: unblock tasks waiting for full room state when the un-partial-stating of that room is received over the replication stream. [rei:frrj/streams/unpsr] (#14474)	2 years ago
realtyem	854a6884d8	Modernize unit tests configuration settings for workers. (#14568) ... Use the newer foo_instances configuration instead of the deprecated flags to enable specific features (e.g. start_pushers).	2 years ago
Patrick Cloke	efd108b45d	Accept & store thread IDs for receipts (implement MSC3771). (#13782) ... Updates the `/receipts` endpoint and receipt EDU handler to parse a `thread_id` from the body and insert it in the database.	2 years ago
Patrick Cloke	f3fba4914d	Reduce the number of tests using TCP replication. (#13543) ... Uses Redis replication in additional test cases (instead of TCP replication). A small step towards dropping TCP replication.	2 years ago
reivilibre	177b884ad7	Lay some foundation work to allow workers to only subscribe to some kinds of messages, reducing replication traffic. (#12672)	3 years ago
Patrick Cloke	3e4af36bc8	Rename get_tcp_replication to get_replication_command_handler. (#12192) ... Since the object it returns is a ReplicationCommandHandler. This is clean-up from adding support to Redis where the command handler was added as an additional layer of abstraction from the TCP protocol.	3 years ago
Richard van der Hoff	e24ff8ebe3	Remove `HomeServer.get_datastore()` (#12031) ... The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733	3 years ago
Patrick Cloke	d0e78af35e	Add missing type hints to synapse.replication. (#11938)	3 years ago
reivilibre	642a42edde	Flatten the synapse.rest.client package (#10600)	3 years ago
Jonathan de Jong	93729719b8	Use inline type hints in `tests/` (#10350) ... This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)	3 years ago
Jonathan de Jong	495b214f4f	Fix (final) Bugbear violations (#9838)	4 years ago
Jonathan de Jong	4b965c862d	Remove redundant "coding: utf-8" lines (#9786) ... Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	4 years ago
Patrick Cloke	0b3112123d	Use mock from the stdlib. (#9772)	4 years ago
Erik Johnston	b5efcb577e	Make it possible to use dmypy (#9692) ... Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.	4 years ago
Erik Johnston	2927921942	Clean up `ShardedWorkerHandlingConfig` (#9466) ... * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.	4 years ago
Eric Eastwood	0a00b7ff14	Update black, and run auto formatting over the codebase (#9381) ... - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](80d6dc9783/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version	4 years ago
Dan Callahan	aff1eb7c67	Tell Black to format code for Python 3.5 (#8664) ... This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>	4 years ago
Patrick Cloke	cc9bb3dc3f	Convert the message handler to async/await. (#7884)	4 years ago
Patrick Cloke	7d2532be36	Discard RDATA from already seen positions. (#7648)	5 years ago
Richard van der Hoff	164f50f5f2	fix mypy for tests/replication (#7518)	5 years ago
Richard van der Hoff	6c1f7c722f	Fix limit logic for AccountDataStream (#7384) ... Make sure that the AccountDataStream presents complete updates, in the right order. This is much the same fix as #7337 and #7358, but applied to a different stream.	5 years ago
Erik Johnston	18c1e52d82	Clean up replication unit tests. (#7490)	5 years ago
Richard van der Hoff	d5aa7d93ed	Fix catchup-on-reconnect for the Federation Stream (#7374) ... looks like we managed to break this during the refactorathon.	5 years ago
Erik Johnston	0e719f2398	Thread through instance name to replication client. (#7369) ... For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.	5 years ago
Erik Johnston	3085cde577	Use `stream.current_token()` and remove `stream_positions()` (#7172) ... We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.	5 years ago
Erik Johnston	37f6823f5b	Add instance name to RDATA/POSITION commands (#7364) ... This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.	5 years ago
Erik Johnston	3eab76ad43	Don't relay REMOTE_SERVER_UP cmds to same conn. (#7352) ... For direct TCP connections we need the master to relay REMOTE_SERVER_UP commands to the other connections so that all instances get notified about it. The old implementation just relayed to all connections, assuming that sending back to the original sender of the command was safe. This is not true for redis, where commands sent get echoed back to the sender, which was causing master to effectively infinite loop sending and then re-receiving REMOTE_SERVER_UP commands that it sent. The fix is to ensure that we only relay to *other* connections and not to the connection we received the notification from. Fixes #7334.	5 years ago
Richard van der Hoff	c2e1a2110f	Fix limit logic for EventsStream (#7358) ... * Factor out functions for injecting events into database I want to add some more flexibility to the tools for injecting events into the database, and I don't want to clutter up HomeserverTestCase with them, so let's factor them out to a new file. * Rework TestReplicationDataHandler This wasn't very easy to work with: the mock wrapping was largely superfluous, and it's useful to be able to inspect the received rows, and clear out the received list. * Fix AssertionErrors being thrown by EventsStream Part of the problem was that there was an off-by-one error in the assertion, but also the limit logic was too simple. Fix it all up and add some tests.	5 years ago
Erik Johnston	fce663889b	Add some replication tests (#7278) ... Specifically some tests for the typing stream, which means we test streams that fetch missing updates via HTTP (rather than via the DB). We also shuffle things around a bit so that we create two separate `HomeServer` objects, rather than trying to insert a slaved store into places. Note: `test_typing.py` is heavily inspired by `test_receipts.py`	5 years ago
Richard van der Hoff	82d8b1dd1f	Another go at fixing one-word commands (#7326) ... I messed this up last time I tried (#7239 / e13c6c7).	5 years ago
Erik Johnston	5016b162fc	Move client command handling out of TCP protocol (#7185) ... The aim here is to move the command handling out of the TCP protocol classes and to also merge the client and server command handling (so that we can reuse them for redis protocol). This PR simply moves the client paths to the new `ReplicationCommandHandler`, a future PR will move the server paths too.	5 years ago
Erik Johnston	4cff617df1	Move catchup of replication streams to worker. (#7024) ... This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.	5 years ago
Richard van der Hoff	a564b92d37	Convert `*StreamRow` classes to inner classes (#7116) ... This just helps keep the rows closer to their streams, so that it's easier to see what the format of each stream is.	5 years ago
Erik Johnston	48c3a96886	Port synapse.replication.tcp to async/await (#6666) ... * Port synapse.replication.tcp to async/await * Newsfile * Correctly document type of on_<FOO> functions as async * Don't be overenthusiastic with the asyncing....	5 years ago
Amber Brown	0f87b912ab	Implementation of MSC2314 (#6176)	5 years ago
Amber Brown	b36c82576e	Run Black on the tests again (#5170)	6 years ago
Richard van der Hoff	a5798de067	Move replication.tcp.streams into a package	6 years ago
Richard van der Hoff	9bde730ef8	Fix bug where read-receipts lost their timestamps (#4927) ... Make sure that they are sent correctly over the replication stream. Fixes: #4898	6 years ago