76 Commits (1fb9a2d0bf2506ca6e5343cb340a441585ca1c07)

Author	SHA1	Message	Date
Erik Johnston	1fb9a2d0bf	Limit how often GC happens by time. (#9902) ... Synapse can be quite memory intensive, and unless care is taken to tune the GC thresholds it can end up thrashing, causing noticable performance problems for large servers. We fix this by limiting how often we GC a given generation, regardless of current counts/thresholds. This does not help with the reverse problem where the thresholds are set too high, but that should only happen in situations where they've been manually configured. Adds a `gc_min_seconds_between` config option to override the defaults. Fixes #9890.	4 years ago
Richard van der Hoff	3ff2251754	Improved validation for received requests (#9817) ... * Simplify `start_listening` callpath * Correctly check the size of uploaded files	4 years ago
Richard van der Hoff	59d24c5bef	pass a reactor into SynapseSite (#9874)	4 years ago
Erik Johnston	9d25a0ae65	Split presence out of master (#9820)	4 years ago
Richard van der Hoff	5a153772c1	remove `HomeServer.get_config` (#9815) ... Every single time I want to access the config object, I have to remember whether or not we use `get_config`. Let's just get rid of it.	4 years ago
Erik Johnston	00a6db9676	Move some replication processing out of generic_worker (#9796) ... Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	4 years ago
Jonathan de Jong	4b965c862d	Remove redundant "coding: utf-8" lines (#9786) ... Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	4 years ago
Andrew Morgan	04819239ba	Add a Synapse Module for configuring presence update routing (#9491) ... At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.	4 years ago
Patrick Cloke	da75d2ea1f	Add type hints for the federation sender. (#9681) ... Includes an abstract base class which both the FederationSender and the FederationRemoteSendQueue must implement.	4 years ago
Richard van der Hoff	7c8402ddb8	Suppress CryptographyDeprecationWarning (#9698) ... This warning is somewhat confusing to users, so let's suppress it	4 years ago
Brendan Abolivier	0b56481caa	Fix lint	4 years ago
Brendan Abolivier	066c703729	Move support for MSC3026 behind an experimental flag	4 years ago
Brendan Abolivier	405aeb0b2c	Implement MSC3026: busy presence state	4 years ago
Richard van der Hoff	4db07f9aef	Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539) ... Should fix some remaining warnings	4 years ago
Erik Johnston	2927921942	Clean up `ShardedWorkerHandlingConfig` (#9466) ... * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.	4 years ago
Erik Johnston	66f4949e7f	Fix deleting pushers when using sharded pushers. (#9465)	4 years ago
Eric Eastwood	0a00b7ff14	Update black, and run auto formatting over the codebase (#9381) ... - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](80d6dc9783/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version	4 years ago
Richard van der Hoff	9c715a5f19	Fix SSO on workers (#9271) ... Fixes #8966. * Factor out build_synapse_client_resource_tree Start a function which will mount resources common to all workers. * Move sso init into build_synapse_client_resource_tree ... so that we don't have to do it for each worker * Fix SSO-login-via-a-worker Expose the SSO login endpoints on workers, like the documentation says. * Update workers config for new endpoints Add documentation for endpoints recently added (#8942, #9017, #9262) * remove submit_token from workers endpoints list this *doesn't* work on workers (yet). * changelog * Add a comment about the odd path for SAML2Resource	4 years ago
Erik Johnston	6633a4015a	Allow moving account data and receipts streams off master (#9104)	4 years ago
Patrick Cloke	d1eb1b96e8	Register the /devices endpoint on workers. (#9092)	4 years ago
Erik Johnston	c9195744a4	Move more encryption endpoints off master (#9068)	4 years ago
Richard van der Hoff	671138f658	Clean up exception handling in the startup code (#9059) ... Factor out the exception handling in the startup code to a utility function, and fix the some logging and exit code stuff.	4 years ago
Erik Johnston	b530eaa262	Allow running sendToDevice on workers (#9044)	4 years ago
Patrick Cloke	68bb26da69	Allow redacting events on workers (#8994) ... Adds the redacts endpoint to workers that have the client listener.	4 years ago
Patrick Cloke	30fba62108	Apply an IP range blacklist to push and key revocation requests. (#8821) ... Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).	4 years ago
Erik Johnston	921a3f8a59	Fix not sending events over federation when using sharded event persisters (#8536) ... * Fix outbound federaion with multiple event persisters. We incorrectly notified federation senders that the minimum persisted stream position had advanced when we got an `RDATA` from an event persister. Notifying of federation senders already correctly happens in the notifier, so we just delete the offending line. * Change some interfaces to use RoomStreamToken. By enforcing use of `RoomStreamTokens` we make it less likely that people pass in random ints that they got from somewhere random.	4 years ago
Patrick Cloke	e4f72ddc44	Move additional tasks to the background worker (#8458)	4 years ago
Patrick Cloke	62894673e6	Allow background tasks to be run on a separate worker. (#8369)	4 years ago
Patrick Cloke	8a4a4186de	Simplify super() calls to Python 3 syntax. (#8344) ... This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py	4 years ago
Patrick Cloke	c619253db8	Stop sub-classing object (#8249)	4 years ago
Erik Johnston	0f1afbe8dc	Change HomeServer definition to work with typing. ... Duplicating function signatures between server.py and server.pyi is silly. This commit changes that by changing all `build_*` methods to `get_*` methods and changing the `_make_dependency_method` to work work as a descriptor that caches the produced value. There are some changes in other files that were made to fix the typing in server.py.	4 years ago
Erik Johnston	7620912d84	Add health check endpoint (#8048)	4 years ago
Erik Johnston	a7bdf98d01	Rename database classes to make some sense (#8033)	4 years ago
Patrick Cloke	3950ae51ef	Ensure that remove_pusher is always async (#7981)	4 years ago
Erik Johnston	2c1b9d6763	Update worker docs with recent enhancements (#7969)	4 years ago
Erik Johnston	84d099ae11	Fix typing replication not being handled on master (#7959) ... Handling of incoming typing stream updates from replication was not hooked up on master, effecting set ups where typing was handled on a different worker. This is really only a problem if the master process is also handling sync requests, which is unlikely for those that are at the stage of moving typing off. The other observable effect is that if a worker restarts or a replication connect drops then the typing worker will issue a `POSITION typing`, triggering master process to try and stream *all* typing updates from position 0. Fixes #7907	4 years ago
Patrick Cloke	00e57b755c	Convert synapse.app to async/await. (#7868)	4 years ago
Erik Johnston	f2e38ca867	Allow moving typing off master (#7869)	4 years ago
Erik Johnston	f299441cc6	Add ability to shard the federation sender (#7798)	4 years ago
Patrick Cloke	8fa7fdd4cb	Pass original request headers from workers to the main process. (#7797)	4 years ago
Richard van der Hoff	03619324fc	Create a ListenerConfig object (#7681) ... This ended up being a bit more invasive than I'd hoped for (not helped by generic_worker duplicating some of the code from homeserver), but hopefully it's an improvement. The idea is that, rather than storing unstructured `dict`s in the config for the listener configurations, we instead parse it into a structured `ListenerConfig` object.	5 years ago
Patrick Cloke	7d2532be36	Discard RDATA from already seen positions. (#7648)	5 years ago
Erik Johnston	ef3934ec8f	Ensure we persist and ack the same token	5 years ago
Erik Johnston	35c308731d	Speed up processing of federation stream RDATA rows. ... Instead of storing and sending an ACK for every single row we send synchronously, we instead do it asynchronously while batching up updates.	5 years ago
Richard van der Hoff	04729b86f8	Fix incorrect exception handling in KeyUploadServlet.on_POST (#7563) ... Introduced in #7556	5 years ago
Richard van der Hoff	00db90f409	Fix recording of federation stream token (#7564) ... A couple of changes of significance: * remove the `_last_ack < federation_position` condition, so that updates will still be correctly processed after restart * Correctly wire up send_federation_ack to the right class.	5 years ago
Erik Johnston	e5c67d04db	Add option to move event persistence off master (#7517)	5 years ago
Patrick Cloke	4429764c9f	Return 200 OK for all OPTIONS requests (#7534)	5 years ago
Erik Johnston	547e4dd83e	Fix exception reporting due to HTTP request errors. (#7556) ... These are business as usual errors, rather than stuff we want to log at error.	5 years ago
Richard van der Hoff	0bbbd10513	Stub out GET presence requests in the frontend proxy (#7545) ... We don't really make any promises about returning accurate presence data when presence is disabled, so we may as well just return a static response, rather than making the master handle a request.	5 years ago