58 Commits (36ae8611fe98977153387308311f7d38b67e39c8)

Author	SHA1	Message	Date
Eric Eastwood	1c802de626	Re-introduce the outbound federation proxy (#15913) ... Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.	1 year ago
Jason Little	224ef0b669	Unix Sockets for HTTP Replication (#15708) ... Unix socket support for `federation` and `client` Listeners has existed now for a little while(since [1.81.0](https://github.com/matrix-org/synapse/pull/15353)), but there was one last hold out before it could be complete: HTTP Replication communication. This should finish it up. The Listeners would have always worked, but would have had no way to be talked to/at. --------- Co-authored-by: Eric Eastwood <madlittlemods@gmail.com> Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Eric Eastwood <erice@element.io>	1 year ago
Eric Eastwood	c9bf644fa0	Revert "Federation outbound proxy" (#15910) ... Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494.	1 year ago
Jason Little	2481b7dfa4	Remove `worker_replication_*` deprecated settings, with helpful errors on startup (#15860) ... Co-authored-by: reivilibre <oliverw@matrix.org>	1 year ago
Eric Eastwood	b07b14b494	Federation outbound proxy (#15773) ... Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.	1 year ago
reivilibre	c775d80b73	Fix a bug introduced in Synapse v1.84.0 where workers do not start up when no `instance_map` was provided. (#15672) ... * Fix #15669: always populate instance map even if it was empty * Fix some tests * Fix more tests * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * CI fix: don't forget to update apt repository sources before installing olddeps deps * Add test testing the backwards compatibility --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>	2 years ago
Jason Little	e4f545c452	Remove `worker_replication_*` settings (#15491) ... * Add master to the instance_map as part of Complement, have ReplicationEndpoint look at instance_map for master. * Fix typo in drive by. * Remove unnecessary worker_replication_* bits from unit tests and add master to instance_map(hopefully in the right place) * Several updates: 1. Switch from master to main for naming the main process in the instance_map. Add useful constants for easier adjustment of names in the future. 2. Add backwards compatibility for worker_replication_* to allow time to transition to new style. Make sure to prioritize declaring main directly on the instance_map. 3. Clean up old comments/commented out code. 4. Adjust unit tests to match with new code. 5. Adjust Complement setup infrastructure to only add main to the instance_map if workers are used and remove now unused options from the worker.yaml template. * Initial Docs upload * Changelog * Missed some commented out code that can go now * Remove TODO comment that no longer holds true. * Fix links in docs * More docs * Remove debug logging * Apply suggestions from code review Co-authored-by: reivilibre <olivier@librepush.net> * Apply suggestions from code review Co-authored-by: reivilibre <olivier@librepush.net> * Update version to latest, include completeish before/after examples in upgrade notes. * Fix up and docs too --------- Co-authored-by: reivilibre <olivier@librepush.net>	2 years ago
Jason Little	e12d788bb7	Switch `InstanceLocationConfig` to a pydantic `BaseModel` (#15431) ... * Switch InstanceLocationConfig to a pydantic BaseModel, apply Strict* types and add a few helper methods(that will make more sense in follow up work). Co-authored-by: David Robertson <davidr@element.io>	2 years ago
Jason Little	56efa9b167	Experimental Unix socket support (#15353) ... * Add IReactorUNIX to ISynapseReactor type hint. * Create listen_unix(). Two options, 'path' to the file and 'mode' of permissions(not umask, recommend 666 as default as nginx/other reverse proxies write to it and it's setup as user www-data) For the moment, leave the option to always create a PID lockfile turned on by default * Create UnixListenerConfig and wire it up. Rename ListenerConfig to TCPListenerConfig, then Union them together into ListenerConfig. This spidered around a bit, but I think I got it all. Metrics and manhole have been placed behind a conditional in case of accidental putting them onto a unix socket. Use new helpers to get if a listener is configured for TLS, and to help create a site tag for logging. There are 2 TODO things in parse_listener_def() to finish up at a later point. * Refactor SynapseRequest to handle logging correctly when using a unix socket. This prevents an exception when an IP address can not be retrieved for a request. * Make the 'Synapse now listening on Unix socket' log line a little prettier. * No silent failures on generic workers when trying to use a unix socket with metrics or manhole. * Inline variables in app/_base.py * Update docstring for listen_unix() to remove reference to a hardcoded permission of 0o666 and add a few comments saying where the default IS declared. * Disallow both a unix socket and a ip/port combo on the same listener resource * Linting * Changelog * review: simplify how listen_unix returns(and get rid of a type: ignore) * review: fix typo from ConfigError in app/homeserver.py * review: roll conditional for http_options.tag into get_site_tag() helper(and add docstring) * review: enhance the conditionals for checking if a port or path is valid, remove a TODO line * review: Try updating comment in get_client_ip_if_available to clarify what is being retrieved and why * Pretty up how 'Synapse now listening on Unix Socket' looks by decoding the byte string. * review: In parse_listener_def(), raise ConfigError if neither socket_path nor port is declared(and fix a typo)	2 years ago
realtyem	df390a8e67	Refactor `federation_sender` and `pusher` configuration loading. (#14496) ... To avoid duplicating the same logic for handling legacy configuration settings. This should help in applying similar logic to other worker types.	2 years ago
David Robertson	01a0527892	Fix version that `worker_main_http_uri` is redundant from (#14476) ... * Fix version that `worker_main_http_uri` is redundant from * Changelog	2 years ago
realtyem	c15e9a0edb	Remove need for `worker_main_http_uri` setting to use /keys/upload. (#14400)	2 years ago
Tuomas Ojamies	b5ab2c428a	Support using SSL on worker endpoints. (#14128) ... * Fix missing SSL support in worker endpoints. * Add changelog * SSL for Replication endpoint * Remove unit test change * Refactor listener creation to reduce duplicated code * Fix the logger message * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Add config documentation for new TLS option Co-authored-by: Tuomas Ojamies <tojamies@palantir.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>	2 years ago
Patrick Cloke	32fc3b7ba4	Remove configuration options for direct TCP replication. (#13647) ... Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.	2 years ago
Shay	493c2fc44a	Remove code generating comments in configuration file (#12941)	2 years ago
reivilibre	699192fc1a	Add the `update_user_directory_from_worker` configuration option (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. (#12654) ... Co-authored-by: Shay <hillerys@element.io>	3 years ago
reivilibre	c2d50e9f6c	Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452)	3 years ago
Patrick Cloke	4586119f0b	Add missing type hints to config classes. (#12402)	3 years ago
Patrick Cloke	10a88ba91c	Use auto_attribs/native type hints for attrs classes. (#11692)	3 years ago
Olivier Wilkinson (reivilibre)	4dd9ea8f4f	Revert "Revert accidental fast-forward merge from v1.49.0rc1" ... This reverts commit 158d73ebdd.	3 years ago
Sean Quah	158d73ebdd	Revert accidental fast-forward merge from v1.49.0rc1 ... Revert "Sort internal changes in changelog" Revert "Update CHANGES.md" Revert "1.49.0rc1" Revert "Revert "Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505) (#11527)" Revert "Refactors in `_generate_sync_entry_for_rooms` (#11515)" Revert "Correctly register shutdown handler for presence workers (#11518)" Revert "Fix `ModuleApi.looping_background_call` for non-async functions (#11524)" Revert "Fix 'delete room' admin api to work on incomplete rooms (#11523)" Revert "Correctly ignore invites from ignored users (#11511)" Revert "Fix the test breakage introduced by #11435 as a result of concurrent PRs (#11522)" Revert "Stabilise support for MSC2918 refresh tokens as they have now been merged into the Matrix specification. (#11435)" Revert "Save the OIDC session ID (sid) with the device on login (#11482)" Revert "Add admin API to get some information about federation status (#11407)" Revert "Include bundled aggregations in /sync and related fixes (#11478)" Revert "Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)" Revert "Update backward extremity docs to make it clear that it does not indicate whether we have fetched an events' `prev_events` (#11469)" Revert "Support configuring the lifetime of non-refreshable access tokens separately to refreshable access tokens. (#11445)" Revert "Add type hints to `synapse/tests/rest/admin` (#11501)" Revert "Revert accidental commits to develop." Revert "Newsfile" Revert "Give `tests.server.setup_test_homeserver` (nominally!) the same behaviour" Revert "Move `tests.utils.setup_test_homeserver` to `tests.server`" Revert "Convert one of the `setup_test_homeserver`s to `make_test_homeserver_synchronous`" Revert "Disambiguate queries on `state_key` (#11497)" Revert "Comments on the /sync tentacles (#11494)" Revert "Clean up tests.storage.test_appservice (#11492)" Revert "Clean up `tests.storage.test_main` to remove use of legacy code. (#11493)" Revert "Clean up `tests.test_visibility` to remove legacy code. (#11495)" Revert "Minor cleanup on recently ported doc pages (#11466)" Revert "Add most of the missing type hints to `synapse.federation`. (#11483)" Revert "Avoid waiting for zombie processes in `synctl stop` (#11490)" Revert "Fix media repository failing when media store path contains symlinks (#11446)" Revert "Add type annotations to `tests.storage.test_appservice`. (#11488)" Revert "`scripts-dev/sign_json`: support for signing events (#11486)" Revert "Add MSC3030 experimental client and federation API endpoints to get the closest event to a given timestamp (#9445)" Revert "Port wiki pages to documentation website (#11402)" Revert "Add a license header and comment. (#11479)" Revert "Clean-up get_version_string (#11468)" Revert "Link background update controller docs to summary (#11475)" Revert "Additional type hints for config module. (#11465)" Revert "Register the login redirect endpoint for v3. (#11451)" Revert "Update openid.md" Revert "Remove mention of OIDC certification from Dex (#11470)" Revert "Add a note about huge pages to our Postgres doc (#11467)" Revert "Don't start Synapse master process if `worker_app` is set (#11416)" Revert "Expose worker & homeserver as entrypoints in `setup.py` (#11449)" Revert "Bundle relations of relations into the `/relations` result. (#11284)" Revert "Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)" Revert "Eliminate a few `Any`s in `LruCache` type hints (#11453)" Revert "Remove unnecessary `json.dumps` from `tests.rest.admin` (#11461)" Revert "Merge branch 'master' into develop" This reverts commit 26b5d2320f. This reverts commit bce4220f38. This reverts commit 966b5d0fa0. This reverts commit 088d748f2c. This reverts commit 14d593f72d. This reverts commit 2a3ec6facf. This reverts commit eccc49d755. This reverts commit b1ecd19c5d. This reverts commit 9c55dedc8c. This reverts commit 2d42e586a8. This reverts commit 2f053f3f82. This reverts commit a15a893df8. This reverts commit 8b4b153c9e. This reverts commit 494ebd7347. This reverts commit a77c369897. This reverts commit 4eb77965cd. This reverts commit 637df95de6. This reverts commit e5f426cd54. This reverts commit 8cd68b8102. This reverts commit 6cae125e20. This reverts commit 7be88fbf48. This reverts commit b3fd99b74a. This reverts commit f7ec6e7d9e. This reverts commit 5640992d17. This reverts commit d26808dd85. This reverts commit f91624a595. This reverts commit 16d39a5490. This reverts commit 8a4c296987. This reverts commit 49e1356ee3. This reverts commit d2279f471b. This reverts commit b50e39df57. This reverts commit 858d80bf0f. This reverts commit 435f044807. This reverts commit f61462e1be. This reverts commit a6f1a3abec. This reverts commit 84dc50e160. This reverts commit ed635d3285. This reverts commit 7b62791e00. This reverts commit 153194c771. This reverts commit f44d729d4c. This reverts commit a265fbd397. This reverts commit b9fef1a7cd. This reverts commit b0eb64ff7b. This reverts commit f1795463bf. This reverts commit 70cbb1a5e3. This reverts commit 42bf020463. This reverts commit 379f2650cf. This reverts commit 7ff22d6da4. This reverts commit 5a0b652d36. This reverts commit 432a174bc1. This reverts commit b14f8a1baf, reversing changes made to e713855dca.	3 years ago
Patrick Cloke	f44d729d4c	Additional type hints for config module. (#11465) ... This adds some misc. type hints to helper methods used in the `synapse.config` module.	3 years ago
Nick Barrett	af54167516	Enable passing typing stream writers as a list. (#11237) ... This makes the typing stream writer config match the other stream writers that only currently support a single worker.	3 years ago
Erik Johnston	9d25a0ae65	Split presence out of master (#9820)	4 years ago
Jonathan de Jong	4b965c862d	Remove redundant "coding: utf-8" lines (#9786) ... Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	4 years ago
Erik Johnston	2927921942	Clean up `ShardedWorkerHandlingConfig` (#9466) ... * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.	4 years ago
Eric Eastwood	0a00b7ff14	Update black, and run auto formatting over the codebase (#9381) ... - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](80d6dc9783/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version	4 years ago
Erik Johnston	6633a4015a	Allow moving account data and receipts streams off master (#9104)	4 years ago
Erik Johnston	b530eaa262	Allow running sendToDevice on workers (#9044)	4 years ago
Patrick Cloke	96358cb424	Add authentication to replication endpoints. (#8853) ... Authentication is done by checking a shared secret provided in the Synapse configuration file.	4 years ago
Patrick Cloke	62894673e6	Allow background tasks to be run on a separate worker. (#8369)	4 years ago
Erik Johnston	04cc249b43	Add experimental support for sharding event persister. Again. (#8294) ... This is *not* ready for production yet. Caveats: 1. We should write some tests... 2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.	4 years ago
Brendan Abolivier	9f8abdcc38	Revert "Add experimental support for sharding event persister. (#8170)" (#8242) ... * Revert "Add experimental support for sharding event persister. (#8170)" This reverts commit 82c1ee1c22. * Changelog	4 years ago
Erik Johnston	82c1ee1c22	Add experimental support for sharding event persister. (#8170) ... This is *not* ready for production yet. Caveats: 1. We should write some tests... 2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.	4 years ago
Erik Johnston	2c1b9d6763	Update worker docs with recent enhancements (#7969)	4 years ago
Erik Johnston	f2e38ca867	Allow moving typing off master (#7869)	4 years ago
Richard van der Hoff	03619324fc	Create a ListenerConfig object (#7681) ... This ended up being a bit more invasive than I'd hoped for (not helped by generic_worker duplicating some of the code from homeserver), but hopefully it's an improvement. The idea is that, rather than storing unstructured `dict`s in the config for the listener configurations, we instead parse it into a structured `ListenerConfig` object.	5 years ago
Erik Johnston	e5c67d04db	Add option to move event persistence off master (#7517)	5 years ago
Erik Johnston	1de36407d1	Add `instance_map` config and route replication calls (#7495)	5 years ago
Amber Brown	f743108a94	Refactor HomeserverConfig so it can be typechecked (#6137)	5 years ago
Matthew Hodgson	7ef319aefe	fix broken copyrights	5 years ago
Amber Brown	0d0f6d12bc	Fix logging in workers (#5729) ... This also adds a worker blacklist.	5 years ago
Amber Brown	6a85cb5ef7	Remove non-dedicated logging options and command line arguments (#5678)	5 years ago
Richard van der Hoff	c3c6b00d95	Pass config_dir_path and data_dir_path into Config.read_config. (#5522) ... * Pull config_dir_path and data_dir_path calculation out of read_config_files * Pass config_dir_path and data_dir_path into read_config	6 years ago
Richard van der Hoff	6cda36777b	Drop support for cpu_affinity (#5525) ... This has no useful purpose on python3, and is generally a source of confusion.	6 years ago
Amber Brown	32e7c9e7f2	Run Black. (#5482)	6 years ago
Erik Johnston	9ad448c1e5	Correctly handle all command line options	6 years ago
Erik Johnston	72bfaf746d	Allow passing --daemonize to workers	6 years ago
Erik Johnston	24dd73028a	Add replication http endpoint for event sending	7 years ago
Erik Johnston	f4d93ae424	Actually make it work	7 years ago