12986 Commits (4db07f9aefbd2b93df8f8b5c5153183ea1539b32)

Author	SHA1	Message	Date
Richard van der Hoff	4db07f9aef	Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539) ... Should fix some remaining warnings	4 years ago
Erik Johnston	a4fa044c00	Fix 'rejected_events_metadata' background update (#9537) ... Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.	4 years ago
Patrick Cloke	922788c604	Purge chain cover tables when purging events. (#9498)	4 years ago
Dirk Klimpel	d790d0d314	Add type hints to user admin API. (#9521)	4 years ago
Patrick Cloke	0c330423bc	Bump the mypy and mypy-zope versions. (#9529)	4 years ago
Erik Johnston	16f9f93eb7	Make deleting stale pushers a background update (#9536)	4 years ago
Aaron Raimist	0279e0e086	Prevent presence background jobs from running when presence is disabled (#9530) ... Prevent presence background jobs from running when presence is disabled Signed-off-by: Aaron Raimist <aaron@raim.ist>	4 years ago
Patrick Cloke	aee10768d8	Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)" ... This reverts commit f5c93fc993. This is being backed out due to a regression (#9507) and additional review feedback being provided.	4 years ago
Erik Johnston	7f5d753d06	Re-run rejected metadata background update. (#9503) ... It landed in schema version 58 after 59 had been created, causing some servers to not run it. The main effect of was that not all rooms had their chain cover calculated correctly. After the BG updates complete the chain covers will get fixed when a new state event in the affected rooms is received.	4 years ago
Erik Johnston	16108c579d	Fix SQL delta file taking a long time to run (#9516) ... Fixes #9504	4 years ago
Dirk Klimpel	f00c4e7af0	Add type hints to device and event report admin API (#9519)	4 years ago
Patrick Cloke	ad8589d392	Fix a bug when a room alias is given to the admin join endpoint (#9506)	4 years ago
Patrick Cloke	16ec8c3272	(Hopefully) stop leaking file descriptors in media repo. (#9497) ... By consuming the response if the headers imply that the content is too large.	4 years ago
Patrick Cloke	a0bc9d387e	Use the proper Request in type hints. (#9515) ... This also pins the Twisted version in the mypy job for CI until proper type hints are fixed throughout Synapse.	4 years ago
Jonathan de Jong	e12077a78a	Allow bytecode again (#9502) ... In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again. Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	4 years ago
Tim Leung	ddb240293a	Add support for no_proxy and case insensitive env variables (#9372) ... ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519)) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](bdb941be42/Lib/urllib/request.py (L2488)) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk	4 years ago
Richard van der Hoff	15090de850	SSO: redirect to public URL before setting cookies (#9436) ... ... otherwise, we don't get the cookie back.	4 years ago
Erik Johnston	2566dc57ce	Test that we require validated email for email pushers (#9496)	4 years ago
Erik Johnston	1e62d9ee8c	Ensure pushers are deleted for deactivated accounts (#9285)	4 years ago
Erik Johnston	b5c4fe1971	1.28.0	4 years ago
Richard van der Hoff	d8e95e5452	Add support for X-Forwarded-Proto (#9472) ... rewrite XForwardedForRequest to set `isSecure()` based on `X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.	4 years ago
Erik Johnston	7cc571510b	Add SQL delta for deleting stale pushers (#9479)	4 years ago
Jonathan de Jong	f5c93fc993	Fix #8518 (sync requests being cached wrongly on timeout) (#9358) ... This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false". This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>	4 years ago
Erik Johnston	2927921942	Clean up `ShardedWorkerHandlingConfig` (#9466) ... * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.	4 years ago
Erik Johnston	0b5c967813	Refactor to ensure we call check_consistency (#9470) ... The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.	4 years ago
Patrick Cloke	7292b7c0eb	Add back the deprecated SAML endpoint. (#9474)	4 years ago
Richard van der Hoff	713145d3de	Add a comment about systemd-python. (#9464) ... This confused me for a while.	4 years ago
Patrick Cloke	65a9eb8994	Include newly added sequences in the port DB script. (#9449) ... And ensure the consistency of `event_auth_chain_id`.	4 years ago
Erik Johnston	66f4949e7f	Fix deleting pushers when using sharded pushers. (#9465)	4 years ago
Richard van der Hoff	1b2d6d55c5	Remove vestiges of uploads_path config (#9462) ... `uploads_path` was a thing that was never used; most of it was removed in #6628 but a few vestiges remained.	4 years ago
Dirk Klimpel	71c9f8de6d	Add an `order_by` field to list users' media admin API. (#8978)	4 years ago
Andrew Morgan	0a363f9ca4	Remove cache for get_shared_rooms_for_users (#9416) ... This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed. The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately. Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.	4 years ago
Andrew Morgan	e22b71810e	Clean up the user directory sample config section (#9385) ... The user directory sample config section was a little messy, and didn't adhere to our [recommended config format guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format). This PR cleans that up a bit.	4 years ago
Patrick Cloke	fc8b3d8809	Ratelimit cross-user key sharing requests. (#8957)	4 years ago
Patrick Cloke	a1901abd6b	Add documentation and type hints to parse_duration. (#9432)	4 years ago
Patrick Cloke	c4a55ac4a4	Fix style checking due to updated black.	4 years ago
Patrick Cloke	b114a45f5f	Support not providing an IdP icon when choosing a username. (#9440)	4 years ago
Andrew Morgan	8bcfc2eaad	Be smarter about which hosts to send presence to when processing room joins (#9402) ... This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually. --- When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed. It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw. This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence: * If it was a local user join, send that user's latest presence to all servers in the room * If it was a remote user join, send the presence for all local users in the room to that homeserver We deduplicate by inserting all of those pending updates into a dictionary of the form: ``` { server_name1: {presence_update1, ...}, server_name2: {presence_update1, presence_update2, ...} } ``` Only after building this dict do we then start sending out presence updates.	4 years ago
Andrew Morgan	13e9029f44	Add a config option to prioritise local users in user directory search results (#9383) ... This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default. Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical. This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.	4 years ago
Erik Johnston	3d2acc930f	Return a 404 if we don't have the original file	4 years ago
AndrewFerr	9bc74743d5	Add configs to make profile data more private (#9203) ... Add off-by-default configuration settings to: - disable putting an invitee's profile info in invite events - disable profile lookup via federation Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>	4 years ago
Patrick Cloke	1381cd05b0	1.28.0rc1	4 years ago
Erik Johnston	b106080fb4	Regenerate exact thumbnails if missing	4 years ago
Patrick Cloke	9ee3b9775f	Remove deprecated SAML2 callback URL since it does not work. (#9434) ... Updates documentation from #9289 and removes a deprecated endpoint which didn't work as expected.	4 years ago
Rishabh Arya	e17553e185	Parse ui_auth.session_timeout as a duration (instead of treating it as ms) (#9426)	4 years ago
Patrick Cloke	8ec2217103	Reduce the memory usage of previewing media files. (#9421) ... This reduces the memory usage of previewing media files which end up larger than the `max_spider_size` by avoiding buffering content internally in treq. It also checks the `Content-Length` header in additional places instead of streaming the content to check the body length.	4 years ago
Patrick Cloke	43f1c82457	Add back the guard against the user directory stream position not existing. (#9428) ... As the comment says, this guard was there for when the initial user directory update has yet to happen.	4 years ago
Dirk Klimpel	c8d9383cfb	Add the shadow-banning status to the display user admin API. (#9400)	4 years ago
Andrew Morgan	a25661b2eb	Remove dead notify_for_states presence method (#9408)	4 years ago
Andrew Morgan	3e5749b99f	Fix only handling the last presence state for each user (#9425) ... This is a small bug that I noticed while working on #8956. We have a for-loop which attempts to strip all presence changes for each user except for the final one, as we don't really care about older presence: 9e19c6aab4/synapse/handlers/presence.py (L368-L371) `new_states_dict` stores this stripped copy of latest presence state for each user, before it is... put into a new variable `new_state`, which is just overridden by the subsequent for loop. I believe this was instead meant to override `new_states`. Without doing so, it effectively meant: 1. The for loop had no effect. 2. We were still processing old presence state for users.	4 years ago