431 Commits (77e56deffcf685ca5d0a264059f0c326e53e99af)

Author	SHA1	Message	Date
Richard van der Hoff	3ada9b4264	Drop support for sqlite<3.22 as well	4 years ago
Patrick Cloke	01dd90b0f0	Add type hints to DictionaryCache and TTLCache. (#9442)	4 years ago
Patrick Cloke	2a99cc6524	Use the chain cover index in get_auth_chain_ids. (#9576) ... This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.	4 years ago
Patrick Cloke	cb7fc7523e	Add a basic test for purging rooms. (#9541) ... Unfortunately this doesn't test re-joining the room since that requires having another homeserver to query over federation, which isn't easily doable in unit tests.	4 years ago
Dirk Klimpel	c8d9383cfb	Add the shadow-banning status to the display user admin API. (#9400)	4 years ago
Eric Eastwood	0a00b7ff14	Update black, and run auto formatting over the codebase (#9381) ... - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](80d6dc9783/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version	4 years ago
Erik Johnston	6633a4015a	Allow moving account data and receipts streams off master (#9104)	4 years ago
Erik Johnston	350d9923cd	Make chain cover index bg update go faster (#9124) ... We do this by allowing a single iteration to process multiple rooms at a time, as there are often a lot of really tiny rooms, which can massively slow things down.	4 years ago
Erik Johnston	7036e24e98	Add background update for add chain cover index (#9029)	4 years ago
Dirk Klimpel	7a2e9b549d	Remove user's avatar URL and displayname when deactivated. (#8932) ... This only applies if the user's data is to be erased.	4 years ago
Erik Johnston	1315a2e8be	Use a chain cover index to efficiently calculate auth chain difference (#8868)	4 years ago
Patrick Cloke	23d701864f	Improve the performance of calculating ignored users in large rooms (#9024) ... This allows for efficiently finding which users ignore a particular user. Co-authored-by: Erik Johnston <erik@matrix.org>	4 years ago
Erik Johnston	70586aa63e	Try and drop stale extremities. (#8929) ... If we see stale extremities while persisting events, and notice that they don't change the result of state resolution, we drop them.	4 years ago
Brendan Abolivier	f2783fc201	Use the simple dictionary in full text search for the user directory (#8959) ... * Use the simple dictionary in fts for the user directory * Clarify naming	4 years ago
Dirk Klimpel	06006058d7	Make search statement in List Room and User Admin API case-insensitive (#8931)	4 years ago
Dirk Klimpel	0a34cdfc66	Add number of local devices to Room Details Admin API (#8886)	4 years ago
Erik Johnston	df4b1e9c74	Pass room_id to get_auth_chain_difference (#8879) ... This is so that we can choose which algorithm to use based on the room ID.	4 years ago
Richard van der Hoff	90cf1eec44	Remove redundant mocking	4 years ago
Patrick Cloke	30fba62108	Apply an IP range blacklist to push and key revocation requests. (#8821) ... Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).	4 years ago
Erik Johnston	c5b6abd53d	Correctly handle unpersisted events when calculating auth chain difference. (#8827) ... We do state res with unpersisted events when calculating the new current state of the room, so that should be the only thing impacted. I don't think this is tooooo big of a deal as: 1. the next time a state event happens in the room the current state should correct itself; 2. in the common case all the unpersisted events' auth events will be pulled in by other state, so will still return the correct result (or one which is sufficiently close to not affect the result); and 3. we mostly use the state at an event to do important operations, which isn't affected by this.	4 years ago
Dirk Klimpel	3f0ff53158	Remove deprecated `/_matrix/client/*/admin` endpoints (#8785) ... These are now only available via `/_synapse/admin/v1`.	4 years ago
Erik Johnston	f737368a26	Add admin API for logging in as a user (#8617)	4 years ago
Richard van der Hoff	be8fa65d0b	Remove redundant calls to `render()`	4 years ago
Richard van der Hoff	c3e3552ec4	fixup test	4 years ago
Richard van der Hoff	ebc405446e	Add a `custom_headers` param to `make_request` (#8760) ... Some tests want to set some custom HTTP request headers, so provide a way to do that before calling requestReceived().	4 years ago
Erik Johnston	f21e24ffc2	Add ability for access tokens to belong to one user but grant access to another user. (#8616) ... We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.	4 years ago
Dan Callahan	aff1eb7c67	Tell Black to format code for Python 3.5 (#8664) ... This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>	4 years ago
Will Hunt	9e0f5a0ac4	Fix get|set_type_stream_id_for_appservice store functions (#8648)	4 years ago
Will Hunt	e8dbbcb64c	Fix get|set_type_stream_id_for_appservice store functions (#8648)	4 years ago
Erik Johnston	2ac908f377	Don't instansiate Requester directly (#8614)	4 years ago
Richard van der Hoff	7b71695388	Combine the two sets of tests for CacheDescriptor	4 years ago
Will Hunt	c276bd9969	Send some ephemeral events to appservices (#8437) ... Optionally sends typing, presence, and read receipt information to appservices.	4 years ago
Richard van der Hoff	470dedd266	Combine the two sets of DeferredCache tests	4 years ago
Richard van der Hoff	4182bb812f	move DeferredCache into its own module	4 years ago
Richard van der Hoff	9f87da0a84	Rename Cache->DeferredCache	4 years ago
Richard van der Hoff	a34b17e492	Simplify `_locally_reject_invite` ... Update `EventCreationHandler.create_event` to accept an auth_events param, and use it in `_locally_reject_invite` instead of reinventing the wheel.	4 years ago
Erik Johnston	8de3703d21	Make event persisters periodically announce position over replication. (#8499) ... Currently background proccesses stream the events stream use the "minimum persisted position" (i.e. `get_current_token()`) rather than the vector clock style tokens. This is broadly fine as it doesn't matter if the background processes lag a small amount. However, in extreme cases (i.e. SyTests) where we only write to one event persister the background processes will never make progress. This PR changes it so that the `MultiWriterIDGenerator` keeps the current position of a given instance as up to date as possible (i.e using the latest token it sees if its not in the process of persisting anything), and then periodically announces that over replication. This then allows the "minimum persisted position" to advance, albeit with a small lag.	4 years ago
Erik Johnston	fa8934b175	Reduce serialization errors in MultiWriterIdGen (#8456) ... We call `_update_stream_positions_table_txn` a lot, which is an UPSERT that can conflict in `REPEATABLE READ` isolation level. Instead of doing a transaction consisting of a single query we may as well run it outside of a transaction.	4 years ago
Erik Johnston	ae5b2a72c0	Reduce serialization errors in MultiWriterIdGen (#8456) ... We call `_update_stream_positions_table_txn` a lot, which is an UPSERT that can conflict in `REPEATABLE READ` isolation level. Instead of doing a transaction consisting of a single query we may as well run it outside of a transaction.	4 years ago
Erik Johnston	e3debf9682	Add logging on startup/shutdown (#8448) ... This is so we can tell what is going on when things are taking a while to start up. The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.	4 years ago
Erik Johnston	7941372ec8	Make token serializing/deserializing async (#8427) ... The idea is that in future tokens will encode a mapping of instance to position. However, we don't want to include the full instance name in the string representation, so instead we'll have a mapping between instance name and an immutable integer ID in the DB that we can use instead. We'll then do the lookup when we serialize/deserialize the token (we could alternatively pass around an `Instance` type that includes both the name and ID, but that turns out to be a lot more invasive).	4 years ago
Richard van der Hoff	6d2d42f8fb	Rewrite BucketCollector ... This was a bit unweildy for what I wanted: in particular, I wanted to assign each measurement straight into a bucket, rather than storing an intermediate Counter which didn't do any bucketing at all. I've replaced it with something that is hopefully a bit easier to use. (I'm not entirely sure what the difference between a HistogramMetricFamily and a GaugeHistogramMetricFamily is, but given our counters can go down as well as up the latter *sounds* more accurate?)	4 years ago
Erik Johnston	ea70f1c362	Various clean ups to room stream tokens. (#8423)	4 years ago
Erik Johnston	b1433bf231	Don't table scan events on worker startup (#8419) ... * Fix table scan of events on worker startup. This happened because we assumed "new" writers had an initial stream position of 0, so the replication code tried to fetch all events written by the instance between 0 and the current position. Instead, set the initial position of new writers to the current persisted up to position, on the assumption that new writers won't have written anything before that point. * Consider old writers coming back as "new". Otherwise we'd try and fetch entries between the old stale token and the current position, even though it won't have written any rows. Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	4 years ago
Will Hunt	8676d8ab2e	Filter out appservices from mau count (#8404) ... This is an attempt to fix #8403.	4 years ago
Erik Johnston	bd380d942f	Add checks for postgres sequence consistency (#8402)	4 years ago
Erik Johnston	f112cfe5bb	Fix MultiWriteIdGenerator's handling of restarts. (#8374) ... On startup `MultiWriteIdGenerator` fetches the maximum stream ID for each instance from the table and uses that as its initial "current position" for each writer. This is problematic as a) it involves either a scan of events table or an index (neither of which is ideal), and b) if rows are being persisted out of order elsewhere while the process restarts then using the maximum stream ID is not correct. This could theoretically lead to race conditions where e.g. events that are persisted out of order are not sent down sync streams. We fix this by creating a new table that tracks the current positions of each writer to the stream, and update it each time we finish persisting a new entry. This is a relatively small overhead when persisting events. However for the cache invalidation stream this is a much bigger relative overhead, so instead we note that for invalidation we don't actually care about reliability over restarts (as there's no caches to invalidate) and simply don't bother reading and writing to the new table in that particular case.	4 years ago
Erik Johnston	cbabb312e0	Use `async with` for ID gens (#8383) ... This will allow us to hit the DB after we've finished using the generated stream ID.	4 years ago
Patrick Cloke	8a4a4186de	Simplify super() calls to Python 3 syntax. (#8344) ... This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py	4 years ago
Erik Johnston	deedb91732	Fix `MultiWriterIdGenerator.current_position`. (#8257) ... It did not correctly handle IDs finishing being persisted out of order, resulting in the `current_position` lagging until new IDs are persisted.	4 years ago