watcha-synapse

Commit Graph

Author	SHA1	Message	Date
Nick Mills-Barrett	2ee0b6ef4b	Safe async event cache (#13308 ) Fix race conditions in the async cache invalidation logic, by separating the async & local invalidation calls and ensuring any async call i executed first. Signed off by Nick @ Beeper (@Fizzadar).	2 years ago
Nick Mills-Barrett	cc21a431f3	Async get event cache prep (#13242 ) Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches. Signed off by Nick @ Beeper (@Fizzadar)	2 years ago
Brendan Abolivier	4d3b8fb23f	Don't actually one-line the SQL statements we send to the DB (#13129 )	2 years ago
Erik Johnston	0d1d3e0708	Speed up `get_unread_event_push_actions_by_room` (#13005 ) Fixes #11887 hopefully. The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller. This needs two major changes: 1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it 2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`. In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.	2 years ago
reivilibre	fbf904bd54	Fix media thumbnails being unusable before the index had been added in the background. (#12823 )	3 years ago
David Robertson	17e1eb7749	Reduce the number of "untyped defs" (#12716 )	3 years ago
David Robertson	fa0eab9c8e	Use `ParamSpec` in a few places (#12667 )	3 years ago
David Robertson	26c1ad71c5	Use `Concatenate` to annotate `do_execute` (#12666 )	3 years ago
Sean Quah	a50fb411b3	Update `delay_cancellation` to accept any awaitable (#12468 ) This will mainly be useful when dealing with module callbacks, which are all typed as returning `Awaitable`s instead of coroutines or `Deferred`s. Signed-off-by: Sean Quah <seanq@element.io>	3 years ago
reivilibre	e630722f11	Optimise `_update_client_ips_batch_txn` to batch together database operations. (#12252 ) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	3 years ago
Erik Johnston	66053b6bfb	Prefill more stream change caches. (#12372 )	3 years ago
David Robertson	a7293ef16f	Bump `black` and `click` versions (#12320 )	3 years ago
Sean Quah	8a519f8abc	Update `LoggingTransaction.call_after` and `call_on_exception` docstrings (#12315 ) Document the behaviour of `LoggingTransaction.call_after` and `LoggingTransaction.call_on_exception` when transactions are retried. Signed-off-by: Sean Quah <seanq@element.io>	3 years ago
David Robertson	a2b00a4486	Bump `black` and `click` versions (#12320 )	3 years ago
reivilibre	89f11f8c6f	Improve type annotations for `execute_values`. (#12311 )	3 years ago
David Robertson	f4c5e5864c	Use psycopg2 type stubs (#12269 )	3 years ago
Sean Quah	6121056740	Handle cancellation in `DatabasePool.runInteraction()` (#12199 ) To handle cancellation, we ensure that `after_callback`s and `exception_callback`s are always run, since the transaction will complete on another thread regardless of cancellation. We also wait until everything is done before releasing the `CancelledError`, so that logging contexts won't get used after they have been finished. Signed-off-by: Sean Quah <seanq@element.io>	3 years ago
Brendan Abolivier	6d482ba259	Pass `isolation_level` to `runWithConnection` (#11847 ) This was missed in https://github.com/matrix-org/synapse/pull/11799	3 years ago
Nick Barrett	b59d285f7c	Db txn set isolation level (#11799 ) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>	3 years ago
Patrick Cloke	3e0536cd2a	Replace uses of simple_insert_many with simple_insert_many_values. (#11742 ) This should be (slightly) more efficient and it is simpler to have a single method for inserting multiple values.	3 years ago
Patrick Cloke	10a88ba91c	Use auto_attribs/native type hints for attrs classes. (#11692 )	3 years ago
Patrick Cloke	f901f8b70e	Require Collections as the parameters for simple_* methods. (#11580 ) Instead of Iterable since the generators are not allowed due to the potential for their re-use.	3 years ago
Richard van der Hoff	ff6fd52160	checks for generators in database functions (#11564 ) A couple of safety-checks to hopefully stop people doing what I just did, and create a storage function which only works the first time it is called (and not when it is re-run due to a database concurrency error or similar).	3 years ago
Sean Quah	5305a5e881	Type hint the constructors of the data store classes (#11555 )	3 years ago
Richard van der Hoff	f0562183e7	skip some dict munging in event persistence (#11560 ) Create a new dict helper method `simple_insert_many_values_txn`, which takes raw row values, rather than {key=>value} dicts. This saves us a bunch of dict munging, and makes it easier to use generators rather than creating intermediate lists and dicts.	3 years ago
Sean Quah	84fac0f814	Add type annotations to `synapse.metrics` (#10847 )	3 years ago
Erik Johnston	4ee71b9637	Add some background update admin APIs (#11263 ) Fixes #11259	3 years ago
Erik Johnston	82d2168a15	Add metrics to the threadpools (#11178 )	3 years ago
Sean Quah	2b82ec425f	Add type hints for most `HomeServer` parameters (#11095 )	3 years ago
reivilibre	437961744c	Fix remove_stale_pushers job on SQLite. (#10843 )	3 years ago
Erik Johnston	0eae330a26	Use `execute_values` more in PostgreSQL (#10754 ) `execute_values` is a faster version of `execute_batch`.	3 years ago
Eric Eastwood	684d19a11c	Add support for MSC2716 marker events (#10498 ) * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(args, kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, args, *kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(args, *kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(args, *kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, args, *kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, args, *kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, args, *kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(args, *kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)	3 years ago
Toni Spets	ba5287f5e8	Allow setting transaction limit for db connections (#10440 ) Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool. Signed-off-by: Toni Spets <toni.spets@iki.fi>	3 years ago
Erik Johnston	38b346a504	Replace `or_ignore` in `simple_insert` with `simple_upsert` (#10442 ) Now that we have `simple_upsert` that should be used in preference to trying to insert and looking for an exception. The main benefit is that we ERROR message don't get written to postgres logs. We also have tidy up the return value on `simple_upsert`, rather than having a tri-state of inserted/not-inserted/unknown.	3 years ago
Jonathan de Jong	95e47b2e78	[pyupgrade] `synapse/` (#10348 ) This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744	3 years ago
Jonathan de Jong	bdfde6dca1	Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381 )	3 years ago
Erik Johnston	6655ea5587	Add script for getting info about recently registered users (#10290 )	3 years ago
Erik Johnston	1092718cac	Fix logging context when opening new DB connection (#10141 ) Fixes #10140	4 years ago
Richard van der Hoff	0acb5010ec	More database opentracing (#10136 ) Add a couple of extra logs/spans, to give a bit of a better idea.	4 years ago
Richard van der Hoff	9eea4646be	Add OpenTracing for database activity. (#10113 ) This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels: * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions. * emit a span while we hold a database connection open * emit a span for each database transaction - actual actual transaction. * emit a span for each database query. I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).	4 years ago
Patrick Cloke	10a08ab88a	Use the parent's logging context name for runWithConnection. (#9895 ) This fixes a regression where the logging context for runWithConnection was reported as runWithConnection instead of the connection name, e.g. "POST-XYZ".	4 years ago
Andrew Morgan	fe604a022a	Remove various bits of compatibility code for Python <3.6 (#9879 ) I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.	4 years ago
Richard van der Hoff	294c675033	Remove `synapse.types.Collection` (#9856 ) This is no longer required, since we have dropped support for Python 3.5.	4 years ago
Jonathan de Jong	4b965c862d	Remove redundant "coding: utf-8" lines (#9786 ) Part of #9744 Removes all redundant `# -- coding: utf-8 --` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	4 years ago
Dan Callahan	1d5f0e3529	Bump black configuration to target py36 (#9781 ) Signed-off-by: Dan Callahan <danc@element.io>	4 years ago
Jonathan de Jong	2ca4e349e9	Bugbear: Add Mutable Parameter fixes (#9682 ) Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>	4 years ago
Richard van der Hoff	24c58ebfc9	remove unused param on `make_tuple_comparison_clause`	4 years ago
Richard van der Hoff	3ada9b4264	Drop support for sqlite<3.22 as well	4 years ago
Andrew Morgan	fae81f2f68	Add a storage method for returning all current presence from all users (#9650 ) Split off from https://github.com/matrix-org/synapse/pull/9491 Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: `b588f16e39/synapse/handlers/presence.py (L1131-L1133)` Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.	4 years ago
Jonathan de Jong	4c3827f2c1	Enable addtional flake8-bugbear linting checks. (#9659 )	4 years ago

1 2 3

150 Commits (watcha-develop)