watcha-synapse

Commit Graph

Author	SHA1	Message	Date
Patrick Cloke	882539e423	Ensure the base Docker image is rebuilt when running complement with workers. (#10686 ) We now always rebuild the matrixdotorg/synapse image, then build the matrixdotorg/synapse-workers image on top of it.	3 years ago
Richard van der Hoff	c8132f4a31	Build debs for bookworm (#10612 )	3 years ago
David Robertson	d2ad397d3c	Stop building a debian package for Groovy Gorilla (#10588 )	3 years ago
David Robertson	314a739160	Also rename in lint.sh	3 years ago
David Robertson	c0ebdfc77e	Kill off the .buildkite dir completely	3 years ago
David Robertson	03fb99a5c8	check-newsfragment: pass pr number explicitly use PULL_REQUEST_NUMBER instead of BUILDKITE_PULL_REQUEST remove the other user of BUILDKITE_PULL_REQUEST, namely merge_base_branch.sh	3 years ago
Eric Eastwood	684d19a11c	Add support for MSC2716 marker events (#10498 ) * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(args, kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, args, *kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(args, *kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(args, *kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, args, *kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, args, *kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, args, *kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(args, *kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)	3 years ago
Erik Johnston	6878e10653	Fix release script URL (#10516 )	3 years ago
Erik Johnston	a7bacccd85	Extend the release script to tag and create the releases. (#10496 )	3 years ago
Patrick Cloke	13944678c3	Use new go test running syntax for complement. (#10488 ) Updates CI and the helper script t ensures all tests are run (in parallel).	3 years ago
Erik Johnston	92a882254b	Change release script to update debian changelog for RCs (#10465 )	3 years ago
Richard van der Hoff	f22252d4f9	Enable docker image caching for the deb build (#10431 )	3 years ago
Eric Eastwood	cd5fcd2731	Disable msc2716 until Complement update is merged (#10463 )	3 years ago
Richard van der Hoff	5f2848f379	build debs in GHA (#10247 ) GHA workflow to build the debs	3 years ago
Brendan Abolivier	d731ed70d9	Fixes to the release script (#10239 ) * rename major/minor into the right semver terminology minor/patch (since this was something that got me very confused the first couple of times I've used the script) * name the release branch based on the new version, not the previous one	3 years ago
Richard van der Hoff	8beead66ae	Send out invite rejections and knocks over federation (#10223 ) ensure that events sent via `send_leave` and `send_knock` are sent on to the rest of the federation.	3 years ago
Eric Eastwood	96f6293de5	Add endpoints for backfilling history (MSC2716) (#9247 ) Work on https://github.com/matrix-org/matrix-doc/pull/2716	3 years ago
Dan Callahan	7dc14730d9	Name release branches just after major.minor (#10013 ) With the prior format, 1.33.0 / 1.33.1 / 1.33.2 got separate branches: release-v1.33.0 release-v1.33.1 release-v1.33.2 Under the new model, all three would share a common branch: release-v1.33 As before, RCs and actual releases exist as tags on these branches. This better reflects our support model, e.g., that the "1.33" series had a formal release followed by two patches / updates. Signed-off-by: Dan Callahan <danc@element.io>	4 years ago
Richard van der Hoff	5447a76332	Remove redundant, unmaintained `convert_server_keys` script. (#10055 )	4 years ago
Patrick Cloke	7d90d6ce9b	Run complement with Synapse workers manually. (#10039 ) Adds an option to complement.sh to run Synapse in worker mode (instead of the default monolith mode).	4 years ago
Jerin J Titus	057ce7b754	Remove tls_fingerprints option (#9280 ) Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>	4 years ago
Richard van der Hoff	6482075c95	Run `black` on the scripts (#9981 ) Turns out these scripts weren't getting linted.	4 years ago
Richard van der Hoff	7562d887e1	Change the format of access tokens away from macaroons (#5588 )	4 years ago
Richard van der Hoff	765473567c	Fix make_full_schema to create the db with the right options and user (#9931 )	4 years ago
Dan Callahan	56c4b47df3	Build Debian packages for Ubuntu 21.04 Hirsute (#9909 ) Signed-off-by: Dan Callahan <danc@element.io>	4 years ago
Jonathan de Jong	495b214f4f	Fix (final) Bugbear violations (#9838 )	4 years ago
Jonathan de Jong	4b965c862d	Remove redundant "coding: utf-8" lines (#9786 ) Part of #9744 Removes all redundant `# -- coding: utf-8 --` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	4 years ago
Erik Johnston	c1dbe84c3d	Add release helper script (#9713 ) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	4 years ago
Dan Callahan	3efde8b69a	Add option to skip unit tests when building debs (#9793 ) Signed-off-by: Dan Callahan <danc@element.io>	4 years ago
Patrick Cloke	abc814dcbf	Enable complement tests for MSC2946. (#9771 ) By providing the additional build tag for `msc2946`.	4 years ago
Richard van der Hoff	9278eb701e	drop support for stretch and xenial	4 years ago
Patrick Cloke	452991527a	MSC3083: Check for space membership during a local join of restricted rooms. (#9735 ) When joining a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.	4 years ago
Patrick Cloke	5fe38e07e7	Revert "Use 'dmypy run' in lint.sh instead of 'mypy' (#9701 )" (#9720 )	4 years ago
Andrew Morgan	ac99774dac	Rewrite complement.sh (#9685 ) This PR rewrites the original complement.sh script with a number of improvements: * We can now use a local checkout of Complement (configurable with `COMPLEMENT_DIR`), though the default behaviour still downloads the master branch. * You can now specify a regex of test names to run, or just run all tests. * We now use the Synapse test blacklist tag (so all tests will pass).	4 years ago
Andrew Morgan	f380bb77d1	Use 'dmypy run' in lint.sh instead of 'mypy' (#9701 ) For it's obvious performance benefits. `dmypy` support landed in #9692.	4 years ago
Quentin Gliech	d4c4798a25	Use interpreter from $PATH instead of absolute paths in various scripts using /usr/bin/env (#9689 ) On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work. `/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>	4 years ago
Richard van der Hoff	0e35584734	federation_client: handle inline signing_keys in hs.yaml (#9647 )	4 years ago
Richard van der Hoff	201178db1a	federation_client: stop adding URL prefix (#9645 )	4 years ago
Andrew Morgan	22db45bd4d	Prevent the config-lint script erroring out on any sample_config changes (#9562 ) I noticed that I'd occasionally have `scripts-dev/lint.sh` fail when messing about with config options in my PR. The script calls `scripts-dev/config-lint.sh`, which attempts some validation on the sample config. It does this by using `sed` to edit the sample_config, and then seeing if the file changed using `git diff`. The problem is: if you changed the sample_config as part of your commit, this script will error regardless. This PR attempts to change the check so that existing, unstaged changes to the sample_config will not cause the script to report an invalid file.	4 years ago
Eric Eastwood	0a00b7ff14	Update black, and run auto formatting over the codebase (#9381 ) - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](`80d6dc9783/docs/code_style.md`) - Update `code_style.md` docs around installing black to use the correct version	4 years ago
Eric Eastwood	80d6dc9783	Remove conflicting sqlite tables that are "reserved" (shadow fts4 tables) (#9003 ) Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests. Fix #8996	4 years ago
Andrew Morgan	31c5382d7a	Align the directories linted in CI with the defaults in scripts-dev/lint.sh (#9191 ) The lists of source directories to lint between `tox.ini` and `lint.sh` became out of sync. This PR tightens them up and adds some comments reminding any future readers to keep the list in sync.	4 years ago
Patrick Cloke	5d34f40d49	Add type hints to the push module. (#8901 )	4 years ago
Richard van der Hoff	97b35ee259	Add a script to sign arbitrary json objects. (#8772 )	4 years ago
Erik Johnston	22eeb6bc54	Fix cache call signature to accept `on_invalidate`. (#8684 ) Cached functions accept an `on_invalidate` function, which we failed to add to the type signature. It's rarely used in the files that we have typed, which is why we haven't noticed it before.	4 years ago
Patrick Cloke	00b24aa545	Support generating structured logs in addition to standard logs. (#8607 ) This modifies the configuration of structured logging to be usable from the standard Python logging configuration. This also separates the formatting of logs from the transport allowing JSON logs to files or standard logs to sockets.	4 years ago
kleph	29ce6d43b5	Run mypy as part of the lint.sh script. (#8633 )	4 years ago
Andrew Morgan	654e239b25	Add option to scripts-dev/lint.sh to only lint files changed since the last git commit (#8472 ) This PR makes several changes to the `./scripts-dev/lint.sh` script, which lints the codebase with a number of tools: * Adds usage information, with `-h` flag to show it. Otherwise it will show when providing an unknown flag. * Adds option `-d` which will check both staged and unstaged files that have changed since the last commit and add them to the list of files to lint. - Note that only files without an extension, or with a `.py` extension will be allowed. This prevents editing bash scripts causing the linters to break on non-python files. * Improves the print-out of which files/directories are being linted.	4 years ago
Patrick Cloke	d9b55bd830	Add Ubuntu 20.10 (Groovy Gorilla) to build scripts. (#8475 )	4 years ago
Andrew Morgan	01f82bfe32	Remove docs/sphinx and related references (#8480 ) https://github.com/matrix-org/synapse/tree/develop/docs/sphinx doesn't seem to really be utilised or changed recently since the initial commit. I like the idea of exportable documentation of the codebase, but at the moment after running through the build instructions the generated website wasn't very useful...	4 years ago

1 2 3

145 Commits (e1641b46d19c9745f512e623544b2bddfc89551d)