164 Commits (9d0098595e49755e420eb6f654d5e2e2327868fe)

Author	SHA1	Message	Date
Richard van der Hoff	b38bdae3a2	Fix AssertionErrors after purging events (#11642) ... * Fix AssertionErrors after purging events If you purged a bunch of events from your database, and then restarted synapse without receiving more events, then you would get a bunch of AssertionErrors on restart. This fixes the situation by rewinding the stream processors. * `check-newsfragment`: ignore deleted newsfiles	3 years ago
Shay	13c974ed35	Drop Bionic from Debian builds (#11633) ... * update Trove classifiers to remove py36 * stop building bionic * update dh-virtualenv * newsfragment * fix newsfragment * update version refs * another try at correct tag * Update changelog	3 years ago
Olivier Wilkinson (reivilibre)	4dd9ea8f4f	Revert "Revert accidental fast-forward merge from v1.49.0rc1" ... This reverts commit 158d73ebdd.	3 years ago
Sean Quah	158d73ebdd	Revert accidental fast-forward merge from v1.49.0rc1 ... Revert "Sort internal changes in changelog" Revert "Update CHANGES.md" Revert "1.49.0rc1" Revert "Revert "Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505) (#11527)" Revert "Refactors in `_generate_sync_entry_for_rooms` (#11515)" Revert "Correctly register shutdown handler for presence workers (#11518)" Revert "Fix `ModuleApi.looping_background_call` for non-async functions (#11524)" Revert "Fix 'delete room' admin api to work on incomplete rooms (#11523)" Revert "Correctly ignore invites from ignored users (#11511)" Revert "Fix the test breakage introduced by #11435 as a result of concurrent PRs (#11522)" Revert "Stabilise support for MSC2918 refresh tokens as they have now been merged into the Matrix specification. (#11435)" Revert "Save the OIDC session ID (sid) with the device on login (#11482)" Revert "Add admin API to get some information about federation status (#11407)" Revert "Include bundled aggregations in /sync and related fixes (#11478)" Revert "Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)" Revert "Update backward extremity docs to make it clear that it does not indicate whether we have fetched an events' `prev_events` (#11469)" Revert "Support configuring the lifetime of non-refreshable access tokens separately to refreshable access tokens. (#11445)" Revert "Add type hints to `synapse/tests/rest/admin` (#11501)" Revert "Revert accidental commits to develop." Revert "Newsfile" Revert "Give `tests.server.setup_test_homeserver` (nominally!) the same behaviour" Revert "Move `tests.utils.setup_test_homeserver` to `tests.server`" Revert "Convert one of the `setup_test_homeserver`s to `make_test_homeserver_synchronous`" Revert "Disambiguate queries on `state_key` (#11497)" Revert "Comments on the /sync tentacles (#11494)" Revert "Clean up tests.storage.test_appservice (#11492)" Revert "Clean up `tests.storage.test_main` to remove use of legacy code. (#11493)" Revert "Clean up `tests.test_visibility` to remove legacy code. (#11495)" Revert "Minor cleanup on recently ported doc pages (#11466)" Revert "Add most of the missing type hints to `synapse.federation`. (#11483)" Revert "Avoid waiting for zombie processes in `synctl stop` (#11490)" Revert "Fix media repository failing when media store path contains symlinks (#11446)" Revert "Add type annotations to `tests.storage.test_appservice`. (#11488)" Revert "`scripts-dev/sign_json`: support for signing events (#11486)" Revert "Add MSC3030 experimental client and federation API endpoints to get the closest event to a given timestamp (#9445)" Revert "Port wiki pages to documentation website (#11402)" Revert "Add a license header and comment. (#11479)" Revert "Clean-up get_version_string (#11468)" Revert "Link background update controller docs to summary (#11475)" Revert "Additional type hints for config module. (#11465)" Revert "Register the login redirect endpoint for v3. (#11451)" Revert "Update openid.md" Revert "Remove mention of OIDC certification from Dex (#11470)" Revert "Add a note about huge pages to our Postgres doc (#11467)" Revert "Don't start Synapse master process if `worker_app` is set (#11416)" Revert "Expose worker & homeserver as entrypoints in `setup.py` (#11449)" Revert "Bundle relations of relations into the `/relations` result. (#11284)" Revert "Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)" Revert "Eliminate a few `Any`s in `LruCache` type hints (#11453)" Revert "Remove unnecessary `json.dumps` from `tests.rest.admin` (#11461)" Revert "Merge branch 'master' into develop" This reverts commit 26b5d2320f. This reverts commit bce4220f38. This reverts commit 966b5d0fa0. This reverts commit 088d748f2c. This reverts commit 14d593f72d. This reverts commit 2a3ec6facf. This reverts commit eccc49d755. This reverts commit b1ecd19c5d. This reverts commit 9c55dedc8c. This reverts commit 2d42e586a8. This reverts commit 2f053f3f82. This reverts commit a15a893df8. This reverts commit 8b4b153c9e. This reverts commit 494ebd7347. This reverts commit a77c369897. This reverts commit 4eb77965cd. This reverts commit 637df95de6. This reverts commit e5f426cd54. This reverts commit 8cd68b8102. This reverts commit 6cae125e20. This reverts commit 7be88fbf48. This reverts commit b3fd99b74a. This reverts commit f7ec6e7d9e. This reverts commit 5640992d17. This reverts commit d26808dd85. This reverts commit f91624a595. This reverts commit 16d39a5490. This reverts commit 8a4c296987. This reverts commit 49e1356ee3. This reverts commit d2279f471b. This reverts commit b50e39df57. This reverts commit 858d80bf0f. This reverts commit 435f044807. This reverts commit f61462e1be. This reverts commit a6f1a3abec. This reverts commit 84dc50e160. This reverts commit ed635d3285. This reverts commit 7b62791e00. This reverts commit 153194c771. This reverts commit f44d729d4c. This reverts commit a265fbd397. This reverts commit b9fef1a7cd. This reverts commit b0eb64ff7b. This reverts commit f1795463bf. This reverts commit 70cbb1a5e3. This reverts commit 42bf020463. This reverts commit 379f2650cf. This reverts commit 7ff22d6da4. This reverts commit 5a0b652d36. This reverts commit 432a174bc1. This reverts commit b14f8a1baf, reversing changes made to e713855dca.	3 years ago
Richard van der Hoff	f61462e1be	`scripts-dev/sign_json`: support for signing events (#11486)	3 years ago
Patrick Cloke	a4521ce0a8	Support the stable /hierarchy endpoint from MSC2946 (#11329) ... This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.	3 years ago
Patrick Cloke	9d1971a5c4	Return the stable `event` field from `/send_join` per MSC3083. (#11413) ... This does not remove the unstable field and still parses both. Handling of the unstable field will need to be removed in the future.	3 years ago
Patrick Cloke	5505da2109	Remove msc2716 from the list of tests for complement. (#11389) ... As the tests are currently failing and not run in CI.	3 years ago
Patrick Cloke	60ecb6b4d4	Fix running complement.sh script. (#11368) ... By reverting changes from #11166 in this script. Specifically commit 13f084eb58.	3 years ago
Dan Callahan	d7141e0b8b	Fix Shellcheck SC2006: Use $(...) notation ... Use $(...) notation instead of legacy backticked `...`. https://github.com/koalaman/shellcheck/wiki/SC2006 Signed-off-by: Dan Callahan <danc@element.io>	3 years ago
Dan Callahan	13f084eb58	Fix Shellcheck SC2086: Quote to prevent splitting ... Double quote to prevent globbing and word splitting. https://github.com/koalaman/shellcheck/wiki/SC2086 Signed-off-by: Dan Callahan <danc@element.io>	3 years ago
Dan Callahan	6a9d84a676	Fix Shellcheck SC2166: test -a is not well defined ... Prefer [ p ] && [ q ] as [ p -a q ] is not well defined. https://github.com/koalaman/shellcheck/wiki/SC2166 Signed-off-by: Dan Callahan <danc@element.io>	3 years ago
Dan Callahan	64adbb7b54	Fix Shellcheck SC2046: Quote to prevent word split ... Quote this to prevent word splitting https://www.shellcheck.net/wiki/SC2046 Signed-off-by: Dan Callahan <danc@element.io>	3 years ago
Dan Callahan	12d79ff1b6	Fix Shellcheck SC2164: exit in case cd fails. ... Use `cd ... || exit` in case cd fails. https://github.com/koalaman/shellcheck/wiki/SC2164 Signed-off-by: Dan Callahan <danc@element.io>	3 years ago
Richard van der Hoff	57501d9194	Update `sign_json` to support inline key config (#11139) ... It's been possible to configure a key inline in the homeserver.yaml since 13bc1e0746. Update `sign_json` to work with this.	3 years ago
Hillery Shay	5dcacdf6d1	Add support for ubuntu 21.10 "Impish Indri" (#11024) ... * support ubuntu 21.10 indri * add changelog * update to correct codename Co-authored-by: Brendan Abolivier <github@brendanabolivier.com> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>	3 years ago
reivilibre	b742cb2e4a	Release script improvements (#10966) ... Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	3 years ago
Nick Barrett	c80878d22a	Add `--run-background-updates` option to `update_database` script. (#10954) ... Signed-off-by: Nick Barrett <nick@beeper.com>	3 years ago
Erik Johnston	d37841787a	Sign the git tag in release script (#10925)	3 years ago
David Robertson	ea01d4c2de	Update postgresql testing script (#10906) ... - Use sytest:bionic. Sytest:latest is two years old (do we want CI to push out latest at all?) and comes with Python 3.5, which we explictly no longer support. The script now runs under PostgreSQL 10 as a result. - Advertise script in the docs - Move pg testing script to scripts-dev directory - Write to host as the script's exector, not root A few changes to make it speedier to re-run the tests: - Create blank DB in the container, not the script, so we don't have to `initdb` each time - Use a named volume to persist the tox environment, so we don't have to fetch and install a bunch of packages from PyPI each time Co-authored-by: reivilibre <olivier@librepush.net>	3 years ago
Brendan Abolivier	5154afc00d	Add a script to update the debian changelog for non-Debian systems (#10778) ... When releasing 1.42.0 with @Azrenbeth and talking with @clokep yesterday I realised doing the dch incantations related to releasing Synapse wasn't trivial on eg a macOS system, so this is a script to run in a Debian container to make things a bit easier.	3 years ago
Patrick Cloke	882539e423	Ensure the base Docker image is rebuilt when running complement with workers. (#10686) ... We now always rebuild the matrixdotorg/synapse image, then build the matrixdotorg/synapse-workers image on top of it.	3 years ago
Richard van der Hoff	c8132f4a31	Build debs for bookworm (#10612)	3 years ago
David Robertson	d2ad397d3c	Stop building a debian package for Groovy Gorilla (#10588)	3 years ago
David Robertson	314a739160	Also rename in lint.sh	3 years ago
David Robertson	c0ebdfc77e	Kill off the .buildkite dir completely	3 years ago
David Robertson	03fb99a5c8	check-newsfragment: pass pr number explicitly ... use PULL_REQUEST_NUMBER instead of BUILDKITE_PULL_REQUEST remove the other user of BUILDKITE_PULL_REQUEST, namely merge_base_branch.sh	3 years ago
Eric Eastwood	684d19a11c	Add support for MSC2716 marker events (#10498) ... * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(*args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, *args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, *args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` * Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)	3 years ago
Erik Johnston	6878e10653	Fix release script URL (#10516)	3 years ago
Erik Johnston	a7bacccd85	Extend the release script to tag and create the releases. (#10496)	3 years ago
Patrick Cloke	13944678c3	Use new go test running syntax for complement. (#10488) ... Updates CI and the helper script t ensures all tests are run (in parallel).	3 years ago
Erik Johnston	92a882254b	Change release script to update debian changelog for RCs (#10465)	3 years ago
Richard van der Hoff	f22252d4f9	Enable docker image caching for the deb build (#10431)	3 years ago
Eric Eastwood	cd5fcd2731	Disable msc2716 until Complement update is merged (#10463)	3 years ago
Richard van der Hoff	5f2848f379	build debs in GHA (#10247) ... GHA workflow to build the debs	3 years ago
Brendan Abolivier	d731ed70d9	Fixes to the release script (#10239) ... * rename major/minor into the right semver terminology minor/patch (since this was something that got me very confused the first couple of times I've used the script) * name the release branch based on the new version, not the previous one	3 years ago
Richard van der Hoff	8beead66ae	Send out invite rejections and knocks over federation (#10223) ... ensure that events sent via `send_leave` and `send_knock` are sent on to the rest of the federation.	3 years ago
Eric Eastwood	96f6293de5	Add endpoints for backfilling history (MSC2716) (#9247) ... Work on https://github.com/matrix-org/matrix-doc/pull/2716	3 years ago
Dan Callahan	7dc14730d9	Name release branches just after major.minor (#10013) ... With the prior format, 1.33.0 / 1.33.1 / 1.33.2 got separate branches: release-v1.33.0 release-v1.33.1 release-v1.33.2 Under the new model, all three would share a common branch: release-v1.33 As before, RCs and actual releases exist as tags on these branches. This better reflects our support model, e.g., that the "1.33" series had a formal release followed by two patches / updates. Signed-off-by: Dan Callahan <danc@element.io>	4 years ago
Richard van der Hoff	5447a76332	Remove redundant, unmaintained `convert_server_keys` script. (#10055)	4 years ago
Patrick Cloke	7d90d6ce9b	Run complement with Synapse workers manually. (#10039) ... Adds an option to complement.sh to run Synapse in worker mode (instead of the default monolith mode).	4 years ago
Jerin J Titus	057ce7b754	Remove tls_fingerprints option (#9280) ... Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>	4 years ago
Richard van der Hoff	6482075c95	Run `black` on the scripts (#9981) ... Turns out these scripts weren't getting linted.	4 years ago
Richard van der Hoff	7562d887e1	Change the format of access tokens away from macaroons (#5588)	4 years ago
Richard van der Hoff	765473567c	Fix make_full_schema to create the db with the right options and user (#9931)	4 years ago
Dan Callahan	56c4b47df3	Build Debian packages for Ubuntu 21.04 Hirsute (#9909) ... Signed-off-by: Dan Callahan <danc@element.io>	4 years ago
Jonathan de Jong	495b214f4f	Fix (final) Bugbear violations (#9838)	4 years ago
Jonathan de Jong	4b965c862d	Remove redundant "coding: utf-8" lines (#9786) ... Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	4 years ago
Erik Johnston	c1dbe84c3d	Add release helper script (#9713) ... Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	4 years ago
Dan Callahan	3efde8b69a	Add option to skip unit tests when building debs (#9793) ... Signed-off-by: Dan Callahan <danc@element.io>	4 years ago