watcha
/
watcha-synapse
mirror of https://github.com/watcha-fr/synapse
1
1
Fork 0
Code Issues Projects Releases Wiki Activity

Recommend poetry in docs (#12475)

Browse Source 
* Recommend poetry in docs

- readme
- contributor guide
- upgrade notes
- new dev cheat sheet for poetry

Co-authored-by: Shay <hillerys@element.io>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
code_spécifique_watcha
David Robertson 3 years ago committed by GitHub
parent d0c1f4ca4c
commit ecef741add
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 301 additions and 50 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 31
      README.rst
  2. 1
      changelog.d/12475.doc
  3. 46
      docs/development/contributing_guide.md
  4. 239
      docs/development/dependencies.md
  5. 34
      docs/upgrade.md

31
README.rst
Unescape View File

@ -293,24 +293,27 @@ directory of your choice::
git clone https://github.com/matrix-org/synapse.git
cd synapse
Synapse has a number of external dependencies, that are easiest
to install using pip and a virtualenv::
Synapse has a number of external dependencies. We maintain a fixed development
environment using [poetry](https://python-poetry.org/). First, install poetry. We recommend
python3 -m venv ./env
source ./env/bin/activate
pip install -e ".[all,dev]"
pip install --user pipx
pipx install poetry
This will run a process of downloading and installing all the needed
dependencies into a virtual env. If any dependencies fail to install,
try installing the failing modules individually::
as described `here <https://python-poetry.org/docs/#installing-with-pipx>`_.
(See `poetry's installation docs <https://python-poetry.org/docs/#installation>`
for other installation methods.) Then ask poetry to create a virtual environment
from the project and install Synapse's dependencies::
poetry install --extras "all test"
pip install -e "module-name"
This will run a process of downloading and installing all the needed
dependencies into a virtual env.
We recommend using the demo which starts 3 federated instances running on ports `8080` - `8082`
./demo/start.sh
poetry run ./demo/start.sh
(to stop, you can use `./demo/stop.sh`)
(to stop, you can use `poetry run ./demo/stop.sh`)
See the `demo documentation <https://matrix-org.github.io/synapse/develop/development/demo.html>`_
for more information.
@ -318,14 +321,14 @@ for more information.
If you just want to start a single instance of the app and run it directly::
# Create the homeserver.yaml config once
python -m synapse.app.homeserver \
poetry run synapse_homeserver \
--server-name my.domain.name \
--config-path homeserver.yaml \
--generate-config \
--report-stats=[yes|no]
# Start the app
python -m synapse.app.homeserver --config-path homeserver.yaml
poetry run synapse_homeserver --config-path homeserver.yaml
Running the unit tests
@ -334,7 +337,7 @@ Running the unit tests
After getting up and running, you may wish to run Synapse's unit tests to
check that everything is installed correctly::
trial tests
poetry run trial tests
This should end with a 'PASSED' result (note that exact numbers will
differ)::

1
changelog.d/12475.doc
Unescape View File

@ -0,0 +1 @@
Strongly recommend `poetry` for development.

46
docs/development/contributing_guide.md
Unescape View File

@ -48,19 +48,28 @@ can find many good git tutorials on the web.
# 4. Install the dependencies
Once you have installed Python 3 and added the source, please open a terminal and
setup a *virtualenv*, as follows:
Synapse uses the [poetry](https://python-poetry.org/) project to manage its dependencies
and development environment. Once you have installed Python 3 and added the
source, you should install `poetry`.
Of their installation methods, we recommend
[installing `poetry` using `pipx`](https://python-poetry.org/docs/#installing-with-pipx),
```shell
pip install --user pipx
pipx install poetry
```
but see poetry's [installation instructions](https://python-poetry.org/docs/#installation)
for other installation methods.
Next, open a terminal and install dependencies as follows:
```sh
cd path/where/you/have/cloned/the/repository
python3 -m venv ./env
source ./env/bin/activate
pip install wheel
pip install -e ".[all,dev]"
pip install tox
poetry install --extras all
```
This will install the developer dependencies for the project.
This will install the runtime and developer dependencies for the project.
# 5. Get in touch.
@ -117,11 +126,10 @@ The linters look at your code and do two things:
- ensure that your code follows the coding style adopted by the project;
- catch a number of errors in your code.
The linter takes no time at all to run as soon as you've [downloaded the dependencies into your python virtual environment](#4-install-the-dependencies).
The linter takes no time at all to run as soon as you've [downloaded the dependencies](#4-install-the-dependencies).
```sh
source ./env/bin/activate
./scripts-dev/lint.sh
poetry run ./scripts-dev/lint.sh
```
Note that this script *will modify your files* to fix styling errors.
@ -131,15 +139,13 @@ If you wish to restrict the linters to only the files changed since the last com
(much faster!), you can instead run:
```sh
source ./env/bin/activate
./scripts-dev/lint.sh -d
poetry run ./scripts-dev/lint.sh -d
```
Or if you know exactly which files you wish to lint, you can instead run:
```sh
source ./env/bin/activate
./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder
poetry run ./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder
```
## Run the unit tests (Twisted trial).
@ -148,16 +154,14 @@ The unit tests run parts of Synapse, including your changes, to see if anything
was broken. They are slower than the linters but will typically catch more errors.
```sh
source ./env/bin/activate
trial tests
poetry run trial tests
```
If you wish to only run *some* unit tests, you may specify
another module instead of `tests` - or a test class or a method:
```sh
source ./env/bin/activate
trial tests.rest.admin.test_room tests.handlers.test_admin.ExfiltrateData.test_invite
poetry run trial tests.rest.admin.test_room tests.handlers.test_admin.ExfiltrateData.test_invite
```
If your tests fail, you may wish to look at the logs (the default log level is `ERROR`):
@ -169,7 +173,7 @@ less _trial_temp/test.log
To increase the log level for the tests, set `SYNAPSE_TEST_LOG_LEVEL`:
```sh
SYNAPSE_TEST_LOG_LEVEL=DEBUG trial tests
SYNAPSE_TEST_LOG_LEVEL=DEBUG poetry run trial tests
```
By default, tests will use an in-memory SQLite database for test data. For additional
@ -180,7 +184,7 @@ database state to be stored in a file named `test.db` under the trial process'
working directory. Typically, this ends up being `_trial_temp/test.db`. For example:
```sh
SYNAPSE_TEST_PERSIST_SQLITE_DB=1 trial tests
SYNAPSE_TEST_PERSIST_SQLITE_DB=1 poetry run trial tests
```
The database file can then be inspected with:

239
docs/development/dependencies.md
Unescape View File

@ -0,0 +1,239 @@
# Managing dependencies with Poetry
This is a quick cheat sheet for developers on how to use [`poetry`](https://python-poetry.org/).
# Background
Synapse uses a variety of third-party Python packages to function as a homeserver.
Some of these are direct dependencies, listed in `pyproject.toml` under the
`[tool.poetry.dependencies]` section. The rest are transitive dependencies (the
things that our direct dependencies themselves depend on, and so on recursively.)
We maintain a locked list of all our dependencies (transitive included) so that
we can track exactly which version of each dependency appears in a given release.
See [here](https://github.com/matrix-org/synapse/issues/11537#issue-1074469665)
for discussion of why we wanted this for Synapse. We chose to use
[`poetry`](https://python-poetry.org/) to manage this locked list; see
[this comment](https://github.com/matrix-org/synapse/issues/11537#issuecomment-1015975819)
for the reasoning.
The locked dependencies get included in our "self-contained" releases: namely,
our docker images and our debian packages. We also use the locked dependencies
in development and our continuous integration.
Separately, our "broad" dependencies—the version ranges specified in
`pyproject.toml`—are included as metadata in our "sdists" and "wheels" [uploaded
to PyPI](https://pypi.org/project/matrix-synapse). Installing from PyPI or from
the Synapse source tree directly will _not_ use the locked dependencies; instead,
they'll pull in the latest version of each package available at install time.
## Example dependency
An example may help. We have a broad dependency on
[`phonenumbers`](https://pypi.org/project/phonenumbers/), as declared in
this snippet from pyproject.toml [as of Synapse 1.57](
https://github.com/matrix-org/synapse/blob/release-v1.57/pyproject.toml#L133
):
```toml
[tool.poetry.dependencies]
# ...
phonenumbers = ">=8.2.0"
```
In our lockfile this is
[pinned]( https://github.com/matrix-org/synapse/blob/dfc7646504cef3e4ff396c36089e1c6f1b1634de/poetry.lock#L679-L685)
to version 8.12.44, even though
[newer versions are available](https://pypi.org/project/phonenumbers/#history).
```toml
[[package]]
name = "phonenumbers"
version = "8.12.44"
description = "Python version of Google's common library for parsing, formatting, storing and validating international phone numbers."
category = "main"
optional = false
python-versions = "*"
```
The lockfile also includes a
[cryptographic checksum](https://github.com/matrix-org/synapse/blob/release-v1.57/poetry.lock#L2178-L2181)
of the sdists and wheels provided for this version of `phonenumbers`.
```toml
[metadata.files]
# ...
phonenumbers = [
{file = "phonenumbers-8.12.44-py2.py3-none-any.whl", hash = "sha256:cc1299cf37b309ecab6214297663ab86cb3d64ae37fd5b88e904fe7983a874a6"},
{file = "phonenumbers-8.12.44.tar.gz", hash = "sha256:26cfd0257d1704fe2f88caff2caabb70d16a877b1e65b6aae51f9fbbe10aa8ce"},
]
```
We can see this pinned version inside the docker image for that release:
```
$ docker pull matrixdotorg/synapse:v1.57.0
...
$ docker run --entrypoint pip matrixdotorg/synapse:v1.57.0 show phonenumbers
Name: phonenumbers
Version: 8.12.44
Summary: Python version of Google's common library for parsing, formatting, storing and validating international phone numbers.
Home-page: https://github.com/daviddrysdale/python-phonenumbers
Author: David Drysdale
Author-email: dmd@lurklurk.org
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires:
Required-by: matrix-synapse
```
Whereas the wheel metadata just contains the broad dependencies:
```
$ cd /tmp
$ wget https://files.pythonhosted.org/packages/ca/5e/d722d572cc5b3092402b783d6b7185901b444427633bd8a6b00ea0dd41b7/matrix_synapse-1.57.0rc1-py3-none-any.whl
...
$ unzip -c matrix_synapse-1.57.0rc1-py3-none-any.whl matrix_synapse-1.57.0rc1.dist-info/METADATA | grep phonenumbers
Requires-Dist: phonenumbers (>=8.2.0)
```
# Tooling recommendation: direnv
[`direnv`](https://direnv.net/) is a tool for activating environments in your
shell inside a given directory. Its support for poetry is unofficial (a
community wiki recipe only), but works solidly in our experience. We thoroughly
recommend it for daily use. To use it:
1. [Install `direnv`](https://direnv.net/docs/installation.html) - it's likely
packaged for your system already.
2. Teach direnv about poetry. The [shell config here](https://github.com/direnv/direnv/wiki/Python#poetry)
needs to be added to `~/.config/direnv/direnvrc` (or more generally `$XDG_CONFIG_HOME/direnv/direnvrc`).
3. Mark the synapse checkout as a poetry project: `echo layout poetry > .envrc`.
4. Convince yourself that you trust this `.envrc` configuration and project.
Then formally confirm this to `direnv` by running `direnv allow`.
Then whenever you navigate to the synapse checkout, you should be able to run
e.g. `mypy` instead of `poetry run mypy`; `python` instead of
`poetry run python`; and your shell commands will automatically run in the
context of poetry's venv, without having to run `poetry shell` beforehand.
# How do I...
## ...reset my venv to the locked environment?
```shell
poetry install --extras all --remove-untracked
```
## ...run a command in the `poetry` virtualenv?
Use `poetry run cmd args` when you need the python virtualenv context.
To avoid typing `poetry run` all the time, you can run `poetry shell`
to start a new shell in the poetry virtualenv context. Within `poetry shell`,
`python`, `pip`, `mypy`, `trial`, etc. are all run inside the project virtualenv
and isolated from the rest o the system.
Roughly speaking, the translation from a traditional virtualenv is:
- `env/bin/activate` -> `poetry shell`, and
- `deactivate` -> close the terminal (Ctrl-D, `exit`, etc.)
See also the direnv recommendation above, which makes `poetry run` and
`poetry shell` unnecessary.
## ...inspect the `poetry` virtualenv?
Some suggestions:
```shell
# Current env only
poetry env info
# All envs: this allows you to have e.g. a poetry managed venv for Python 3.7,
# and another for Python 3.10.
poetry env list --full-path
poetry run pip list
```
Note that `poetry show` describes the abstract *lock file* rather than your
on-disk environment. With that said, `poetry show --tree` can sometimes be
useful.
## ...add a new dependency?
Either:
- manually update `pyproject.toml`; then `poetry lock --no-update`; or else
- `poetry add packagename`. See `poetry add --help`; note the `--dev`,
`--extras` and `--optional` flags in particular.
- **NB**: this specifies the new package with a version given by a "caret bound". This won't get forced to its lowest version in the old deps CI job: see [this TODO](https://github.com/matrix-org/synapse/blob/4e1374373857f2f7a911a31c50476342d9070681/.ci/scripts/test_old_deps.sh#L35-L39).
Include the updated `pyproject.toml` and `poetry.lock` files in your commit.
## ...remove a dependency?
This is not done often and is untested, but
```shell
poetry remove packagename
```
ought to do the trick. Alternatively, manually update `pyproject.toml` and
`poetry lock --no-update`. Include the updated `pyproject.toml` and poetry.lock`
files in your commit.
## ...update the version range for an existing dependency?
Best done by manually editing `pyproject.toml`, then `poetry lock --no-update`.
Include the updated `pyproject.toml` and `poetry.lock` in your commit.
## ...update a dependency in the locked environment?
Use
```shell
poetry update packagename
```
to use the latest version of `packagename` in the locked environment, without
affecting the broad dependencies listed in the wheel.
There doesn't seem to be a way to do this whilst locking a _specific_ version of
`packagename`. We can workaround this (crudely) as follows:
```shell
poetry add packagename==1.2.3
# This should update pyproject.lock.
# Now undo the changes to pyproject.toml. For example
# git restore pyproject.toml
# Get poetry to recompute the content-hash of pyproject.toml without changing
# the locked package versions.
poetry lock --no-update
```
Either way, include the updated `poetry.lock` file in your commit.
## ...export a `requirements.txt` file?
```shell
poetry export --extras all
```
Be wary of bugs in `poetry export` and `pip install -r requirements.txt`.
Note: `poetry export` will be made a plugin in Poetry 1.2. Additional config may
be required.
## ...build a test wheel?
I usually use
```shell
poetry run pip install build && poetry run python -m build
```
because [`build`](https://github.com/pypa/build) is a standardish tool which
doesn't require poetry. (It's what we use in CI too). However, you could try
`poetry build` too.

34
docs/upgrade.md
Unescape View File

@ -19,32 +19,36 @@ this document.
packages](setup/installation.md#prebuilt-packages), you will need to follow the
normal process for upgrading those packages.
- If Synapse was installed using pip then upgrade to the latest
version by running:
```bash
pip install --upgrade matrix-synapse
```
- If Synapse was installed from source, then:
1. Activate the virtualenv before upgrading. For example, if
Synapse is installed in a virtualenv in `~/synapse/env` then
1. Obtain the latest version of the source code. Git users can run
`git pull` to do this.
2. If you're running Synapse in a virtualenv, make sure to activate it before
upgrading. For example, if Synapse is installed in a virtualenv in `~/synapse/env` then
run:
```bash
source ~/synapse/env/bin/activate
pip install --upgrade .
```
Include any relevant extras between square brackets, e.g. `pip install --upgrade ".[postgres,oidc]"`.
2. If Synapse was installed using pip then upgrade to the latest
version by running:
```bash
pip install --upgrade matrix-synapse
```
If Synapse was installed using git then upgrade to the latest
version by running:
3. If you're using `poetry` to manage a Synapse installation, run:
```bash
git pull
pip install --upgrade .
poetry install
```
Include any relevant extras with `--extras`, e.g. `poetry install --extras postgres --extras oidc`.
It's probably easiest to run `poetry install --extras all`.
3. Restart Synapse:
4. Restart Synapse:
```bash
synctl restart

Write Preview
Loading…
Cancel
Save