mirror of https://github.com/watcha-fr/synapse
commit
484a0ebdfc
@ -0,0 +1 @@ |
||||
Respond with M_NOT_FOUND when profiles are not found locally or over federation. Fixes #3585 |
@ -0,0 +1 @@ |
||||
Refactor HTTP replication endpoints to reduce code duplication |
@ -0,0 +1 @@ |
||||
Add ability to limit number of monthly active users on the server |
@ -0,0 +1 @@ |
||||
Refactor location of docker build script. |
@ -0,0 +1 @@ |
||||
Tests now correctly execute on Python 3. |
@ -0,0 +1 @@ |
||||
Basic support for room versioning |
@ -0,0 +1 @@ |
||||
Fix occasional glitches in the synapse_event_persisted_position metric |
@ -0,0 +1 @@ |
||||
Ability to whitelist specific threepids against monthly active user limiting |
@ -0,0 +1 @@ |
||||
Add some metrics for the appservice and federation event sending loops |
@ -0,0 +1,124 @@ |
||||
# Synapse Docker |
||||
|
||||
This Docker image will run Synapse as a single process. It does not provide a database |
||||
server or a TURN server, you should run these separately. |
||||
|
||||
## Run |
||||
|
||||
We do not currently offer a `latest` image, as this has somewhat undefined semantics. |
||||
We instead release only tagged versions so upgrading between releases is entirely |
||||
within your control. |
||||
|
||||
### Using docker-compose (easier) |
||||
|
||||
This image is designed to run either with an automatically generated configuration |
||||
file or with a custom configuration that requires manual editing. |
||||
|
||||
An easy way to make use of this image is via docker-compose. See the |
||||
[contrib/docker](../contrib/docker) |
||||
section of the synapse project for examples. |
||||
|
||||
### Without Compose (harder) |
||||
|
||||
If you do not wish to use Compose, you may still run this image using plain |
||||
Docker commands. Note that the following is just a guideline and you may need |
||||
to add parameters to the docker run command to account for the network situation |
||||
with your postgres database. |
||||
|
||||
``` |
||||
docker run \ |
||||
-d \ |
||||
--name synapse \ |
||||
-v ${DATA_PATH}:/data \ |
||||
-e SYNAPSE_SERVER_NAME=my.matrix.host \ |
||||
-e SYNAPSE_REPORT_STATS=yes \ |
||||
docker.io/matrixdotorg/synapse:latest |
||||
``` |
||||
|
||||
## Volumes |
||||
|
||||
The image expects a single volume, located at ``/data``, that will hold: |
||||
|
||||
* temporary files during uploads; |
||||
* uploaded media and thumbnails; |
||||
* the SQLite database if you do not configure postgres; |
||||
* the appservices configuration. |
||||
|
||||
You are free to use separate volumes depending on storage endpoints at your |
||||
disposal. For instance, ``/data/media`` coud be stored on a large but low |
||||
performance hdd storage while other files could be stored on high performance |
||||
endpoints. |
||||
|
||||
In order to setup an application service, simply create an ``appservices`` |
||||
directory in the data volume and write the application service Yaml |
||||
configuration file there. Multiple application services are supported. |
||||
|
||||
## Environment |
||||
|
||||
Unless you specify a custom path for the configuration file, a very generic |
||||
file will be generated, based on the following environment settings. |
||||
These are a good starting point for setting up your own deployment. |
||||
|
||||
Global settings: |
||||
|
||||
* ``UID``, the user id Synapse will run as [default 991] |
||||
* ``GID``, the group id Synapse will run as [default 991] |
||||
* ``SYNAPSE_CONFIG_PATH``, path to a custom config file |
||||
|
||||
If ``SYNAPSE_CONFIG_PATH`` is set, you should generate a configuration file |
||||
then customize it manually. No other environment variable is required. |
||||
|
||||
Otherwise, a dynamic configuration file will be used. The following environment |
||||
variables are available for configuration: |
||||
|
||||
* ``SYNAPSE_SERVER_NAME`` (mandatory), the current server public hostname. |
||||
* ``SYNAPSE_REPORT_STATS``, (mandatory, ``yes`` or ``no``), enable anonymous |
||||
statistics reporting back to the Matrix project which helps us to get funding. |
||||
* ``SYNAPSE_NO_TLS``, set this variable to disable TLS in Synapse (use this if |
||||
you run your own TLS-capable reverse proxy). |
||||
* ``SYNAPSE_ENABLE_REGISTRATION``, set this variable to enable registration on |
||||
the Synapse instance. |
||||
* ``SYNAPSE_ALLOW_GUEST``, set this variable to allow guest joining this server. |
||||
* ``SYNAPSE_EVENT_CACHE_SIZE``, the event cache size [default `10K`]. |
||||
* ``SYNAPSE_CACHE_FACTOR``, the cache factor [default `0.5`]. |
||||
* ``SYNAPSE_RECAPTCHA_PUBLIC_KEY``, set this variable to the recaptcha public |
||||
key in order to enable recaptcha upon registration. |
||||
* ``SYNAPSE_RECAPTCHA_PRIVATE_KEY``, set this variable to the recaptcha private |
||||
key in order to enable recaptcha upon registration. |
||||
* ``SYNAPSE_TURN_URIS``, set this variable to the coma-separated list of TURN |
||||
uris to enable TURN for this homeserver. |
||||
* ``SYNAPSE_TURN_SECRET``, set this to the TURN shared secret if required. |
||||
|
||||
Shared secrets, that will be initialized to random values if not set: |
||||
|
||||
* ``SYNAPSE_REGISTRATION_SHARED_SECRET``, secret for registrering users if |
||||
registration is disable. |
||||
* ``SYNAPSE_MACAROON_SECRET_KEY`` secret for signing access tokens |
||||
to the server. |
||||
|
||||
Database specific values (will use SQLite if not set): |
||||
|
||||
* `POSTGRES_DB` - The database name for the synapse postgres database. [default: `synapse`] |
||||
* `POSTGRES_HOST` - The host of the postgres database if you wish to use postgresql instead of sqlite3. [default: `db` which is useful when using a container on the same docker network in a compose file where the postgres service is called `db`] |
||||
* `POSTGRES_PASSWORD` - The password for the synapse postgres database. **If this is set then postgres will be used instead of sqlite3.** [default: none] **NOTE**: You are highly encouraged to use postgresql! Please use the compose file to make it easier to deploy. |
||||
* `POSTGRES_USER` - The user for the synapse postgres database. [default: `matrix`] |
||||
|
||||
Mail server specific values (will not send emails if not set): |
||||
|
||||
* ``SYNAPSE_SMTP_HOST``, hostname to the mail server. |
||||
* ``SYNAPSE_SMTP_PORT``, TCP port for accessing the mail server [default ``25``]. |
||||
* ``SYNAPSE_SMTP_USER``, username for authenticating against the mail server if any. |
||||
* ``SYNAPSE_SMTP_PASSWORD``, password for authenticating against the mail server if any. |
||||
|
||||
## Build |
||||
|
||||
Build the docker image with the `docker build` command from the root of the synapse repository. |
||||
|
||||
``` |
||||
docker build -t docker.io/matrixdotorg/synapse . -f docker/Dockerfile |
||||
``` |
||||
|
||||
The `-t` option sets the image tag. Official images are tagged `matrixdotorg/synapse:<version>` where `<version>` is the same as the release tag in the synapse git repository. |
||||
|
||||
You may have a local Python wheel cache available, in which case copy the relevant |
||||
packages in the ``cache/`` directory at the root of the project. |
@ -0,0 +1,201 @@ |
||||
# -*- coding: utf-8 -*- |
||||
# Copyright 2018 New Vector |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
import logging |
||||
|
||||
from twisted.internet import defer |
||||
|
||||
from synapse.util.caches.descriptors import cached |
||||
|
||||
from ._base import SQLBaseStore |
||||
|
||||
logger = logging.getLogger(__name__) |
||||
|
||||
# Number of msec of granularity to store the monthly_active_user timestamp |
||||
# This means it is not necessary to update the table on every request |
||||
LAST_SEEN_GRANULARITY = 60 * 60 * 1000 |
||||
|
||||
|
||||
class MonthlyActiveUsersStore(SQLBaseStore): |
||||
def __init__(self, dbconn, hs): |
||||
super(MonthlyActiveUsersStore, self).__init__(None, hs) |
||||
self._clock = hs.get_clock() |
||||
self.hs = hs |
||||
self.reserved_users = () |
||||
|
||||
@defer.inlineCallbacks |
||||
def initialise_reserved_users(self, threepids): |
||||
# TODO Why can't I do this in init? |
||||
store = self.hs.get_datastore() |
||||
reserved_user_list = [] |
||||
|
||||
# Do not add more reserved users than the total allowable number |
||||
for tp in threepids[:self.hs.config.max_mau_value]: |
||||
user_id = yield store.get_user_id_by_threepid( |
||||
tp["medium"], tp["address"] |
||||
) |
||||
if user_id: |
||||
self.upsert_monthly_active_user(user_id) |
||||
reserved_user_list.append(user_id) |
||||
else: |
||||
logger.warning( |
||||
"mau limit reserved threepid %s not found in db" % tp |
||||
) |
||||
self.reserved_users = tuple(reserved_user_list) |
||||
|
||||
@defer.inlineCallbacks |
||||
def reap_monthly_active_users(self): |
||||
""" |
||||
Cleans out monthly active user table to ensure that no stale |
||||
entries exist. |
||||
|
||||
Returns: |
||||
Deferred[] |
||||
""" |
||||
def _reap_users(txn): |
||||
|
||||
thirty_days_ago = ( |
||||
int(self._clock.time_msec()) - (1000 * 60 * 60 * 24 * 30) |
||||
) |
||||
# Purge stale users |
||||
|
||||
# questionmarks is a hack to overcome sqlite not supporting |
||||
# tuples in 'WHERE IN %s' |
||||
questionmarks = '?' * len(self.reserved_users) |
||||
query_args = [thirty_days_ago] |
||||
query_args.extend(self.reserved_users) |
||||
|
||||
sql = """ |
||||
DELETE FROM monthly_active_users |
||||
WHERE timestamp < ? |
||||
AND user_id NOT IN ({}) |
||||
""".format(','.join(questionmarks)) |
||||
|
||||
txn.execute(sql, query_args) |
||||
|
||||
# If MAU user count still exceeds the MAU threshold, then delete on |
||||
# a least recently active basis. |
||||
# Note it is not possible to write this query using OFFSET due to |
||||
# incompatibilities in how sqlite and postgres support the feature. |
||||
# sqlite requires 'LIMIT -1 OFFSET ?', the LIMIT must be present |
||||
# While Postgres does not require 'LIMIT', but also does not support |
||||
# negative LIMIT values. So there is no way to write it that both can |
||||
# support |
||||
query_args = [self.hs.config.max_mau_value] |
||||
query_args.extend(self.reserved_users) |
||||
sql = """ |
||||
DELETE FROM monthly_active_users |
||||
WHERE user_id NOT IN ( |
||||
SELECT user_id FROM monthly_active_users |
||||
ORDER BY timestamp DESC |
||||
LIMIT ? |
||||
) |
||||
AND user_id NOT IN ({}) |
||||
""".format(','.join(questionmarks)) |
||||
txn.execute(sql, query_args) |
||||
|
||||
yield self.runInteraction("reap_monthly_active_users", _reap_users) |
||||
# It seems poor to invalidate the whole cache, Postgres supports |
||||
# 'Returning' which would allow me to invalidate only the |
||||
# specific users, but sqlite has no way to do this and instead |
||||
# I would need to SELECT and the DELETE which without locking |
||||
# is racy. |
||||
# Have resolved to invalidate the whole cache for now and do |
||||
# something about it if and when the perf becomes significant |
||||
self._user_last_seen_monthly_active.invalidate_all() |
||||
self.get_monthly_active_count.invalidate_all() |
||||
|
||||
@cached(num_args=0) |
||||
def get_monthly_active_count(self): |
||||
"""Generates current count of monthly active users |
||||
|
||||
Returns: |
||||
Defered[int]: Number of current monthly active users |
||||
""" |
||||
|
||||
def _count_users(txn): |
||||
sql = "SELECT COALESCE(count(*), 0) FROM monthly_active_users" |
||||
|
||||
txn.execute(sql) |
||||
count, = txn.fetchone() |
||||
return count |
||||
return self.runInteraction("count_users", _count_users) |
||||
|
||||
def upsert_monthly_active_user(self, user_id): |
||||
""" |
||||
Updates or inserts monthly active user member |
||||
Arguments: |
||||
user_id (str): user to add/update |
||||
Deferred[bool]: True if a new entry was created, False if an |
||||
existing one was updated. |
||||
""" |
||||
is_insert = self._simple_upsert( |
||||
desc="upsert_monthly_active_user", |
||||
table="monthly_active_users", |
||||
keyvalues={ |
||||
"user_id": user_id, |
||||
}, |
||||
values={ |
||||
"timestamp": int(self._clock.time_msec()), |
||||
}, |
||||
lock=False, |
||||
) |
||||
if is_insert: |
||||
self._user_last_seen_monthly_active.invalidate((user_id,)) |
||||
self.get_monthly_active_count.invalidate(()) |
||||
|
||||
@cached(num_args=1) |
||||
def _user_last_seen_monthly_active(self, user_id): |
||||
""" |
||||
Checks if a given user is part of the monthly active user group |
||||
Arguments: |
||||
user_id (str): user to add/update |
||||
Return: |
||||
Deferred[int] : timestamp since last seen, None if never seen |
||||
|
||||
""" |
||||
|
||||
return(self._simple_select_one_onecol( |
||||
table="monthly_active_users", |
||||
keyvalues={ |
||||
"user_id": user_id, |
||||
}, |
||||
retcol="timestamp", |
||||
allow_none=True, |
||||
desc="_user_last_seen_monthly_active", |
||||
)) |
||||
|
||||
@defer.inlineCallbacks |
||||
def populate_monthly_active_users(self, user_id): |
||||
"""Checks on the state of monthly active user limits and optionally |
||||
add the user to the monthly active tables |
||||
|
||||
Args: |
||||
user_id(str): the user_id to query |
||||
""" |
||||
if self.hs.config.limit_usage_by_mau: |
||||
last_seen_timestamp = yield self._user_last_seen_monthly_active(user_id) |
||||
now = self.hs.get_clock().time_msec() |
||||
|
||||
# We want to reduce to the total number of db writes, and are happy |
||||
# to trade accuracy of timestamp in order to lighten load. This means |
||||
# We always insert new users (where MAU threshold has not been reached), |
||||
# but only update if we have not previously seen the user for |
||||
# LAST_SEEN_GRANULARITY ms |
||||
if last_seen_timestamp is None: |
||||
count = yield self.get_monthly_active_count() |
||||
if count < self.hs.config.max_mau_value: |
||||
yield self.upsert_monthly_active_user(user_id) |
||||
elif now - last_seen_timestamp > LAST_SEEN_GRANULARITY: |
||||
yield self.upsert_monthly_active_user(user_id) |
@ -0,0 +1,27 @@ |
||||
/* Copyright 2018 New Vector Ltd |
||||
* |
||||
* Licensed under the Apache License, Version 2.0 (the "License"); |
||||
* you may not use this file except in compliance with the License. |
||||
* You may obtain a copy of the License at |
||||
* |
||||
* http://www.apache.org/licenses/LICENSE-2.0 |
||||
* |
||||
* Unless required by applicable law or agreed to in writing, software |
||||
* distributed under the License is distributed on an "AS IS" BASIS, |
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
* See the License for the specific language governing permissions and |
||||
* limitations under the License. |
||||
*/ |
||||
|
||||
-- a table of monthly active users, for use where blocking based on mau limits |
||||
CREATE TABLE monthly_active_users ( |
||||
user_id TEXT NOT NULL, |
||||
-- Last time we saw the user. Not guaranteed to be accurate due to rate limiting |
||||
-- on updates, Granularity of updates governed by |
||||
-- synapse.storage.monthly_active_users.LAST_SEEN_GRANULARITY |
||||
-- Measured in ms since epoch. |
||||
timestamp BIGINT NOT NULL |
||||
); |
||||
|
||||
CREATE UNIQUE INDEX monthly_active_users_users ON monthly_active_users(user_id); |
||||
CREATE INDEX monthly_active_users_time_stamp ON monthly_active_users(timestamp); |
@ -1,65 +0,0 @@ |
||||
# -*- coding: utf-8 -*- |
||||
# Copyright 2018 New Vector Ltd |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
from twisted.internet import defer |
||||
|
||||
import tests.utils |
||||
|
||||
|
||||
class InitTestCase(tests.unittest.TestCase): |
||||
def __init__(self, *args, **kwargs): |
||||
super(InitTestCase, self).__init__(*args, **kwargs) |
||||
self.store = None # type: synapse.storage.DataStore |
||||
|
||||
@defer.inlineCallbacks |
||||
def setUp(self): |
||||
hs = yield tests.utils.setup_test_homeserver() |
||||
|
||||
hs.config.max_mau_value = 50 |
||||
hs.config.limit_usage_by_mau = True |
||||
self.store = hs.get_datastore() |
||||
self.clock = hs.get_clock() |
||||
|
||||
@defer.inlineCallbacks |
||||
def test_count_monthly_users(self): |
||||
count = yield self.store.count_monthly_users() |
||||
self.assertEqual(0, count) |
||||
|
||||
yield self._insert_user_ips("@user:server1") |
||||
yield self._insert_user_ips("@user:server2") |
||||
|
||||
count = yield self.store.count_monthly_users() |
||||
self.assertEqual(2, count) |
||||
|
||||
@defer.inlineCallbacks |
||||
def _insert_user_ips(self, user): |
||||
""" |
||||
Helper function to populate user_ips without using batch insertion infra |
||||
args: |
||||
user (str): specify username i.e. @user:server.com |
||||
""" |
||||
yield self.store._simple_upsert( |
||||
table="user_ips", |
||||
keyvalues={ |
||||
"user_id": user, |
||||
"access_token": "access_token", |
||||
"ip": "ip", |
||||
"user_agent": "user_agent", |
||||
"device_id": "device_id", |
||||
}, |
||||
values={ |
||||
"last_seen": self.clock.time_msec(), |
||||
} |
||||
) |
@ -0,0 +1,123 @@ |
||||
# -*- coding: utf-8 -*- |
||||
# Copyright 2018 New Vector |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
from twisted.internet import defer |
||||
|
||||
import tests.unittest |
||||
import tests.utils |
||||
from tests.utils import setup_test_homeserver |
||||
|
||||
FORTY_DAYS = 40 * 24 * 60 * 60 |
||||
|
||||
|
||||
class MonthlyActiveUsersTestCase(tests.unittest.TestCase): |
||||
def __init__(self, *args, **kwargs): |
||||
super(MonthlyActiveUsersTestCase, self).__init__(*args, **kwargs) |
||||
|
||||
@defer.inlineCallbacks |
||||
def setUp(self): |
||||
self.hs = yield setup_test_homeserver() |
||||
self.store = self.hs.get_datastore() |
||||
|
||||
@defer.inlineCallbacks |
||||
def test_initialise_reserved_users(self): |
||||
|
||||
user1 = "@user1:server" |
||||
user1_email = "user1@matrix.org" |
||||
user2 = "@user2:server" |
||||
user2_email = "user2@matrix.org" |
||||
threepids = [ |
||||
{'medium': 'email', 'address': user1_email}, |
||||
{'medium': 'email', 'address': user2_email} |
||||
] |
||||
user_num = len(threepids) |
||||
|
||||
yield self.store.register( |
||||
user_id=user1, |
||||
token="123", |
||||
password_hash=None) |
||||
|
||||
yield self.store.register( |
||||
user_id=user2, |
||||
token="456", |
||||
password_hash=None) |
||||
|
||||
now = int(self.hs.get_clock().time_msec()) |
||||
yield self.store.user_add_threepid(user1, "email", user1_email, now, now) |
||||
yield self.store.user_add_threepid(user2, "email", user2_email, now, now) |
||||
yield self.store.initialise_reserved_users(threepids) |
||||
|
||||
active_count = yield self.store.get_monthly_active_count() |
||||
|
||||
# Test total counts |
||||
self.assertEquals(active_count, user_num) |
||||
|
||||
# Test user is marked as active |
||||
|
||||
timestamp = yield self.store._user_last_seen_monthly_active(user1) |
||||
self.assertTrue(timestamp) |
||||
timestamp = yield self.store._user_last_seen_monthly_active(user2) |
||||
self.assertTrue(timestamp) |
||||
|
||||
# Test that users are never removed from the db. |
||||
self.hs.config.max_mau_value = 0 |
||||
|
||||
self.hs.get_clock().advance_time(FORTY_DAYS) |
||||
|
||||
yield self.store.reap_monthly_active_users() |
||||
|
||||
active_count = yield self.store.get_monthly_active_count() |
||||
self.assertEquals(active_count, user_num) |
||||
|
||||
@defer.inlineCallbacks |
||||
def test_can_insert_and_count_mau(self): |
||||
count = yield self.store.get_monthly_active_count() |
||||
self.assertEqual(0, count) |
||||
|
||||
yield self.store.upsert_monthly_active_user("@user:server") |
||||
count = yield self.store.get_monthly_active_count() |
||||
|
||||
self.assertEqual(1, count) |
||||
|
||||
@defer.inlineCallbacks |
||||
def test__user_last_seen_monthly_active(self): |
||||
user_id1 = "@user1:server" |
||||
user_id2 = "@user2:server" |
||||
user_id3 = "@user3:server" |
||||
result = yield self.store._user_last_seen_monthly_active(user_id1) |
||||
self.assertFalse(result == 0) |
||||
yield self.store.upsert_monthly_active_user(user_id1) |
||||
yield self.store.upsert_monthly_active_user(user_id2) |
||||
result = yield self.store._user_last_seen_monthly_active(user_id1) |
||||
self.assertTrue(result > 0) |
||||
result = yield self.store._user_last_seen_monthly_active(user_id3) |
||||
self.assertFalse(result == 0) |
||||
|
||||
@defer.inlineCallbacks |
||||
def test_reap_monthly_active_users(self): |
||||
self.hs.config.max_mau_value = 5 |
||||
initial_users = 10 |
||||
for i in range(initial_users): |
||||
yield self.store.upsert_monthly_active_user("@user%d:server" % i) |
||||
count = yield self.store.get_monthly_active_count() |
||||
self.assertTrue(count, initial_users) |
||||
yield self.store.reap_monthly_active_users() |
||||
count = yield self.store.get_monthly_active_count() |
||||
self.assertEquals(count, initial_users - self.hs.config.max_mau_value) |
||||
|
||||
self.hs.get_clock().advance_time(FORTY_DAYS) |
||||
yield self.store.reap_monthly_active_users() |
||||
count = yield self.store.get_monthly_active_count() |
||||
self.assertEquals(count, 0) |
Loading…
Reference in new issue