Comment from Matthew:
This fixes a potential issue where the Prosody process gets blocked on sockets
waiting for them to close. Unlike non-TLS sockets, closing a TLS socket sends
layer 7 data, and this can cause problems for sockets which are in the process
of being cleaned up.
This depends on LuaSec changes which are not yet upstream.
From Martijn's original email:
So first my analysis of luasec. in ssl.c the socket is put into blocking
mode right before calling SSL_shutdown() inside meth_destroy(). My best
guess to why this is is because meth_destroy is linked to the __close
and __gc methods, which can't exactly be called multiple times and
luasec does want to make sure that a tls session is shutdown as clean
as possible.
I can't say I disagree with this reasoning and don't want to change this
behaviour. My solution to this without changing the current behaviour is
to introduce a shutdown() method. I am aware that this overlaps in a
conflicting way with tcp's shutdown method, but it stays close to the
OpenSSL name. This method calls SSL_shutdown() in the current
(non)blocking mode of the underlying socket and returns a boolean
whether or not the shutdown is completed (matching SSL_shutdown()'s 0
or 1 return values), and returns the familiar ssl_ioerror() strings on
error with a false for completion. This error can then be used to
determine if we have wantread/wantwrite to finalize things. Once
meth_shutdown() has been called once a shutdown flag will be set, which
indicates to meth_destroy() that the SSL_shutdown() has been handled
by the application and it shouldn't be needed to set the socket to
blocking mode. I've left the SSL_shutdown() call in the
LSEC_STATE_CONNECTED to prevent TOCTOU if the application reaches a
timeout for the shutdown code, which might allow SSL_shutdown() to
clean up anyway at the last possible moment.
Another thing I've changed to luasec is the call to socket_setblocking()
right before calling close(2) in socket_destroy() in usocket.c.
According to the latest POSIX[0]:
Note that the requirement for close() on a socket to block for up to
the current linger interval is not conditional on the O_NONBLOCK
setting.
Which I read to mean that removing O_NONBLOCK on the socket before close
doesn't impact the behaviour and only causes noise in system call
tracers. I didn't touch the windows bits of this, since I don't do
windows.
For the prosody side of things I've made the TLS shutdown bits resemble
interface:onwritable(), and put it under a combined guard of self._tls
and self.conn.shutdown. The self._tls bit is there to prevent getting
stuck on this condition, and self.conn.shutdown is there to prevent the
code being called by instances where the patched luasec isn't deployed.
The destroy() method can be called from various places and is read by
me as the "we give up" error path. To accommodate for these unexpected
entrypoints I've added a single call to self.conn:shutdown() to prevent
the socket being put into blocking mode. I have no expectations that
there is any other use here. Same as previous, the self.conn.shutdown
check is there to make sure it's not called on unpatched luasec
deployments and self._tls is there to make sure we don't call shutdown()
on tcp sockets.
I wouldn't recommend logging of the conn:shutdown() error inside
close(), since a lot of clients simply close the connection before
SSL_shutdown() is done.
Attempt to fix a bug where connections are somehow closed twice, leading
to bad things happening elsewhere.
With LuaSec, closed connections are generally already too closed to
write anything to anyway since it does not support unidirectional
shutdown.
Prevent Bad Things from happening when the buffer gets full.
This of course opens up the possibility of intentionally killing
connections by sending much stuff, which need to be mitigated with rate
limits elsewhere.
Problem: The string slice operations when a lot of data gets buffered
ends up being expensive and memory-consuming. We have util.dbuffer for
precisely this kind of thing.
I want to keep the behavior of writebuffer being upgraded from nil to a
string to full buffer since the last step involves three table
allocations, where the previous buffer method only used one. Avoiding
those allocations for simple writes like white space keep alive feels
like it would keep memory churn down.
This work was started in 2020
Allows creating listening sockets and accepting client connections
before Prosody starts.
This is unlike normal Prosody dynamic resource management, where ports
may added and removed at any time, and the ports defined by the config.
Weird things happen if these are closed (e.g. due to reload) so here we
prevent closing and ensure sockets are reused when opened again.
Handling signal events the same way as all other events makes sense and
seems safer than the signal handling just jumping around in C and
messing with Lua states.
Should call timers less frequently when many sockets are waiting for
processing. May help under heavy load.
Requested by Ge0rG
Backport of 2bcd84123eba requested by Roi
Likely affected rescheduling but have no reports of this.
After readding a timer, it would have been issued a new id. Rescheduling
would use the previous id, thus not working.
Main difference is that timers are not checked unconditionally before
each poll, only when running out of previous poll results (hidden by
util.poll). This removes a final poll at shutdown that usually delays
the 'not quitting' condition check by one second.
Brings back SCRAM-SHA-*-PLUS from its hiatus brought on by the earlier
channel binding method being undefined for TLS 1.3, and the increasing
deployment of TLS 1.3.
See 1bfd238e05ad and #1542
Requires future version of LuaSec, once support for this key material
export method is merged.
See https://github.com/brunoos/luasec/pull/187
return foo and foo() crops multiple return values to a single one, so
any second return values etc were last, mostly error details.
Introduced in 7e9ebdc75ce4
This is a Linux(?) socket option that delays the accept signal until
there is data available to read. E.g. with HTTP this might mean that a
whole request can be handled without going back trough another turn of
the main loop, and an initial client <stream> can be responded to.
This may have effects on latency and resource use, as the server does
not need to allocate resources until really needed.
Requires a patch to LuaSocket adding this socket option,
https://github.com/lunarmodules/luasocket/pull/378
sysctl tweaks
net.ipv4.tcp_fastopen=3
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 0
net.ipv4.tcp_fastopen_key=$(</proc/sys/kernel/random/uuid)
Disabled by default since it an advanced performance tweak unlikely to
be needed by most servers.
This now requires that the network backend exposes a tls_builder
function, which essentially wraps the former util.sslconfig.new()
function, passing a factory to create the eventual SSL context.
That allows a net.server backend to pick whatever it likes as SSL
context factory, as long as it understands the config table passed by
the SSL config builder. Heck, a backend could even mock and replace the
entire SSL config builder API.
For this, various accessor functions are now provided directly on the
sockets, which reach down into the LuaSec implementation to obtain the
information.
While this may seem of little gain at first, it hides the implementation
detail of the LuaSec+LuaSocket combination that the actual socket and
the TLS layer are separate objects.
The net gain here is that an alternative implementation does not have to
emulate that specific implementation detail and "only" has to expose
LuaSec-compatible data structures on the new functions.
This makes sure that a timer that returns 0 (or less) does not prevent
runtimers() from completing, as well as making sure a timer added with
zero timeout from within a timer does not run until the next tick.
Thanks tmolitor
Partial backports of the following commits from trunk:
6c804b6b2ca2 net.http: Pass server name along for SNI (fixes#1408)
75d2874502c3 net.server_select: SNI support (#409)
9a905888b96c net.server_event: Add SNI support (#409)
adc0672b700e net.server_epoll: Add support for SNI (#409)
d4390c427a66 net.server: Handle server name (SNI) as extra argument
Some lines seem to have gotten the wrong indentation, possibly caused by
Meld which often ignores lines with only whitespace changes and leaves
their previous indentation.
With opportunistic writes enabled, writes can return what :onwritable()
returns, thus :onwritable() should return something sensible at each
spot.
Should prevent whatever caused
> Error writing to connection: (nil)
Tho this was probably harmless
If the connection is closed by the peer, any buffered data is given a
last chance to be sent (see f27b9319e0da). If the connection is Really
closed, no attempt to write will occur, instead epoll will raise the
error flag and :onreadable() will be invoked again, where it will try to
:close() again for the same reason, thus looping until the connection
somehow gets destroyed.
By clearing the _connected flag, the second time it passes :onreadable()
it should go directly to :destroy(), breaking the loop.
Thanks Link Mauve for reporting
ff4e34c448a4 broke the way net.http.server streams downloads from disk
because it made writes from the ondrain callback no longer reset the
want-write flag, causing the download to halt.
Writes from the predrain handler still must not trigger anything but
additions to the buffer, since it is about to do all the socket writing
already.
This is not a pretty way to signal this... but it is the current API
interface:inittls() is a new code path which did not go past the point
in interface:starttls() where it set starttls to false, leading mod_tls
to offer starttls on direct TLS connections
Thanks Martin for discovering.
The intent is to ensure 'ondisconnect' only gets called once, while
giving buffered outgoing data a last chance to be delivered via the
:close() path in case the connection was only shutdown in one direction.
Before 22825cb5dcd8 connection attempts that failed (e.g. connection
refused) would be immediately destroyed. After, it would schedule
another write cycle and then report 'ondisconnect' again when failing.
Thanks Martin for reporting
Instead try to write any remaining buffered data. If the write attempt
also fails with "closed" then there's nothing we can do and the socket
is gone.
This reverts what appears to be a mistakenly included part of c8aa66595072
Thanks jonas’ for noticing
Previously it was unclear whether "client port" was the port that the
client connected to, or from. I hereby declare that the client port is
the source port and the server port is the destination port.
Incoming and outgoing connections can be distinguished by looking at
the_server reference, which only incoming connections have.
Only relevant because a "dirty" connection (with incoming data in
LuaSocket's buffer) does not count as "readable" according to epoll, so
special care needs to be taken to keep on processing it.
Allows sneaking in things in the write buffer just before it's sent to
the network stack. For example ack requests, compression flushes or
other things that make sense to send after stanzas or other things.
This ensures any additional trailing data sent is included in the same
write, and possibly the same TCP packet. Other methods used such as
timers or nextTick might not have the same effect as it depends on
scheduling.
Should prevent further opportunistic write attempts after the kernel
buffers are full and stops accepting writes.
When combined with `keep_buffers = false` it should stop it from
repeatedly recreating the buffer table and concatenating it back into a
string when there's a lot to write.
Kim Alvefur