72e0ff88ba
mod_http_file_share: Fix measuring how long periodic task take
4 years ago
9af7bb723b
mod_http_file_share: Measure current total usage
...
In order to allow monitoring. Especially as there's not much in the way
of hard numbers on how much space gets used.
4 years ago
5392ef1dd6
mod_http_file_share: Fix traceback in global quota debug logging (thanks Martin)
...
Error in util.human.units.format because of B(nil) when the global quota
is unset.
4 years ago
ae7f6c34f0
mod_http_file_share: Add optional global quota on total storage usage
...
Before, maximum storage usage (assuming all users upload as much as they
could) would depend on the quota, retention period and number of users.
Since number of users can vary, this makes it hard to know how much
storage will be needed.
Adding a limit to the total overall storage use solves this, making it
simple to set it to some number based on what storage is actually
available.
Summary job run less often than the prune job since it touches the
entire archive; and started before the prune job since it's needed
before the first upload.
4 years ago
1904899421
mod_http_file_share: Update comment about x-frame-options
...
X-Frame-Options was replaced by the Content-Security-Policy
'frame-ancestors' directive, but Internet Explorer does not support that
part of CSP.
Since it's just one line it doesn't hurt to keep until some future
spring cleaning event :)
5 years ago
2f7da2c6ea
mod_http_file_share: Build list of measuring buckets for configured size limit
...
Creates buckets up to the configured size limit or 1TB, whichever is
smaller, e.g. {1K, 4K, 16K, ... 4M, 16M}
5 years ago
a854ec1e4c
mod_http_file_share: Handle out of bounds Range request
...
Turns out you can seek past the end of the file without getting an
error.
Also rejects empty range instead of sending the whole file.
5 years ago
f5757cae7f
mod_http_file_share: Support download resumption via Range requests
...
Only a starting point is supported due to the way response:send_file()
sends everything it gets from the provided file handle but does not have
any way to specify how much to read.
This matches what Conversations appears to be doing.
5 years ago
ecd6b31eea
mod_http_file_share: Include expiry time of the upload itself in token
...
Lets an external upload service know this so it can do expiry itself.
Could possibly have been calculated based on the token expiry or
issuance time, explicit > implicit.
5 years ago
e590c9c92e
mod_http_file_share: Include time of issuance in auth token
...
In case an external upload service wants to have the original creation
time, or calculate the token expiry itself.
5 years ago
bb06c42b07
mod_http_file_share: Group related properties for readability
5 years ago
f7131f81f2
mod_http_file_share: Log error opening file for writing
...
util.error.coerce() doesn't work well with iolib
5 years ago
de641f6e24
mod_http_file_share: Fix logging of error opening file
...
It's annoying that Lua interpolates the filename into the error message.
5 years ago
d8613bb7b9
mod_http_file_share: Include storage in prune time measurement
...
Maybe the original idea was that you would measure storage separately?
5 years ago
29a56e7747
mod_http_file_share: Add internal command to check files consistency
...
Background: Found a few files in my store that did not match the size
recorded in the slot, so I needed a way to check which which those were.
As it was a bit too much to type into the shell I added it here instead.
5 years ago
1be4a5c297
mod_http_file_share: Fix reporting of missing files
...
This just gave an unhelpful 500 error.
It would be nice to have some wrapper code that could untangle the
embedded filename in the io libs errors.
5 years ago
9631d45fcc
mod_http_file_share: Report number of items in caches to statsmanager
...
This is neat, O(1) reporting, why don't we do this everywhere?
Gives you an idea of how much stuff is in the caches, which may help
inform decisions on whether the size is appropriate.
5 years ago
dfaf43cc3f
mod_http_file_share: Skip removal of nothing
...
In case none of the expired files could be deleted then it's a waste of
an API call to try to remove any of the metadata at all.
5 years ago
a2648b9d64
mod_http_file_share: Remove correct entries when not all expired files were deleted
...
If any of the expired files could not be deleted then we should not
forget about that, we should complain loudly and try again.
The code got this backwards and would have removed only the entries
referring to still existing files.
Test procedure:
1. Upload a file
2. chown root:root http_file_share/
3. In uploads.list, decrease 'when' enough to ensure expiry
4. Reload mod_http_file_share
5. Should see an error in the logs about failure to delete the file
6. Should see that the metadata in uploads.list is still there
7. chown http_file_share/ back to the previous owner
8. Reload mod_http_file_share
9. Should see logs about successful removal of expired file
10. Should see that the metadata in uploads.list is gone
11. Should see that the file was deleted
5 years ago
0a5ac437b9
mod_http_file_share: Fix traceback on missing file-type
...
attempt to index a nil value (local 'filetype') casued by the :gsub call
5 years ago
b01915e81c
mod_http_file_share: Allow credentials via CORS (needed for auth token)
5 years ago
900fe6a1cb
mod_http_file_share: Return a message from the base URL
5 years ago
3151984042
mod_http_file_share: Prevent attempt to upload again after completion
5 years ago
9501c7b976
mod_http_file_share: Support sending 100 Continue
...
E.g. curl will ask for this when sending large uploads. Removes a delay
while it waits for an error or go-agead.
5 years ago
cdbb77ccbf
mod_http_file_share: Collect cache hit/miss statistics for downloads
5 years ago
e45dee37bd
mod_http_file_share: Measure how long it takes to prune expired files
5 years ago
da5cd9c963
mod_http_file_share: Collect statistics of files uploaded
5 years ago
5d9802727b
mod_http_file_share: Reorder arguments
...
'filetype' is optional, so having it last seems sensible.
'slot' is pretty important, so moving it earlier seems sensible.
5 years ago
f5baa83714
mod_http_file_share: Update cached value while it is reasonably fresh
...
This should ensure that cache entries until the oldest file that counted
to the last 24h becomes older than 24h.
5 years ago
864708efb7
mod_http_file_share: Cache quotas to avoid hitting storage
5 years ago
855542e2ca
mod_http_file_share: Split out some variables for later reuse
5 years ago
4d3ffbf0f7
mod_http_file_share: Add support for daily upload quotas.
...
Daily instead of total quotas, should be more efficient to calculate.
Still O(n), but a smaller n. Less affected by total retention period.
5 years ago
b1b3f09a5c
mod_http_file_share: Add missing semicolon
...
Last comma or semicolon isn't required but makes the diffs nicer once
you add another item after it.
5 years ago
1998e74fd4
mod_http_file_share: Clarify log message
...
No expired ... what? Could be inferred from the module logging it, but
better to be explicit.
5 years ago
88ce2f1c94
mod_http_file_share: Indicate missing token via WWW-Authenticate header
5 years ago
610278afa0
mod_http_file_share: Clarify message about missing Authorization header
5 years ago
0289a561e6
mod_http_file_share: Allow started uploads to complete after token expired
...
Otherwise uploads taking longer than 5 minutes would be rejected on
completion, and that's probably annoying.
Thanks jonas’
5 years ago
fab6ebb1e6
mod_http_file_share: Serve configurable set of safe mime types inline (thanks jonas’)
...
Otherwise people complain about browser 'Save as' dialog.
5 years ago
22e4d92ce5
mod_http_file_share: Cache file metadata
...
For faster access by avoiding archive ID.
No benchmarks were harmed in the making of this commit.
... no benchmarks were performed at all.
5 years ago
318d357557
mod_http_file_share: Extract all file properties into variables earlier
...
A step towards adding caching, which will unpack into the same
variables.
5 years ago
217450a727
mod_http_file_share: Fix the obligatory misplaced closing bracket (thanks scansion)
5 years ago
2cff8727c7
mod_http_file_share: Insert pauses to avoid blocknig for long periods
...
Similar to the mod_mam cleanup job
5 years ago
395de7cde5
mod_http_file_share: Add support for removing old files (default 2 weeks)
5 years ago
da882b75ab
mod_http_file_share: More security headers
5 years ago
6ce4972bf4
mod_http_file_share: Strip authorization type prefix a bit earlier
5 years ago
3abab1efc3
mod_http_file_share: Factor out function for generating full filename
5 years ago
17a85e6d3b
mod_http_file_share: Use '.bin' file extension
...
Distinct from '.dat' used by datamanager / internal stortage for Lua
object storage so that they can't easily be loaded by accident that way.
5 years ago
e4a97b8e7a
mod_http_file_share: Validate file size early in HTTP PUT request
5 years ago
93494930ea
mod_http_file_share: Move Authorization type string
...
It belongs with the header more than the token itself
5 years ago
943dc18486
mod_http_file_share: Add some logging
5 years ago
Kim Alvefur