132 Commits (ea2f97e9ed8317dc9a62e45d9a807dca7e3700a5)

Author	SHA1	Message	Date
Jonas Schäfer	b9f73bc724	util.datamanager: fix duplicated word in log message	1 year ago
Kim Alvefur	c9e223e4f7	spelling: non-existing mistakes (thanks timeless)	4 years ago
Kim Alvefur	43351d2b54	Spelling: Fix various spelling mistakes (thanks timeless) ... Words, sometimes I wonder how they even work Maybe I missed something.	4 years ago
Kim Alvefur	934bd2a65e	util.datamanager: Fix missing pack format when reading first index entry ... Thanks MattJ	2 years ago
Kim Alvefur	b97c4ffc16	util.datamanager: Load first item into index earlier ... Should get rid of fseek() call	2 years ago
Kim Alvefur	94f9295042	util.datamanager: Clean up list index files on purge (i.e. user deletion)	2 years ago
Kim Alvefur	8cf6d8d959	util.datamanager: Always reset index after list shift ... Shifting the index does not work reliably yet, better to rebuild it from scratch. Since there is minimal parsing involved in that, it should be more efficient anyway.	2 years ago
Kim Alvefur	e6d0a3e639	util.datamanager: Add way to close indexed list store	2 years ago
Kim Alvefur	cd5b2664be	util.datamanager: Close file handle when done using it ... It gets closed eventually but at high load they could potentially lead to reaching FD limits faster.	2 years ago
Kim Alvefur	1312963f73	util.datamanager: Disable blockwise removal ... In desperate need of tests	2 years ago
Kim Alvefur	8699c98ad8	util.datamanager: Disable block alignment ... Until we have more test coverage. Somehow the index becomes incorrect after inserting padding, unclear why.	2 years ago
Kim Alvefur	33986e97b7	util.datamanager: Pad list writes to avoid crossing block boundaries ... By padding items so that they do not cross block boundaries, it becomes eaiser to delete whole blocks with fallocate() without cutting items in half, improving efficiency of such operations. Since list stores are used for message archives, where the most common deletion operation would be of the oldest entires, at the top of the file. With this, all blocks that contain items to be removed could be deleted without needing to read, delete and write out the whole file.	3 years ago
Kim Alvefur	3346561d43	util.datamanager: Efficiently remove whole blocks to shift lists ... Using the new pposix.remove_blocks() it should be very performant to delete whole sections of a file, given a supporting file system.	3 years ago
Kim Alvefur	50ae2083ca	util.datamanager: Add way to efficiently remove first items in a list ... Copying data without parsing it should be more performant than parsing it serializing back.	3 years ago
Kim Alvefur	deecaafbfb	util.datamanager: Fix indexing first item if not at the very start ... If the first item does not start at position 0 then the index function produces a phantom first entry covering position zero until where the real first item starts. When using the index, this would make it either appear as the first item was missing or cause an off-by-one issue with remaining items.	3 years ago
Kim Alvefur	5ee4b4e8a6	util.datamanager: Reduce log level of left over debug messages to debug (thanks Trung) ... These were mostly 'warn' to make them stand out from the debug noise	3 years ago
Kim Alvefur	96bc5fd439	util.datamanager: Fix missing separator in log line	3 years ago
Kim Alvefur	2d69c8c4ea	util.datamanager: Halve size of list index ... Instead of storing (start, length) tuples, store the offset to the end of items and derive length using the previous entry.	5 years ago
Kim Alvefur	74ea750bc4	util.datamanager: Add O(1) list indexing with on-disk index ... Index file contains offsets and lengths of each item() which allows seeking directly to each item and reading it without parsing the entire file. Also allows tricks like binary search, assuming items have some defined order. We take advantage of the 1-based indexing in tables to store a magic header in the 0 position, so that table index 1 ends up at file index 1.	5 years ago
Kim Alvefur	43531740f9	util: Prefix module imports with prosody namespace	3 years ago
Kim Alvefur	d8fda82e40	util.datamanager: Fix iterating over "users" (thanks marc0s) ... The 'store' path componend used to be unescaped until 756a2a00e7e7 added escaping to address issues with characters like '/' used in PEP, but with a special case for '_' which was already in common use in 'store' path components. Missed adding this escaping here.	6 years ago
Kim Alvefur	ac2d84a47b	util.datamanager: Support iterating over any file extension ... The 'typ' argument to all other functions is the actual file extension, but not here for some reason. May need this for iterating over the .bin files created by mod_http_file_share in the future.	5 years ago
Kim Alvefur	2f9a03eba0	util.datamanager: Fix not passing variables as format string to logger (thanks semgrep) ... Passing error messages as format string could lead to errors, although not a problem anymore since util.format sanitizes this.	5 years ago
Kim Alvefur	0bbfb60aae	util.datamanager: Ignore unused 'errno' variable [luacheck]	6 years ago
Kim Alvefur	f65c017ee1	Fix various spelling mistakes [codespell]	7 years ago
Kim Alvefur	33be2cd463	util.datamanager: Use already known index instead of measuring length each iteration	9 years ago
Kim Alvefur	0f578fb9f3	util.datamanager: Log where the error came from	9 years ago
Kim Alvefur	5c4a8d13e2	util.datamanager: Rearrange locals	9 years ago
Kim Alvefur	43b814a83b	vairious: Add annotation when an empty environment is set [luacheck]	8 years ago
Kim Alvefur	2572082717	util.datamanager: Encode the 'store' path component, preserving underscores	8 years ago
Kim Alvefur	2e06e8e75d	util.datamanager: Remove check for file non-existence since ENOENT tells us that already	9 years ago
Kim Alvefur	d496a96705	util.datamanager: Split long line [luacheck]	9 years ago
Kim Alvefur	cb0953029e	util.datamanager: Expose atomic_store as store_raw	9 years ago
Kim Alvefur	c3aae40b19	util.datamanager: Use atomic store function when writing the first entry of a list	9 years ago
Kim Alvefur	f0c676e8c6	util.datamanager: Import value of ENOENT from util.pposix (if available)	9 years ago
Kim Alvefur	fc38b8c4f4	util.datamanager: Ignore ENOENT (no such file) when loading data	9 years ago
Kim Alvefur	7b4ad0e694	util.datamanager: Use pposix.atomic_append	9 years ago
Kim Alvefur	f36fed39e8	util.datamanager: Shorter pattern	9 years ago
Kim Alvefur	f0bd3ec403	util.datamanager: Remove extraneous decoding of + to space, which we never never encode to	9 years ago
Kim Alvefur	e3ab638262	util.datamanager: Use the 'base' argument to tonumber() to indicate hexadecimal	9 years ago
Kim Alvefur	d56bd9dbdb	util.datamanager: Add annotations to ignore name clashes [luacheck]	9 years ago
Kim Alvefur	c331fe1594	util.datamanager: Rename variables to avoid name clashes [luacheck]	9 years ago
Kim Alvefur	baba98852d	util.datamanager: Import tostring and type (fix global access)	10 years ago
Kim Alvefur	afe389d870	util.datamanager: Explicit handling of each error condition (see #632)	10 years ago
Kim Alvefur	fc2ad03510	util.datamanager: Unreference file handle after closing it to prevent trying to close it again (fixes #632)	10 years ago
Kim Alvefur	30b54f4ad7	util.datamanager: Add append to public api	10 years ago
Kim Alvefur	7816dfc838	util.datamanager: Return extra location info	10 years ago
Kim Alvefur	206330cea1	util.datamanager: Make sure only strings are passed as data to append()	10 years ago
Kim Alvefur	1510dd113d	util.datamanager: Add some comments about the append function	10 years ago
Kim Alvefur	4b380b1cbf	util.datamanager: Handle potential error from :write() call	10 years ago