This is also not strictly necessary, but i found it rather helpful to
be able to identify and distinguish the individual clamonacc threads in
the usual Linux system tools ("ps", "top", "gdb", etc.).
Setting the thread name is already done for the threads started by the
bundled c-thread-pool library ("thread-pool-N"). Only the clamonacc
threads all identify themselfes as "clamonacc".
This commit now sets the following - shortened (due to the 16 character
limit) - thread names:
- "clamonacc-ddd" for the ddd/inotify thread.
- "clamonacc-sq" for the scan queue thread.
- the main clamonacc thread currently keeps the name "clamonacc" in-
herited from the main process.
This is not strictly necessary, but i found it rather helpful to know
what is going on during the debugging process and while trying to under-
stand how clamonacc actually works.
Explicit setting of pthread_sigmask in the individual threads
should not be necessary at all(?), since it is already done in
the main process/thread and should be inherited from there. But
since the sigfillset/sigdelset lines are already there, at least
they should be consistent with regard to the undefined behaviour
caused by ignoring SIGFPE, SIGILL, SIGSEGV, or SIGBUS signals.
Improvements to the logg() API appear to have corrected a bug wherein
debug/warning/error messages were writing to stdout instead of stderr.
Now that they're going to stderr correctly, this fixes the test to match.
Don't crop logs so we can see the whole valgrind output in large logs.
Generate suppression rules, in case they're needed for slight variations
on existing suppressions or sometimes new suppressions.
Increase valgrind's stack size for initial thread because I observed some
unusual errors in custom tests on Debian 10 like this:
7: [INFO]: ==9911== Process terminating with default action of signal 15 (SIGTERM)
7: [INFO]: ==9911== at 0x401A4FB: __open_nocancel (open64_nocancel.c:45)
7: [INFO]: ==9911== by 0x4006F65: open_verify.constprop.12 (dl-load.c:1728)
7: [INFO]: ==9911== by 0x4007737: open_path (dl-load.c:2057)
7: [INFO]: ==9911== by 0x4008C17: _dl_map_object (dl-load.c:2297)
7: [INFO]: ==9911== by 0x400D291: openaux (dl-deps.c:64)
7: [INFO]: ==9911== by 0x401956A: _dl_catch_exception (dl-error-skeleton.c:196)
7: [INFO]: ==9911== by 0x400D605: _dl_map_object_deps (dl-deps.c:248)
7: [INFO]: ==9911== by 0x4003AFA: dl_main (rtld.c:1733)
7: [INFO]: ==9911== by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253)
7: [INFO]: ==9911== by 0x4002117: _dl_start_final (rtld.c:415)
7: [INFO]: ==9911== by 0x4002117: _dl_start (rtld.c:522)
7: [INFO]: ==9911== by 0x4001097: ??? (in /lib/x86_64-linux-gnu/ld-2.28.so)
7: [INFO]: ==9911== by 0x1: ???
7: [INFO]: ==9911== Jump to the invalid address stated on the next line
7: [INFO]: ==9911== at 0x1036: ???
7: [INFO]: ==9911== by 0x4006F65: open_verify.constprop.12 (dl-load.c:1728)
7: [INFO]: ==9911== by 0x4007737: open_path (dl-load.c:2057)
7: [INFO]: ==9911== by 0x4008C17: _dl_map_object (dl-load.c:2297)
7: [INFO]: ==9911== by 0x400D291: openaux (dl-deps.c:64)
7: [INFO]: ==9911== by 0x401956A: _dl_catch_exception (dl-error-skeleton.c:196)
7: [INFO]: ==9911== by 0x400D605: _dl_map_object_deps (dl-deps.c:248)
7: [INFO]: ==9911== by 0x4003AFA: dl_main (rtld.c:1733)
7: [INFO]: ==9911== by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253)
7: [INFO]: ==9911== by 0x4002117: _dl_start_final (rtld.c:415)
7: [INFO]: ==9911== by 0x4002117: _dl_start (rtld.c:522)
7: [INFO]: ==9911== by 0x4001097: ??? (in /lib/x86_64-linux-gnu/ld-2.28.so)
7: [INFO]: ==9911== by 0x1: ???
7: [INFO]: ==9911== Address 0x1036 is not stack'd, malloc'd or (recently) free'd
7: [INFO]: ==9911==
7: [INFO]: ==9911==
7: [INFO]: ==9911== Process terminating with default action of signal 11 (SIGSEGV)
7: [INFO]: ==9911== Bad permissions for mapped region at address 0x1036
7: [INFO]: ==9911== at 0x1036: ???
7: [INFO]: ==9911== by 0x4006F65: open_verify.constprop.12 (dl-load.c:1728)
7: [INFO]: ==9911== by 0x4007737: open_path (dl-load.c:2057)
7: [INFO]: ==9911== by 0x4008C17: _dl_map_object (dl-load.c:2297)
7: [INFO]: ==9911== by 0x400D291: openaux (dl-deps.c:64)
7: [INFO]: ==9911== by 0x401956A: _dl_catch_exception (dl-error-skeleton.c:196)
7: [INFO]: ==9911== by 0x400D605: _dl_map_object_deps (dl-deps.c:248)
7: [INFO]: ==9911== by 0x4003AFA: dl_main (rtld.c:1733)
7: [INFO]: ==9911== by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253)
7: [INFO]: ==9911== by 0x4002117: _dl_start_final (rtld.c:415)
7: [INFO]: ==9911== by 0x4002117: _dl_start (rtld.c:522)
7: [INFO]: ==9911== by 0x4001097: ??? (in /lib/x86_64-linux-gnu/ld-2.28.so)
7: [INFO]: ==9911== by 0x1: ???
Per this conversation: https://bugs.kde.org/show_bug.cgi?id=359705
I'm testing to see if the issue is resolved by increasing valgrind's
main stack size.
https://valgrind.org/docs/manual/manual-core.html claims that the
default is the current `ulimit` value or 16MB, whichever is lower.
The logic for parsing a logical subsignature isn't clearly identified
and has been, perhaps mistakenly or out of convenience, used to when
parsing NDB signatures in addition to LDB subsignatures. What this means
is that you can technically use a PCRE subsignature in an NDB file and
clam won't complain about it. It won't work however, because a PCRE
subsignature requires another matching subsignature to trigger it, but
it will parse. The same is likely true for byte-compare subsignatures.
This commit restructures that logic a bit so subsignature parsing has
its own function and is more organized.
I also renamed the functions a little bit and added lots of comments.
I fixed a few minor warnings relating to format string characters.
The change in str.c:cli_ldbtokenize is to prevent a buffer under-read if
you were to use the function on the start of a buffer, as is now down in
this commit.
FreeBSD and OpenBSD package maintainers identified failures when
pkg-config .pc files not present for curses/ncurses.
Patch courtesy of Stuart Henderson.
Corrected buffer size check in the regex signature parsing code.
This resolves a possible 1-byte stack buffer overwrite (NULL byte).
We determined that this issue is not a vulnerability.
Fixes: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43869
Time is always a nasty thing and without proper timezone data it is even
harder. Set a default timezone (easily to be overridde by the user from
the cli with docker run --env) and as a consequence add the timezone
data package.
Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
Prevent healthcheck from initially failing as the container
is still starting up. ClamAV's database loading can take quite a time,
in which the healthcheck is constantly failing. While it is impossible to
have a default that works everywhere (slower systems vs fast ones). Using 6
minutes tries to find a compromise here but it by no means perfect.
Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
The XLS parsing code for extracting images and XLM macros is alerting
with Heuristics.OLE2.ContainsMacros.XLM if any images are found if the
--alert-macros (AlertOLE2Macros) option is enabled.
This fix moves the alert logic before the XLM & image extraction logic
where we know at least one macro exists, but before we try to extract it
I've wrapped it with an "if (has_xlm)" condition.
Resolves:
- https://github.com/Cisco-Talos/clamav/issues/423
- https://bugzilla.clamav.net/show_bug.cgi?id=12844
The byte compare feature in logical signatures will cause the rule to
alert if it successfully matches regardless of the rest of the logical
signature.
An easy way to test this is with a logical signature that has two
bcomp subsignatures and requires both to match for the rule to alert.
In the following example, we have 4 signatures where
- the first will match both bcomp subsigs.
- the second will match neither.
- the last two match just one bcomp subsig.
In an --allmatch test, you'll find that the 3 of these match, with the
first one matching *twice*, once for each bcomp subsig.
test.ldb:
```
bcomp.both;Engine:51-255,Target:0;0&1&2&3;4141;0(>>5#hb2#=123);4242;2(>>5#hb2#=255)
bcomp.neither;Engine:51-255,Target:0;0&1&2&3;4141;0(>>5#hb2#=124);4242;2(>>5#hb2#=254)
bcomp.second;Engine:51-255,Target:0;0&1&2&3;4141;0(>>5#hb2#=124);4242;2(>>5#hb2#=255)
bcomp.first;Engine:51-255,Target:0;0&1&2&3;4141;0(>>5#hb2#=123);4242;2(>>5#hb2#=254)
```
test.sample:
```
AA = 7B; BB = FF
```
You can also try a similar test to compare the behavior with regular
ac-pattern-match subsigs with this lsig-test.ldb:
```
pattern.both;Engine:51-255,Target:0;0&1;4141;4242
pattern.neither;Engine:51-255,Target:0;0&1;4140;4241
pattern.second;Engine:51-255,Target:0;0&1;4140;4242
pattern.first;Engine:51-255,Target:0;0&1;4141;4241
```
This commit fixes the issue by incrementing the logical subsignature
count for each bcomp subsig match instead of appending an alert for
each bcomp match.
Also removed call to `lsig_sub_matched()` that didn't do anything.
* Added loglevel parameter to logg()
* Fix logg and mprintf internals with new loglevels
* Update all logg calls to set loglevel
* Update all mprintf calls to set loglevel
* Fix hidden logg calls
* Executed clam-format
Add support for compiling with external TomsFastMath library provided by
the system instead of compiling the vendored copy into libclamav.
The vendored source is still built directly into libclamav instead of as
a separate library the way libmspack is done.
The rationale is that:
A) it's more complicated to deal with possibly compiling as static or
dynamic, and also
B) libmspack and libunrar are compiled separately primarily because of
licensing concerns. TomsFastMath public domain, so that isn't a concern.
Resolves: https://bugzilla.clamav.net/show_bug.cgi?id=12562
Update the vendored TomsFastMath (TFM) library to v0.13.1.
Resolves: https://bugzilla.clamav.net/show_bug.cgi?id=11992
I removed compatibility macro's from when libTomMath was used.
This required removing a bunch of faux-error handling because
the fast-math equivalent functions return void, and cannot fail.
The previous version used had named the header "bignum_fast.h"
instead of "tfm.h" and had customizations in that header to enable
TFM_CHECK all the time, and also TFM_NO_ASM if __GNUC__ not defined
or if the system isn't 64bit architecture. This update uses tfm.h
as-is, and has CMake define TFM_CHECK and TFM_NO_ASM as needed.
I've kept bignum.h as an interface to including tfm.h so that in
the future we can more easily add support for system-installed
TomsFastMath instead of the vendored one, taking inspiration from
Debian's patch to support system-TomsFastMath.
See: https://salsa.debian.org/clamav-team/clamav/-/blob/unstable/debian/patches/add-support-for-system-tomsfastmath.patch
* Work with data as &[u8] instead String/&str to avoid unnecessary UTF-8
validation and reuse read buffers.
* Make error handling more concise
* Address Clippy-raised issues
The logging functions use a callback to print log messages. Because the
callback could be anything provided by an application, it isn't safe to
log while holding a mutex.
This commit defers error reporting in cacheset_add() to prevent running
the callback while the mutex is held.
The CIFuzz tool was observed detecting an issue in code unrelated to a
pull request. While it was relatively benign in this case, it has the
potential to accidentally disclose a vulnerability publicly without
allowing a non-disclosure period for us to publish a fixed patch
version.
I'm not comfortable with this risk, so I'm removing the workflow.
RelWithDebInfo is our preferred build type. It has optimizations and
should run faster, but includes debugging symbols for better profiling,
stack traces, etc.
The Rust MinSizeRel support is just Release mode for now. There are
optimizations we can do to shrink it further, but for now at least it
won't actually be Debug (aka slow).
I found two issues with the cl_load fuzz targets, one of which impacts
the scanfile and scanmap fuzz targets:
1. We were defining the preprocessor definitions incorrectly using
"SCAN_TARGETS" instead of "TARGET" in unit_tests/CMakeLists.txt.
For the scan fuzz targets this meant it wasn't properly defining
unique settings for each compiled target.
For the cl_load fuzz target it's worse, it wasn't setting the
database file name correctly which means it rejected the filenames
entirely for not having a legitimate suffix.
2. We were pre-compiling the engine before loading signatures.
You can't load sigs for an engine that's already compiled, so this
would also fail right away without trying to load any sigs.
Resolves https://bugzilla.clamav.net/show_bug.cgi?id=12592
A null-dereference crash happens if the database directory contains:
a good `daily.cvd` symlink
a broken `daily.cld` symlink
or:
a broken `daily.cvd` symlink
a good `daily.cld` symlink
You're not supposed to have both a .cvd and .cld for the same file
at the same time. And making individual symlinks for the database
files is also unexpected.
That is to say that this is not an intended or supported use case
for the database directory. But it's a fairly simple bug to fix.
The issue is that a bad symlink is still added to the database load-
list after the access check fails. This commit skips daily databases
that fail the access check.
The database update script has a line to delete freshclam.dat after the
update, but this fails when attempting to update older images that have
mirrors.dat or no dat file at all. This issue is compounded by a bug
where the -t (--tags) option doesn't work so it tries to update all
images every time, and then of course fails on the older ones.
This commit has the script try removing freshclam.dat or mirrors.dat and
allows it to succeed even if neither exist.
Adds a clamscan test to verify that --allmatch mode works for the
clam.exe program when loading MD5, SHA1, SHA256, PE-section, & PE-import
hash signatures.
The PE section hash scanning code didn't implement the all-match check.
While this check isn't the ideal implementation for all-match mode...
(see the commit message for the previous commit)
...it's simple enough to add the all-match check here for now.
All-match mode is presently not working if you have multiple hash-based
signatures for the same file. That is, only the first signature will
alert before moving on to other parts of the scan.
Our current all-match implementation adds checked into every file parser
(often at multiple layers in the scanning process) to make sure that the
scan continued even if an alert is found, but only when in all-match mode.
This is clearly an error-prone approach.
Instead, I would like to change it so that every alert appends to an alert
list. When in all-match mode, the scan status would continue with "clean"
until the very end when the process is complete and the list of alerts is
evaluated.
Unfortunately, that's a much bigger job. For this specific bug, it is
simple enough to fix the broken logic that aborts the hash scanning
early. In the future, we should rework all of this logic throughout
libclamav, as described above.
In testing, I found that libclang.so/clang.dll is required by bindgen
and was not found on all of our test machines. To resolve this we will
only generate sys.rs bindings when CMake MAINTAINER_MODE option is "ON".
This is unfortunate that we have to commit generated source to version
control. But as a benefit it makes rust-analyzer happier when editing a
workspace that hasn't yet been compiled. And it makes it more reasonable
that the generated sys.rs file generated to the source directory and not
the build directory (something we hadn't resolved yet).
Use bindgen to generate Rust-bindings for some libclamav internal
functions and structures in an new "sys.rs" module.
"sys.rs" will be generated at build-time and unfortunately must be
dropped in the source/libclamav_rust/src directory rather than under the
build directory. As far as I know, Cargo/Rust provide no way to set an
include path to the build directory to separately place generated files.
TODO: Verify if this is true and move sys.rs to the build directory if
possible.
Using the new bindings with:
- the logging module.
- the cdiff module.
Also:
- Removed clamav_rust.h from .gitignore, because we generate it in the
build directory now.
- Removed the hand-written bindings from the cbindgen exclusions.
lib.rs has an annotation that prevents cbindgen from looking at sys.rs.
- Fixed a `u8` -> `c_char` type issue in cdiff in the cli_getdsig() call
parameters.
We received a high FP report rate for Heuristics.PNG.CVE-2010-1205.
This is a 11 year old CVE and is likely no longer relevant anyways.
Removing Heuristics.PNG.CVE-2010-1205 means there is no longer any point
decompressing the PNG image data in the PNG CVE checker. Removing image
data decompression in the CVE checker should improve PNG image scan
times.
The OOXML parser in libclamav may try to extract an entry that is
missing a file name. This results in an invalid 0x1 pointer dereference
in the ZIP parser that is likely to crash the scanning application.
This commit fixes the issue by requiring both the PartName (PN) *and*
the ContentType (CT) variables to be non-NULL or else the entry will be
skipped.
Thank you Laurent Delosieres for reporting this issue.
Rustify cdiff_apply() and clean up error handling:
- Restore [some] safety and clean up error handling.
- Use rust-crypto sha2 instead of OpenSSL's.
Fix signedness of cli_versig2() dsig parameter.
c_char may be an i8 or u8 depending on platform:
https://doc.rust-lang.org/src/std/os/raw/mod.rs.html#91-133
Rustify cmd_close():
- Consolidate DEL/XCHG records.
- Tidy up ADD handling.
- Various error handling cleanup, etc.
Remove some extra clones that don't seem to be neceesary.
Replace writeln format macro with plain write command.
Use a buffered writer for deletes, exchanges, & writes.
Switching from individual `write` syscalls per change to a
buffered writer appears to speed up cdiff-apply by about 2x.
Apply both .cdiff and .script CVD patches.
Note: A script is a non-compressed and unsigned file containing cdiff
commands. There is no header or footer that should be processed.
This Rust-based implementation of the cdiff-apply feature includes
equivalent features as found in the C-based implementation:
- cdiff file signature validation against sha256 of the file contents
- Gz decoding of file contents
- File open command
- File close command
- Signature add command
- Line delete command
- Xchg command
- Move command
- Unlink command
This Rust implementation adds cdiff-apply unit tests to verify correct
functionality.
Frank Fegert