We used to generate the tarball in the build pipeline, but that required
cloning the source repo a second time in the build pipeline.
The top-level pipeline will now generate the tarball and hands it off
to downstream pipelines, that way the build pipeline will work w/out
having to know which git repository to clone.
The end goal being that our internal QA suite can trigger off of the
public git repo or an internal git repo.
Have to manually link libtinfo (`-ltinfo`) because our FindLLVM
didn't add it to the LLVM_LIBRARIES variable for us. See:
- https://stackoverflow.com/questions/21477407/llvm-3-5-fails-to-link
Have to remove the CXX_STANDARD setting at the top of CMakeLists.txt
because of c++90 / c++11 ABI compatibility issues w/ LLVM. See:
- https://maleadt.github.io/LLVM.jl/dev/man/troubleshooting/
Rename "llvm/Config/config.h" "llvm/Config/llvm-config.h" because
LLVM renamed it in 2.8.
Have to link LLVM manually with the test binaries that use the
clamav object library instead of libclamav shared library.
CMake does not propagate library dependencies from object files.
I tested on ubuntu:16.04 with LLVM 3.6 built from source using:
```
/usr/local/bin/cmake .. -D CMAKE_INSTALL_PREFIX=/opt/llvm/3.6 \
-D LLVM_ENABLE_RTTI=ON
```
Then built clamav w/:
```
/usr/local/bin/cmake .. -D CMAKE_INSTALL_PREFIX=`pwd`/install \
-D BYTECODE_RUNTIME="llvm" \
-D LLVM_ROOT_DIR="/opt/llvm/3.6" \
-D LLVM_FIND_VERSION="3.6.0" && make && make install
```
CMake is now required to build.
The built-in LLVM is no longer available.
Also removed support for libltdl calls, which is not used in the CMake
builds, was only used when building with Autotools.
TODO: Fix CMake LLVM support & update to work with modern versions.
If you test in a docker container and don't create a user/switch to run
as something other than root, than the tests will fail.
Default freshclam behavior is to drop to run as the "clamav" user when
run as root. This change sets the DatabaseOwner to the current user, so
it the tests will pass when run as root.
Add options to build a source tarball.
This will replace Autotools' `make dist` feature.
Although CMake doesn't have an autogen step, this is still necessary if
we want to bundle in a local copy of the online documentation or if we
want to vendor dependencies from github/etc at configure time in the tarball.
There is a scan logic issue where the main libclamav scanning functions
create an extra "nested" fmap for each file being scanned. This is
slightly inefficient for a normal scan, but causes a major performance
issue when using ENGINE_OPTIONS_FORCE_TO_DISK. It causes every scanned
file to be duplicated in the temp directory before the scan.
We fix this by using `cli_magic_scan()` in `scan_common()` instead
of `cli_magic_scan_nested_fmap_type()`. We can do this now that the
`cl_scandesc_callback()` API creates an fmap for the caller, instead of
the old logic where `scan_common()` called different API's depending on
whether or not we have an fmap or a file descriptor.
I'd foolishly set the Iconv detection module to use uppercase to
differentiate it from the upstream (original) module. I also kind of
forgot to change the documentation to use the uppercase name. In
reviewing that, I decided it's better to match the upstream
FindIconv.cmake module as closely as possible in case they fix the issue
described here, so that we can stop carrying a custom module without
breaking the CMake options for everyone:
b26cdc5453
This reverts the CMake options for setting the Iconv include and library
paths back to match the upstream before 0.104 ships with stable CMake
support.
The WebServerHandler classes used for the tests have a "Test_" prefix
which causes pytest to print some warnings thinking they're special test
classes. This commit gets rid of the warnings.
If pytest is available in the path, but wasn't found in the Python 3
installation discovered by CMake, then the test suite will fail.
This can happen when two different Python 3 installations exist, eg 3.6
and 3.7 and when the newest one (prefered by CMake) doesn't have pytest.
This commit will use the correct command depending on where pytest was
(or wasn't) found. It will also fail more gracefully if neither pytest
nor unittest was found (this happens with some wierd python installs).
The milter default was not set correctly so it was starting
clamav-milter by default. Added default true/false settings for each of
the docker daemon environment variables.
Added command to entrypoint to change database directory ownership to
the clamav user, in case it is a mounted volume (which get root ownership
by default).
Removed the clamav user & group from the build image, as it isn't needed
until the base image.
Added announcement details and acknowledgements to the News document.
Added details to the Docker Readme describing:
- the tag naming convention
- how to mount database volumes
- various other tweaks
Fixed the path for the update script in the GitHub Action, and changed
from alpine-latest to ubuntu-latest, because alpine-latest doesn't seem
to be an option.
To reduce the need for constant database updates, also push a docker
image containing the virus database.
Ideally, this script is called every time a major virus database update
happens, but as that requires integration with other systems, lets start
by having a scheduled update.
To do this a github action workflow was added on a schedule that runs
once a month. It does require the proper secrets to be setup however and
made accessible.
Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
This dockerfile allows users/services compile the current codebase and
use the container as a dockerized clamav daemon.
Useful for testing, but also for running clamd containerized.
Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
As pytest can be installed in many ways, we can check for pytests
existence next to if it was pip-installed.
Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
There are some merits of having the configs in `/etc/clamav` and the databases
in `/var/lib/clamav` to match the way the distros all package it and so these
directories can be mounted volumes if needed.
The DATABASE_DIRECTORY and APP_CONFIG_DIRECTORY variables don't support
absolute paths. This patch changes that.
The testcase.py script is using Python 3.6+ syntax for the NamedTuple.
This commit reverts to the old 3.5 syntax until we can drop support for
Python 3.5.
The named "shared" is confusing, especially now that these features are
built as a static library instead of being directly compiled into the
various applications.
The previous UnRAR vuln fix worked for the (old, gone) Visual Studio
solution but broke the CMake build. We didn't notice because we tested
and approved it with Visual Studio before switching to CMake.
This change switches it to use the correct macro for the
libclamunrar_iface extension which is ".dll" instead of "lib".
The size of the UE buffer for the new Adobe Reader X encryption support
was not properly recorded and may result in reading too far into the UE
buffer. This patch checks the size of the UE buffer and rejects it if
the length is not 32, as it does with the other AES256 CBC method.
The XLM formula parser failed to account for string records that claim
to be longer than the formula data. This fix skips over the invalid
string records.
Also fixed an unrelated XLM parsing bug where BIFF name records weren't
handled on builds lacking the json-c library, resulting in verbose error
output. See https://bugzilla.clamav.net/show_bug.cgi?id=12639
After the previous improvements, there is still an issue wherein a
private database mirror may not yet have the very latest CVD and CDIFF.
In this case, and if the clients are still using the official DNS server
to check the latest versions, then the clients will try several times to
update with CDIFFs before downloading the entire CVD. This is clearly
not desirable, especially because that CVD is probably also 1 version
older than expected.
This commit keeps track of the number of patches and will only ask for
the CVD after the patches failed if no patches were downloaded AND we're
more than 1 version behind. What that means is that if we're 1 version
behind and the server is too -- it will log some warnings but succeed.
Or if we are multple versions behind but we did get some patches, we'll
settle for a minor update.
This change will essentially allow FreshClam to be used with private
mirrors that aren't updated the second a new daily database is
published. Without this change, private mirror operators may end up
frequently serving the entire daily CVD to fussy FreshClam clients.
If a new CVD download gets a version that is older than advertised, keep
the older version and retry the update so the incremental (CDIFF) update
gets us to the latest version.
Add back the mirrors.dat file to the database directory.
This new version of mirros.dat will store:
- A randomly generated UUID for the FreshClam User-Agent.
- A retry-after timestamp that so FreshClam won't try to update after
having received an HTTP 429 response until the Retry-After timeout has
expired.
Also: FreshClam will now exit with a failure in daemon mode if an HTTP
403 (Forbidden) was received, because retrying later won't help any.
The FreshClam user will have to take actions to get unblocked.
Added special warning messages for 403 and 429 HTTP codes.
For 403, FreshClam will fail (non-zero exit code) if not in daemon-mode.
For 429, FreshClam will succeed (exit 0) if not in daemon-mode.
Adds If-Modified-Since header for CVD downloads (not just CVD-head)
which should reduce data usage if DNS is advertising a newer version
than is actually available, which seems to happen sometimes due to
caching issues, it should still fail out when this happens - it just
won't have to download the older CVD, and should detect the HTTP 304
(Not-Modified) response instead.
Also replaced "Freshclam" with "FreshClam" in a few places, for
consistency.
I found one of the original executables that caused me to
add the lists of AutoIt functions / keywords to ClamAV and
it turns out that it also has these lists embedded within
it for some reason... I've updated ClamAV's lists based
on that. I tried using different versions of the AutoIt
compiler to generate an executable that uses this method
of storing the AutoIt keywords in lists and referencing
in the code by index but haven't been able to find a version
that does this. I'm not sure why that's the case x_x.
Maybe this is an extra layer of deobfuscation added by
a third-party program that wraps the built-in AutoIt
interpreter executable?
Example of malware using this:
2a4784980e48bd5e41e7e94bd5e796a93761d857dad4ecb8c36ebe2c7a055f7b
The fuzz targets should use malloc instead of mmap because libFuzzer
doesn't seem to track mmap leaks.
The ENABLE_FUZZ feature disables HAVE_MMAP which is supposed to
disable use of mmap in favor of malloc, but it doesn't because
checks for the ANONYMOUS_MAP feature execute even if HAVE_MMAP
is disabled, and some mmap usage in libclamav only tests if
ANONYMOUS_MAP is defined without checking HAVE_MMAP.
This commit will only define ANONYMOUS_MAP if HAVE_MMAP is set.
- 192959 Resource leak - In cli_bcomp_compare_check: Leak of
memory or pointers to system resources. Several fail cases
could lead to `buffer` or `tmp_buffer` being leaked
- 192934 Resource leak - In cli_bcomp_normalize_buffer: Leak of
memory or pointers to system resources. `hex_buffer` leaked
under certain conditions
- 185977 Resource leak - In ole2_process_property: Leak of memory
or pointers to system resources. A fail case could lead to
`outstr` and `outstr2` being leaked
- 185941 Resource leak - In header_cb (clamsubmit): Leak of
memory or pointers to system resources. A fail case could lead
to `mem` being leaked
- 185925 Resource leak - In load_oneyara: Leak of memory or
pointers to system resources. Several fail cases could lead
to `newident` being leaked
- 185918 Resource leak - In parsehwp3_docsummary: Leak of memory
or pointers to system resources. Not actually a leak, but
caused by checking for a condition that can’t occur.
- 185915 Resource leak - In parsehwp3_docinfo: Leak of memory or
pointers to system resources. Not actually a leak, but caused
by checking for a condition that can’t occur.
- 147644 Resource leak - In tcpserver: Leak of memory or pointers
to system resources. A fail case could lead to `info` being leaked
- 147642 Resource leak - In onas_ht_add_hierarchy: Leak of memory
or pointers to system resources. Several fail cases could lead
to `hnode` or `elem` memory leaks
Coverity warnings:
- 293628 Uninitialized pointer read - In reload_db: Reads
an uninitialized pointer or its target. A fail case
could lead to `rldata` being used before initialization
- 293627 Uninitialized pointer read - In reload_th: Reads
an uninitialized pointer or its target. A fail case could
lead to `engine` being used before initialization
- 265483 Uninitialized pointer write - In parseEmailFile:
Write to target of an uninitialized pointer. A fail case
could lead `ret` to be dereferenced and written to
- 265482 Resource leak - In parseEmailFile: Leak of memory
or pointers to system resources. A fail case could lead
to `head` being leaked
- 225221 Resource leak - In onas_get_opt_list: Leak of memory
or pointers to system resources. A fail case could lead to
`opt_list` being leaked
- 225181 Resource leak - In onas_ht_rm_hierarchy: Leak of
memory or pointers to system resources. A fail case could
lead to `prntname` being leaked
- 193874 Resource leak - In cli_genfname: Leak of memory
or pointers to system resources. A fail case could lead
to `sanitized_prefix` being leaked
- 225196 Resource leak - In onas_fan_eloop: Leak of memory
or pointers to system resources. A fail cases could lead
to `event_data` being leaked
Also, I added some unresolved comments regarding clamonacc
functionality, and added a version compatibility check that
is shown in the example code in the `fanotify` man page
Example: #NoTrayIcon will result in ##NoTrayIcon.
Background: Prefixbyte is already included in the string.
Added two other TODO at issues I noticed when doing a autoit
decompiler in Python.
I can't test these changes nor I know how clamav scans the decompiled
autoit script for malicious content.
So I leave it to you it implement these / or not.
The plus-bug is probably not very critical however some incorrect
quoted / unclose string can be a create some serve problem for a
parser (you may apply later to the decompiled output).
This makes the blob allocator behave the same on Windows and on
Linux by defaulting to memory allocations in units of the page size
instead of performing the allocations all at once (for better or
worse; it's unclear which is better on modern systems that are
already expected to have > 1 GB of memory to load the official
signature sets).
This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673
Changes in 0.103 to order of operations for creating fmaps and
performaing hashes of fmaps resulted errors when scanning files that are
4096M and a different (but related) error when scanning files > 4096M.
This is despite the fact that scanning is supposed to be limited to
--max-scansize (MaxScanSize) and was also apparently limited to
INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes
for a few large file formats.
(see https://bugzilla.clamav.net/show_bug.cgi?id=10960)
This last limitation was not documented, so I added it to the sample
clamd.conf.
Anyways, the main issue is that the fmap module was using "unsigned int"
and was then enforcing a limitation (verbose error messages) when that
a map length exceeded the capapacity of an unsigned int. This commit
switches the associated variables over to uint64_t, and while fmaps are
still limited to size_t in other places, the fmap module will at least
work with files > 4G on 64bit systems.
In testing this, I found that the time to hash a file, particularly when
hashing a file on an NTFS partition from Linux was really slow because
we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing
this to 10MB chunks speeds up scanning of large files.
Finally, now that hashing is performed immediately when an fmap is
created for a file, hashing of files larger than max-scansize was
occuring. This commit adds checks to bail out early if the file size
exceeds the maximum before creating an fmap. It will alert with the
Heuristics.Limits.Exceeded name if the heuristic is enabled.
Also fixed CheckFmapFeatures.cmake module that detects if
sysconf(_SC_PAGESIZE) is available.
Also add support for using pytest instead of python's unittest to make
it easier to find & read failed test results.
Clean up the log output in check_clamd when printing "wrong reply"
results.
Switch from using collections.namedtuple() to defining our own
NamedTuple subclass, which adds member variable typing.
Joel Esler