ClamAV initalization's rarload() function tries to load
libclamunrar_iface from the install path before checking under
LD_LIBRARY_PATH.
This means the unit tests will use the wrong unrar library if testing on
a system where ClamAV is already installed.
In the event there is an ABI break between versions, this will cause a
bunch of tests to fail.
This commit fixes the issue by checking for libclamunrar_iface under
LD_LIBRARY_PATH *first* before checking in the install lib directory.
Note in the previous version we were also checking LD_LIBRARY_PATH on
Windows, which is not a thing. I removed this.
Fixes: https://github.com/Cisco-Talos/clamav/issues/1249
Also removed check for WARN_DLOPEN_FAIL define, which was not used, and
mistakenly set for the unrar library build target.
Prevent allocating more than 1GB regardless of what is requested.
RAR dictionary sizes may not be larger than 1GB, at least in the current
version.
This is a cherry-pick of commit 9b444e7e02
This is a cherry-pick of commit 24f225c21f
Modification to unrar codebase allowing skipping of files within
Solid archives when parsing in extraction mode, enabling us to skip
encrypted files while still scanning metadata and potentially
scanning unencrypted files later in the archive.
UnRAR logic replaces directory symlinks found within archive file entry
file paths with actual directories by deleting them after they're
extracted.
Unfortunately, this logic extends to deleting existing directories if you
set the `DestName` instead of the `DestPath` in this API:
rc = RARProcessFile(hArchive, RAR_EXTRACT, NULL, destFilePath);
In the future UnRAR may change to disable the `LinksToDirs()` feature
if using the `DestName` parameter. In the meantime, this commit
completely disables it for our use case.
Prevent allocating more than 1GB regardless of what is requested.
RAR dictionary sizes may not be larger than 1GB, at least in the current
version.
This is a cherry-pick of commit 9b444e7e02
This is a cherry-pick of commit 24f225c21f
Modification to unrar codebase allowing skipping of files within
Solid archives when parsing in extraction mode, enabling us to skip
encrypted files while still scanning metadata and potentially
scanning unencrypted files later in the archive.
UnRAR logic replaces directory symlinks found within archive file entry
file paths with actual directories by deleting them after they're
extracted.
Unfortunately, this logic extends to deleting existing directories if you
set the `DestName` instead of the `DestPath` in this API:
rc = RARProcessFile(hArchive, RAR_EXTRACT, NULL, destFilePath);
In the future UnRAR may change to disable the `LinksToDirs()` feature
if using the `DestName` parameter. In the meantime, this commit
completely disables it for our use case.
libclamav.map: Add missing symbol and correct symbol version.
libclamunrar.map: Use symbol version-script for libclamunrar, too.
Thank you to Sebastian Andrzej Siewior for the help.
Also fix a unittest linker issue...
Adding libclamav.map causes libclamav to no longer export zlib
when zlib is statically linked.
What was weird is that libxml2 depends on zlib and the check_clamav
unit test program was using those symbols from libclamav.
Introducing libclamav.map broke that even though we were explicitly
trying to link check_clamav with ZLIB::ZLIB as well.
For reasons I can't explain, linking check_clamav with the
ClamAV::common library managed to properly link it with ZLIB::ZLIB
and so the undefined references go away.
Also in this commit, I've removed the `.map` files from .gitignore
I'm not sure why they were ignored before.
The file 'isnt.cpp' (is NT) contains code that is only ever needed in Windows
builds. Changes since UnRAR 6.0.7 break the build for this file on
non-Windows systems.
This commit only builds 'isnt.cpp' when on Windows.
Prevent allocating more than 1GB regardless of what is requested.
RAR dictionary sizes may not be larger than 1GB, at least in the current
version.
This is a cherry-pick of commit 9b444e7e02
This is a cherry-pick of commit 24f225c21f
Modification to unrar codebase allowing skipping of files within
Solid archives when parsing in extraction mode, enabling us to skip
encrypted files while still scanning metadata and potentially
scanning unencrypted files later in the archive.
To build with code signing, the macOS build must have:
-G Xcode \
-D CLAMAV_SIGN_FILE=ON \
-D CODE_SIGN_IDENTITY="...your codesign ID..." \
-D DEVELOPMENT_TEAM_ID="...your team ID..." \
You can find the codesign ID using:
/usr/bin/env xcrun security find-identity -v -p codesigning
The team ID should also be listed in the identity description.
Also I changed the package name for APPLE to be "clamav" so it doesn't
put "ClamAV <version>" in the PKG PackageInfo like this:
com.cisco.ClamAV 0.104.0.libraries
Instead, it should just be something like:
com.cisco.clamav.libraries
Version is a separate field in that file and shouldn't be in the name.
The libclamunrar (and libclamunrar_iface) SO versions tracked
libclamav's SO version in the old Autotools build system.
We accidentally rolled it backwards, setting it to be similar to UnRAR's
project version. Since the official UnRAR project doesn't have a Unix SO
version that we "should" match, and to prevent the theoretical
possibility of having a collision if an old and new clamav were
installed on the same box, we should make libclamunrar's version track
libclamav as it was before (and before 0.104 is released with CMake
being the stable, and only, build system).
Xcode (and perhaps some other generators?) do not like targets that have
only object files. See:
https://cmake.org/cmake/help/latest/command/add_library.html#object-libraries
And: https://cmake.org/pipermail/cmake/2016-May/063479.html
This issue manifests when using `-G Xcode` on macOS as the library
dylibs being missing when linking with other binaries.
This commit removes the object libraries for libclamav, libfreshclam,
libclamunrar_iface, libclamunrar, libclammspack, and (lib)common
because they were used by static or shared libs that didn't
themselves have any added sources.
Add getter & setter for the debug flag, so it isn't referenced by unit
tests or other code that links with libclamav. This is needed because
global variables are exported symbols on Windows.
CMake/CPack is already used to build:
- TGZ source tarball
- WiX-based installer (Windows)
- ZIP install packages (Windows)
This commit adds support for building:
- macOS PKG installer
- DEB package
- RPM package
This should also enable building FreeBSD packages, but while I was able
to build all of the static dependencies using Mussels, CMake/CPack 3.20
doesn't appear to have the the FreeBSD generator despite being in the
documentation.
The package names are will be in this format:
clamav-<version><suffix>.<os>.<arch>.<extension>
This includes changing the Windows .zip and .msi installer names.
E.g.:
- clamav-0.104.0-rc.macos.x86_64.pkg
- clamav-0.104.0-rc.win.win32.msi
- clamav-0.104.0-rc.win.win32.zip
- clamav-0.104.0-rc.win.x64.msi
- clamav-0.104.0-rc.linux.x86_64.deb
- clamav-0.104.0-rc.linux.x86_64.rpm
Notes about building the packages:
I've only tested this with building ClamAV using static dependencies that
I build using the clamav_deps "host-static" recipes from the "clamav"
Mussels cookbook. Eg:
msl build clamav_deps -t host-static
Here's an example configuration to build clam in this way, installing to
/usr/local/clamav:
```sh
cmake .. \
-D CMAKE_FIND_PACKAGE_PREFER_CONFIG=TRUE \
-D CMAKE_PREFIX_PATH=$HOME/.mussels/install/host-static \
-D CMAKE_INSTALL_PREFIX="/usr/local/clamav" \
-D CMAKE_MODULE_PATH=$HOME/.mussels/install/host-static/lib/cmake \
-D CMAKE_BUILD_TYPE=RelWithDebInfo \
-D ENABLE_EXAMPLES=OFF \
-D JSONC_INCLUDE_DIR="$HOME/.mussels/install/host-static/include/json-c" \
-D JSONC_LIBRARY="$HOME/.mussels/install/host-static/lib/libjson-c.a" \
-D ENABLE_JSON_SHARED=OFF \
-D BZIP2_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D BZIP2_LIBRARY_RELEASE="$HOME/.mussels/install/host-static/lib/libbz2_static.a" \
-D OPENSSL_ROOT_DIR="$HOME/.mussels/install/host-static" \
-D OPENSSL_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D OPENSSL_CRYPTO_LIBRARY="$HOME/.mussels/install/host-static/lib/libcrypto.a" \
-D OPENSSL_SSL_LIBRARY="$HOME/.mussels/install/host-static/lib/libssl.a" \
-D LIBXML2_INCLUDE_DIR="$HOME/.mussels/install/host-static/include/libxml2" \
-D LIBXML2_LIBRARY="$HOME/.mussels/install/host-static/lib/libxml2.a" \
-D PCRE2_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D PCRE2_LIBRARY="$HOME/.mussels/install/host-static/lib/libpcre2-8.a" \
-D CURSES_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D CURSES_LIBRARY="$HOME/.mussels/install/host-static/lib/libncurses.a" \
-D ZLIB_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D ZLIB_LIBRARY="$HOME/.mussels/install/host-static/lib/libz.a" \
-D LIBCHECK_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D LIBCHECK_LIBRARY="$HOME/.mussels/install/host-static/lib/libcheck.a"
```
Set CPACK_PACKAGING_INSTALL_PREFIX to customize the resulting package's
install location. This can be different than the install prefix. E.g.:
```sh
-D CMAKE_INSTALL_PREFIX="/usr/local/clamav" \
-D CPACK_PACKAGING_INSTALL_PREFIX="/usr/local/clamav" \
```
Then `make` and then one of these, depending on the platform:
```sh
cpack # macOS: productbuild is default
cpack -G DEB # Debian-based
cpack -G RPM # RPM-based
```
On macOS you'll need to `pip3 install markdown` so that the NEWS.md file can
be converted to html so it will render in the installer.
On RPM-based systems, you'll need rpmbuild (install rpm-build)
This commit also fixes an issue where the html manual (if present) was
not correctly added to the Windows (or now other) install packages.
Fix num to hex function for Windows installer guid
Fix win32 cpack build
Fix macOS cpack build
This is a cherry-pick of commit 24f225c21f
Modification to unrar codebase allowing skipping of files within
Solid archives when parsing in extraction mode, enabling us to skip
encrypted files while still scanning metadata and potentially
scanning unencrypted files later in the archive.
Adds support to the pcre2 and pthreadw32 Find<Package>.cmake modules for
correctly discovering the debug versions. This change modeled after the
upstream FindBZip2.cmake module.
Also eliminated HAVE_STRUCT_TIMESPEC redefinition warnings in Windows
builds.
In testing on Alpine, I found that most libs were installing to
<prefix>/lib while libclamav installed to <prefix>/lib64. Those who like
multiarch will advocate for lib64, though I only actually noticed it
because clamscan failed to find libclamav.so! Anyways, they should all
install to lib64 by default if that's what how the system is set up.
Using ${CMAKE_INSTALL_FULL_LIBDIR} instead of <prefix>/lib will do that.
Also creates a ZIP for non-Admin (per-user) installs.
WIX requires the license file to have a .txt or .rtf extension so I
added the .txt extension. I've taken the opportunity to migrate the 3rd
party licenses to a COPYING subdirectory and have added licensing
details to the README.md file.
To build the installer, install WIX and simply run `cpack -C Release`
Also removed the explicit --config option from the
clamav-clamonacc.service file because it should not be required and
isn't being generated correctly when using autotools anyways, especially
after changes in this commit.
Enabled the metadata collection feature, scan heuristics, and all-match
mode when fuzzing in the interest of better code coverage.
Also remove deprecated STREAM command.
An ENABLE_TESTS CMake option is provided so that users can disable
testing if they don't want it. Instructions for how to use this
included in the INSTALL.cmake.md file.
If you run `ctest`, each testcase will write out a log file to the
<build>/unit_tests directory.
As with Autotools' make check, the test files are from test/.split
and unit_tests/.split files, but for CMake these are generated at
build time instead of at test time.
On Posix systems, sets the LD_LIBRARY_PATH so that ClamAV-compiled
libraries can be loaded when running tests.
On Windows systems, CTest will identify and collect all library
dependencies and assemble a temporarily install under the
build/unit_tests directory so that the libraries can be loaded when
running tests.
The same feature is used on Windows when using CMake to install to
collect all DLL dependencies so that users don't have to install them
manually afterwards.
Each of the CTest tests are run using a custom wrapper around Python's
unittest framework, which is also responsible for finding and inserting
valgrind into the valgrind tests on Posix systems.
Unlike with Autotools, the CMake CTest Valgrind-tests are enabled by
default, if Valgrind can be found. There's no need to set VG=1.
CTest's memcheck module is NOT supported, because we use Python to
orchestrate our tests.
Added a bunch of Windows compatibility changes to the unit tests.
These were primarily changing / to PATHSEP and making adjustments
to use Win32 C headers and ifdef out the POSIX ones which aren't
available on Windows. Also disabled a bunch of tests on Win32
that don't work on Windows, notably the mmap ones and FD-passing
(i.e. FILEDES) ones.
Add JSON_C_HAVE_INTTYPES_H definition to clamav-config.h to eliminate
warnings on Windows where json.h is included after inttypes.h because
json-c's inttypes replacement relies on it.
This is a it of a hack and may be removed if json-c fixes their
inttypes header stuff in the future.
Add preprocessor definitions on Windows to disable MSVC warnings about
CRT secure and nonstandard functions. While there may be a better
solution, this is needed to be able to see other more serious warnings.
Add missing file comment block and copyright statement for clamsubmit.c.
Also change json-c/json.h include filename to json.h in clamsubmit.c.
The directory name is not required.
Changed the hash table data integer type from long, which is poorly
defined, to size_t -- which is capable of storing a pointer. Fixed a
bunch of casts regarding this variable to eliminate warnings.
Fixed two bugs causing utf8 encoding unit tests to fail on Windows:
- The in_size variable should be the number of bytes, not the character
count. This was was causing the SHIFT_JIS (japanese codepage) to UTF8
transcoding test to only transcode half the bytes.
- It turns out that the MultiByteToWideChar() API can't transcode
UTF16-BE to UTF16-LE. The solution is to just iterate over the buffer
and flip the bytes on each uint16_t. This but was causing the UTF16-BE
to UTF8 tests to fail.
I also split up the utf8 transcoding tests into separate tests so I
could see all of the failures instead of just the first one.
Added a flags parameter to the unit test function to open testfiles
because it turns out that on Windows if a file contains the \r\n it will
replace it with just \n if you opened the file as a text file instead of
as binary. However, if we open the CBC files as binary, then a bunch of
bytecode tests fail. So I've changed the tests to open the CBC files in
the bytecode tests as text files and open all other files as binary.
Ported the feature tests from shell scripts to Python using a modified
version of our QA test-framework, which is largely compatible and will
allow us to migrate some QA tests into this repo. I'd like to add GitHub
Actions pipelines in the future so that all public PR's get some testing
before anyone has to manually review them.
The clamd --log option was missing from the help string, though it
definitely works. I've added it in this commit.
It appears that clamd.c was never clang-format'd, so this commit also
reformats clamd.c.
Some of the check_clamd tests expected the path returned by clamd to
match character for character with original path sent to clamd. However,
as we now evaluate real paths before a scan, the path returned by clamd
isn't going to match the relative (and possibly symlink-ridden) path
passed to clamdscan. I fixed this test by changing the test to search
for the basename: <signature> FOUND within the response instead of
matching the exact path.
Autotools: Link check_clamd with libclamav so we can use our utility
functions in check_clamd.c.
This patch adds experimental-quality CMake build tooling.
The libmspack build required a modification to use "" instead of <> for
header #includes. This will hopefully be included in the libmspack
upstream project when adding CMake build tooling to libmspack.
Removed use of libltdl when using CMake.
Flex & Bison are now required to build.
If -DMAINTAINER_MODE, then GPERF is also required, though it currently
doesn't actually do anything. TODO!
I found that the autotools build system was generating the lexer output
but not actually compiling it, instead using previously generated (and
manually renamed) lexer c source. As a consequence, changes to the .l
and .y files weren't making it into the build. To resolve this, I
removed generated flex/bison files and fixed the tooling to use the
freshly generated files. Flex and bison are now required build tools.
On Windows, this adds a dependency on the winflexbison package,
which can be obtained using Chocolatey or may be manually installed.
CMake tooling only has partial support for building with external LLVM
library, and no support for the internal LLVM (to be removed in the
future). I.e. The CMake build currently only supports the bytecode
interpreter.
Many files used include paths relative to the top source directory or
relative to the current project, rather than relative to each build
target. Modern CMake support requires including internal dependency
headers the same way you would external dependency headers (albeit
with "" instead of <>). This meant correcting all header includes to
be relative to the build targets and not relative to the workspace.
For example, ...
```c
include "../libclamav/clamav.h"
include "clamd/clamd_others.h"
```
... becomes:
```c
// libclamav
include "clamav.h"
// clamd
include "clamd_others.h"
```
Fixes header name conflicts by renaming a few of the files.
Converted the "shared" code into a static library, which depends on
libclamav. The ironically named "shared" static library provides
features common to the ClamAV apps which are not required in
libclamav itself and are not intended for use by downstream projects.
This change was required for correct modern CMake practices but was
also required to use the automake "subdir-objects" option.
This eliminates warnings when running autoreconf which, in the next
version of autoconf & automake are likely to break the build.
libclamav used to build in multiple stages where an earlier stage is
a static library containing utils required by the "shared" code.
Linking clamdscan and clamdtop with this libclamav utils static lib
allowed these two apps to function without libclamav. While this is
nice in theory, the practical gains are minimal and it complicates
the build system. As such, the autotools and CMake tooling was
simplified for improved maintainability and this feature was thrown
out. clamdtop and clamdscan now require libclamav to function.
Removed the nopthreads version of the autotools
libclamav_internal_utils static library and added pthread linking to
a couple apps that may have issues building on some platforms without
it, with the intention of removing needless complexity from the
source. Kept the regular version of libclamav_internal_utils.la
though it is no longer used anywhere but in libclamav.
Added an experimental doxygen build option which attempts to build
clamav.h and libfreshclam doxygen html docs.
The CMake build tooling also may build the example program(s), which
isn't a feature in the Autotools build system.
Changed C standard to C90+ due to inline linking issues with socket.h
when linking libfreshclam.so on Linux.
Generate common.rc for win32.
Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef
from misc.c.
Add CMake files to the automake dist, so users can try the new
CMake tooling w/out having to build from a git clone.
clamonacc changes:
- Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other
similar macros.
- Added a new clamav-clamonacc.service systemd unit file, based on
the work of ChadDevOps & Aaron Brighton.
- Added missing clamonacc man page.
Updates to clamdscan man page, add missing options.
Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use
libclamav).
Rename Windows mspack.dll to libmspack.dll so all ClamAV-built
libraries have the lib-prefix with Visual Studio as with CMake.
Val Snyder