Building with these will fail for Clang because _GNU_SOURCE is not
defined.
-Werror=implicit-function-declaration
-Werror=implicit-int
-Werror=strict-prototypes
Also added fix to use "MATCHES" instead of "EQUALS" in FindLLVM where
the same sort of logic is used.
The libclamav SO ABI has changed since 0.103 due to:
- Removal of the CLAMAV_PUBLIC namespace by no longer using libclamav.map
- Change of: extern const char *cl_strerror(int clerror);
To: extern const char *cl_strerror(cl_error_t clerror);
See: e66c00f41c/libclamav/clamav.h (L1203)
- Introduction of Rust symbols to namespace.
Ref: https://github.com/Cisco-Talos/clamav/issues/775
As such, it seems fair to increase the SO version.
This is an amendment to this prior change for the ClamAV 1.0.0
version bump:
0dd6ab3f16
Instead of incrementing SO Current version to 11, and Age to 2,
we'll increment SO Current version to 11, and reset Age to 0.
So the SO version will effectively go from:
9.1.0
to:
11.0.0
ClamAV requires non-default build options for TomsFastMath to support
bigger floating point numbers. Without this change, database and
Windows EXE/DLL authenticode certificate validation may fail.
The option to build with an external TomsFastMath library should be
removed. The `ENABLE_EXTERNAL_TOMSFASTMATH` build should be ignored,
with some sort of warning.
This commit removes the option. If used, the build will not fail, but
CMake will print a warning that the unused option was ignored.
In preparation for the release candidate, changing the version suffix
from devel to rc.
Increment SO current version to 11, because we added to the public API.
Increment SO age to 2, because we didn't change anything or remove
anything from the public API.
Added functionality level values to the bytecode API enum for recent
patch versions.
The bytecode source files largely use `int` instead of the appropriate
`cl_errot_t` for clamav status codes, as well for boolean variables.
This hides warnings that would indicate bugs, and makes it harder to
read the code.
I haven't gone as in depth as with some other code cleanups. This
largely just replaces function interfactes and ret variables that use
`int` with `cl_error_t`. I also swapped a couple of `int`s to `bool`s.
While doing so I found that the `cli_bytecode_context_setpdf()` function
was incorrectly placed in the `bytecode_api.c` file instead of the next
to similar functions (`cli_bytecode_context_setpe`, etc.) in bytecode.c.
It's not an API function, so I moved it to the correct location.
I also eliminated a couple of compiler warnings:
- LLVM's CFG.h header emits a warning about a multi-line comment, so
that crops up with using LLVM for the bytecode runtime.
I disabled the warning through CMake.
- C doesn't like using the `inline` keyword on cli_dbgmsg in the
declaration in `bytecode2llvm.c` because we're compiling the bytecode
runtimes as a separate object file from the rest of libclamav.
It doesn't appear to be a functional issue, but I swapped that file
over to use `cli_dbgmsg_no_inline()` instead, just in case.
I would hope link-time-optimization will inline it anyways.
A copy-paste error in the markdown support for building the macOS-
installer caused an issue where markdown module may be accidentally
used instead of pytest. Strangely, this still worked okay up until we
converted the clamscan tests into a directory of tests, at which point
the `python3 -m markdown` invocation was upset that it was trying to
execute a directory instead of a .py file.
This commit removes the line that overwrites the python test command
with using the markdown module.
We already set the default build type to RelWithDebInfo for CMake, but
we were setting it *after* adding the Rust module. We need to do it
before, or else the Rust stuff will still default to Debug, which makes
for really slow scans of images that get fuzzy hashed.
CMake has a pretty sweet alternative to pkg-config or to the older
`package-config` script that you'd run with `--libs`, `--ldflags`, etc.
CMake can (optionally) install a YourPackageConfig.cmake alongside your
libs under `<prefix>/lib/cmake/<pkg>/<pkg>Config.cmake`.
If you build something with -D CMAKE_FIND_PACKAGE_PREFER_CONFIG=TRUE,
then it will find that and use that to bring real CMake targets into
your build system for your dependencies, guaranteeing that you get all
the right include paths, library paths, ldflags, etc.
See: https://cmake.org/cmake/help/latest/variable/CMAKE_FIND_PACKAGE_PREFER_CONFIG.html
This works great for us for Curl and LLVM... not so much for libxml2.
Ideally we'd figure out what's wrong and support libxml2's package
config.cmake file, but for now this workaround lets us ignore libxml2
and continu to use this feature for the other libs, like LLVM.
We must pass the LLVM library dependencies to the libclamav_rust
build.rs script so it links the libclamav_rust unit test executable with
LLVM.
Also:
- We can remove the libtinfo dependency that was hardcoded for the LLVM
3.6 support, and must remove it for the build to work on Alpine, macOS.
- Also, increased the libcheck default timeout from 60s to 300s after
experiencing a failure while testing this.
- Also made one of the valgrind suppressions more generic to account for
inline optimization differences observed in testing this.
Modified bytecode JIT code to support llvm versions 8, 9, 10, 11, 12.
Additionally updated FindLLVM.cmake to the current version, found at
https://github.com/ldc-developers/ldc/blob/master/cmake/Modules/FindLLVM.cmake,
as well as making modifications suggested by Micah Snyder to check VERSION_LESS
in CMakeLists.txt to avoid having modifications to FindLLVM.cmake.
Add support for compiling with external TomsFastMath library provided by
the system instead of compiling the vendored copy into libclamav.
The vendored source is still built directly into libclamav instead of as
a separate library the way libmspack is done.
The rationale is that:
A) it's more complicated to deal with possibly compiling as static or
dynamic, and also
B) libmspack and libunrar are compiled separately primarily because of
licensing concerns. TomsFastMath public domain, so that isn't a concern.
Resolves: https://bugzilla.clamav.net/show_bug.cgi?id=12562
Vendoring crate dependencies is required for offline builds.
Some packaging systems require that builds can be performed offline.
This feature enabled vendoring crates at configure time which are
then included in in CPack source packaging.
To build with code signing, the macOS build must have:
-G Xcode \
-D CLAMAV_SIGN_FILE=ON \
-D CODE_SIGN_IDENTITY="...your codesign ID..." \
-D DEVELOPMENT_TEAM_ID="...your team ID..." \
You can find the codesign ID using:
/usr/bin/env xcrun security find-identity -v -p codesigning
The team ID should also be listed in the identity description.
Also I changed the package name for APPLE to be "clamav" so it doesn't
put "ClamAV <version>" in the PKG PackageInfo like this:
com.cisco.ClamAV 0.104.0.libraries
Instead, it should just be something like:
com.cisco.clamav.libraries
Version is a separate field in that file and shouldn't be in the name.
At present the .msi installer is only installing documentation component
files and the vcredist files but fails to install clamav libraries,
programs, and dependencies.
It appears that explicitly installing the NEWS & README files under the
documentation component before calling "include(CPack)" was causing the
MSI installer to think it needed to install the documentation component
but nothing else.
This commit removes the component name, since we don't want to use
components in the Windows MSI installer anyways. This appears to resolve
the issue so that the MSI installer installs all the desired files.
For Windows to match 0.103 installer behavior, include NEWS.md and
README.md and rename the html directory to UserManual during the
install.
Unfortunately I can't match the behavior for the main page for the
user manual. It is now called index.html instead of UserManual.html
and is inside the UserManual directory instead of at the top level.
Xcode (and perhaps some other generators?) do not like targets that have
only object files. See:
https://cmake.org/cmake/help/latest/command/add_library.html#object-libraries
And: https://cmake.org/pipermail/cmake/2016-May/063479.html
This issue manifests when using `-G Xcode` on macOS as the library
dylibs being missing when linking with other binaries.
This commit removes the object libraries for libclamav, libfreshclam,
libclamunrar_iface, libclamunrar, libclammspack, and (lib)common
because they were used by static or shared libs that didn't
themselves have any added sources.
Add getter & setter for the debug flag, so it isn't referenced by unit
tests or other code that links with libclamav. This is needed because
global variables are exported symbols on Windows.
CMake/CPack is already used to build:
- TGZ source tarball
- WiX-based installer (Windows)
- ZIP install packages (Windows)
This commit adds support for building:
- macOS PKG installer
- DEB package
- RPM package
This should also enable building FreeBSD packages, but while I was able
to build all of the static dependencies using Mussels, CMake/CPack 3.20
doesn't appear to have the the FreeBSD generator despite being in the
documentation.
The package names are will be in this format:
clamav-<version><suffix>.<os>.<arch>.<extension>
This includes changing the Windows .zip and .msi installer names.
E.g.:
- clamav-0.104.0-rc.macos.x86_64.pkg
- clamav-0.104.0-rc.win.win32.msi
- clamav-0.104.0-rc.win.win32.zip
- clamav-0.104.0-rc.win.x64.msi
- clamav-0.104.0-rc.linux.x86_64.deb
- clamav-0.104.0-rc.linux.x86_64.rpm
Notes about building the packages:
I've only tested this with building ClamAV using static dependencies that
I build using the clamav_deps "host-static" recipes from the "clamav"
Mussels cookbook. Eg:
msl build clamav_deps -t host-static
Here's an example configuration to build clam in this way, installing to
/usr/local/clamav:
```sh
cmake .. \
-D CMAKE_FIND_PACKAGE_PREFER_CONFIG=TRUE \
-D CMAKE_PREFIX_PATH=$HOME/.mussels/install/host-static \
-D CMAKE_INSTALL_PREFIX="/usr/local/clamav" \
-D CMAKE_MODULE_PATH=$HOME/.mussels/install/host-static/lib/cmake \
-D CMAKE_BUILD_TYPE=RelWithDebInfo \
-D ENABLE_EXAMPLES=OFF \
-D JSONC_INCLUDE_DIR="$HOME/.mussels/install/host-static/include/json-c" \
-D JSONC_LIBRARY="$HOME/.mussels/install/host-static/lib/libjson-c.a" \
-D ENABLE_JSON_SHARED=OFF \
-D BZIP2_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D BZIP2_LIBRARY_RELEASE="$HOME/.mussels/install/host-static/lib/libbz2_static.a" \
-D OPENSSL_ROOT_DIR="$HOME/.mussels/install/host-static" \
-D OPENSSL_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D OPENSSL_CRYPTO_LIBRARY="$HOME/.mussels/install/host-static/lib/libcrypto.a" \
-D OPENSSL_SSL_LIBRARY="$HOME/.mussels/install/host-static/lib/libssl.a" \
-D LIBXML2_INCLUDE_DIR="$HOME/.mussels/install/host-static/include/libxml2" \
-D LIBXML2_LIBRARY="$HOME/.mussels/install/host-static/lib/libxml2.a" \
-D PCRE2_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D PCRE2_LIBRARY="$HOME/.mussels/install/host-static/lib/libpcre2-8.a" \
-D CURSES_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D CURSES_LIBRARY="$HOME/.mussels/install/host-static/lib/libncurses.a" \
-D ZLIB_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D ZLIB_LIBRARY="$HOME/.mussels/install/host-static/lib/libz.a" \
-D LIBCHECK_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D LIBCHECK_LIBRARY="$HOME/.mussels/install/host-static/lib/libcheck.a"
```
Set CPACK_PACKAGING_INSTALL_PREFIX to customize the resulting package's
install location. This can be different than the install prefix. E.g.:
```sh
-D CMAKE_INSTALL_PREFIX="/usr/local/clamav" \
-D CPACK_PACKAGING_INSTALL_PREFIX="/usr/local/clamav" \
```
Then `make` and then one of these, depending on the platform:
```sh
cpack # macOS: productbuild is default
cpack -G DEB # Debian-based
cpack -G RPM # RPM-based
```
On macOS you'll need to `pip3 install markdown` so that the NEWS.md file can
be converted to html so it will render in the installer.
On RPM-based systems, you'll need rpmbuild (install rpm-build)
This commit also fixes an issue where the html manual (if present) was
not correctly added to the Windows (or now other) install packages.
Fix num to hex function for Windows installer guid
Fix win32 cpack build
Fix macOS cpack build
In 0.104.0 we added new load/compile/free progress callback APIs to
clamav.h
This is a backwards compatible change, so we're bumping the current and
age fields, and resetting the revision.
See http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html
for more info about libtool style .so versioning.
The split test files are flagged by some AV's because they look like
broken executables. Instead of splitting the test files to prevent
detections, we should encrypt them. This commit replaces the "reassemble
testfiles" script with a basic "XOR testfiles" script that can be used
to encrypt or decrypt test files. This commit also of course then
replaces all the split files with xor'ed files.
The test and unit_tests directories were a bit of a mess, so I
reorganized them all into unit_tests with all of the test files placed
under "unit_tests/input" using subdirectories for different types of files.
Have to manually link libtinfo (`-ltinfo`) because our FindLLVM
didn't add it to the LLVM_LIBRARIES variable for us. See:
- https://stackoverflow.com/questions/21477407/llvm-3-5-fails-to-link
Have to remove the CXX_STANDARD setting at the top of CMakeLists.txt
because of c++90 / c++11 ABI compatibility issues w/ LLVM. See:
- https://maleadt.github.io/LLVM.jl/dev/man/troubleshooting/
Rename "llvm/Config/config.h" "llvm/Config/llvm-config.h" because
LLVM renamed it in 2.8.
Have to link LLVM manually with the test binaries that use the
clamav object library instead of libclamav shared library.
CMake does not propagate library dependencies from object files.
I tested on ubuntu:16.04 with LLVM 3.6 built from source using:
```
/usr/local/bin/cmake .. -D CMAKE_INSTALL_PREFIX=/opt/llvm/3.6 \
-D LLVM_ENABLE_RTTI=ON
```
Then built clamav w/:
```
/usr/local/bin/cmake .. -D CMAKE_INSTALL_PREFIX=`pwd`/install \
-D BYTECODE_RUNTIME="llvm" \
-D LLVM_ROOT_DIR="/opt/llvm/3.6" \
-D LLVM_FIND_VERSION="3.6.0" && make && make install
```
Add options to build a source tarball.
This will replace Autotools' `make dist` feature.
Although CMake doesn't have an autogen step, this is still necessary if
we want to bundle in a local copy of the online documentation or if we
want to vendor dependencies from github/etc at configure time in the tarball.
I'd foolishly set the Iconv detection module to use uppercase to
differentiate it from the upstream (original) module. I also kind of
forgot to change the documentation to use the uppercase name. In
reviewing that, I decided it's better to match the upstream
FindIconv.cmake module as closely as possible in case they fix the issue
described here, so that we can stop carrying a custom module without
breaking the CMake options for everyone:
b26cdc5453
This reverts the CMake options for setting the Iconv include and library
paths back to match the upstream before 0.104 ships with stable CMake
support.
If pytest is available in the path, but wasn't found in the Python 3
installation discovered by CMake, then the test suite will fail.
This can happen when two different Python 3 installations exist, eg 3.6
and 3.7 and when the newest one (prefered by CMake) doesn't have pytest.
This commit will use the correct command depending on where pytest was
(or wasn't) found. It will also fail more gracefully if neither pytest
nor unittest was found (this happens with some wierd python installs).
As pytest can be installed in many ways, we can check for pytests
existence next to if it was pip-installed.
Signed-off-by: Olliver Schinagl <oliver@schinagl.nl>
There are some merits of having the configs in `/etc/clamav` and the databases
in `/var/lib/clamav` to match the way the distros all package it and so these
directories can be mounted volumes if needed.
The DATABASE_DIRECTORY and APP_CONFIG_DIRECTORY variables don't support
absolute paths. This patch changes that.
The named "shared" is confusing, especially now that these features are
built as a static library instead of being directly compiled into the
various applications.
Also add support for using pytest instead of python's unittest to make
it easier to find & read failed test results.
Clean up the log output in check_clamd when printing "wrong reply"
results.
Switch from using collections.namedtuple() to defining our own
NamedTuple subclass, which adds member variable typing.
Micah Snyder