Some builds using the tarball with the vendored Rust dependencies are
failing.
The onenote-rs dependency is presently tied to a git branch from github
rather than using a release from crates.io. This is the differing factor
though I'm unsure why it is causing the build to try to update the repo
rather than just building the vendored source.
This commit adds a `--offline` parameter to the build options if the
vendored source is detected, in an attempt to force Cargo to use what it
has and stay offline.
In order to generate Rust bindings for C code, Rust's bindgen module
needs to know where to find all headers included by the API.
If they're all inside the project or inside the standard include path
(e.g. /usr/include and /usr/local/include) that's fine. But for third-
party C library headers from outside the standard include path, that's
a problem.
We didn't really notice this problem when generating on Unix systems
until we switched to use OpenSSL 3.1 and tested on systems that have
the OpenSSL 1.1.1 dev package installed.
The ability to find headers outside the project path is also needed to
generate bindings on Windows, if desired.
This commit solves the problem by passing include directories for the
ClamAV::libclamav CMake build target to the Rust build via the
CARGO_INCLUDE_DIRECTORIES environment variable.
Then, in the `libclamav_rust/build.rs` script, where we run bindgen,
we split that `;` separated string into invididual paths and add each
to the bindgen builder.
Some change in Cargo/Rust version 1.70 or 1.71 appears to have broken
the build on Windows because we are incorrectly attempting to check the
native static libraries by compiling an empty file (/dev/null) which
does not exist on Windows.
A simple fix is to make an empty file of our own and use that instead.
Fixes: https://github.com/Cisco-Talos/clamav/issues/990
When switching to openssl 3.x, linking with clamsubmit fails with
undefined openssl symbols. The error message from Xcode is crazy obtuse:
ld: initializer '_OPENSSL_cpuid_setup' is >4GB from start of image in 'anon' from /Users/clamav_jenkins_svc/clamav-mussels-cookbook/test/install-x86_64/lib/libcrypto.a(libcrypto-lib-x86_64cpuid.o)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Anyhow... It seems that we must explicitly link clamsubmit with openssl
now in order for this to work.
In addition to this change, I also found that the CMake FindRust.cmake
module breaks the ability to build specifically for just x86_64 or arm64
(i.e. possibly cross-compiling.
This commit includes a change to accommodate that scenario.
A user building clamav may now set the RUSTFLAGS CMake variable
if additional options such as `--verbose` are desired.
E.g.
cmake .. -D RUSTFLAGS="--verbose"
See `rustc --help` for more details.
Building with these will fail for Clang because _GNU_SOURCE is not
defined.
-Werror=implicit-function-declaration
-Werror=implicit-int
-Werror=strict-prototypes
Also added fix to use "MATCHES" instead of "EQUALS" in FindLLVM where
the same sort of logic is used.
ClamAV requires non-default build options for TomsFastMath to support
bigger floating point numbers. Without this change, database and
Windows EXE/DLL authenticode certificate validation may fail.
The option to build with an external TomsFastMath library should be
removed. The `ENABLE_EXTERNAL_TOMSFASTMATH` build should be ignored,
with some sort of warning.
This commit removes the option. If used, the build will not fail, but
CMake will print a warning that the unused option was ignored.
The CheckFmapFeatures module is failing because HAVE_SYS_STAT_H and
HAVE_SYS_TYPES_H are not defined within the check_c_source_compiles
environment.
This commit adds them in the same way as is done for CheckFDPassing
module.
We already set the default build type to RelWithDebInfo for CMake, but
we were setting it *after* adding the Rust module. We need to do it
before, or else the Rust stuff will still default to Debug, which makes
for really slow scans of images that get fuzzy hashed.
Precompiling the test executable causes `sudo make install` to fail.
The issue is that because we don't know the OUTPUT file path for the
test executable, CMake can't know if it doesn't need to rerun that
command, so it will run it with any call to `make`. The problem is that
cargo is will fail to run with `sudo` on many systems.
This is generally okay, because we just fixed the issue where we had
been compiling the dependencies for each module. Now that we only build
the dependencies once, building the test executable is actually very
fast. So compiling it during the test isn't so bad.
Right now, each Rust-based target added in CMake is being built in its
own directory under the build path. This causes Rust to build each
module from scratch, meaning any dependencies they have in common are
built twice.
The solution in this commit is to specify the top level build directory
as the target directory for every Rust build and test.
Note this also changes where the `clamav_rust.h` file is generated. It
is now also placed in the top-level build directory, instead of under the
`build/libclamav_rust` directory. That's a bit of a side-effect, and
could be rectified if needed, but it appears to have no ill-effects.
It's the same location that we drop the clamav-types.h file, so I think
it's fine, for now. Note that `clamav_rust.h` is not a public header,
it's just so libclamav functions can call into libclamav_rust functions.
Precompile the test executable during the main build, after libclamav
has built. This will make it so the compile time does not count against
the test time.
It also, unfortunately, make the main build take way longer.
Removed 3 duplicate variables from the test ENVIRONMENT variable.
The FindRust.cmake module is hard-coded to expect CMAKE_OSX_ARCHITECTURES
value of "arm64,x86_64" to build a Rust-based Apple universal binary.
If the build uses "x86_64,arm64", which is equivalent, then the build will
fail. Order should not matter.
Modified bytecode JIT code to support llvm versions 8, 9, 10, 11, 12.
Additionally updated FindLLVM.cmake to the current version, found at
https://github.com/ldc-developers/ldc/blob/master/cmake/Modules/FindLLVM.cmake,
as well as making modifications suggested by Micah Snyder to check VERSION_LESS
in CMakeLists.txt to avoid having modifications to FindLLVM.cmake.
The current method of trying to determine the target triple based
the architecture, operating system, etc. is difficult to get right
and maintain. In particular, I found that some installations like the
Alpine package will use "alpine" in the name of the triple instead of
"unknown".
E.g. "aarch64-alpine-linux-musl" instead of "aarch64-unknown-linux-musl".
This makes it nearly impossible to figure out the exact target name
based on target system specifications.
I believe it will be better to use the default target 99% of the time
and require users that which to cross-compile to use a new CMake
variable `-D RUST_TARGET_TRIPLE=<target>` to choose the target.
This is basically how it works for C/C++ anyways.
FreeBSD and OpenBSD package maintainers identified failures when
pkg-config .pc files not present for curses/ncurses.
Patch courtesy of Stuart Henderson.
Add support for compiling with external TomsFastMath library provided by
the system instead of compiling the vendored copy into libclamav.
The vendored source is still built directly into libclamav instead of as
a separate library the way libmspack is done.
The rationale is that:
A) it's more complicated to deal with possibly compiling as static or
dynamic, and also
B) libmspack and libunrar are compiled separately primarily because of
licensing concerns. TomsFastMath public domain, so that isn't a concern.
Resolves: https://bugzilla.clamav.net/show_bug.cgi?id=12562
RelWithDebInfo is our preferred build type. It has optimizations and
should run faster, but includes debugging symbols for better profiling,
stack traces, etc.
The Rust MinSizeRel support is just Release mode for now. There are
optimizations we can do to shrink it further, but for now at least it
won't actually be Debug (aka slow).
In testing, I found that libclang.so/clang.dll is required by bindgen
and was not found on all of our test machines. To resolve this we will
only generate sys.rs bindings when CMake MAINTAINER_MODE option is "ON".
This is unfortunate that we have to commit generated source to version
control. But as a benefit it makes rust-analyzer happier when editing a
workspace that hasn't yet been compiled. And it makes it more reasonable
that the generated sys.rs file generated to the source directory and not
the build directory (something we hadn't resolved yet).
Add a basic unit test for the new libclamav_rust `logging.rs` module.
This test simply initializes logging and then prints out a message with
each of the `log` macros.
Also set the Rust edition to 2018 because the default is the 2015
edition in which using external crates is very clunky.
For the Rust test support in CMake this commit adds support for
cross-compiling the Rust tests.
Rust tests must be built for the same LLVM triple (target platform) as
the rest of the project. In particular this is needed to build both
x64 and x86 packages on a 64bit Windows host.
For Alpine, we observed that the LLVM triple for the host platform tools
may be either:
- x86_64-unknown-linux-musl, or
- x86_64-alpine-linux-musl
To support it either way, we look up the host triple with `rustc -vV`
and use that if the musl libc exists. This is a big hacky and
unfortunately means that we probably can't cross-compile to other
platforms when running on a musl libc host. There are probably
improvements to be made to improve cross compiling support.
The Rust test programs must link with libclamav, libclammspack, and
possibly libclamunrar_iface and libclamunrar plus all of the library
dependencies for those libraries.
To do this, we pass the path of each library in environment variables
when building the libclamav_rust unit test program.
Within `libclamav_rust/build.rs`, we read those environment variables.
If set, we parse each into library path and name components to use
as directives for how to build the unit test program.
See: https://doc.rust-lang.org/cargo/reference/build-scripts.html
Our `build.rs` file ignores the library path environment variables if
thye're not set, which is necessary when building the libclamav_rust
library and when libclamunrar isn't static and for when not linking with
a libiconv external to libc.
Rust test programs are built and executed in subdirectory under:
<target>/<llvm triple>/<config>/deps
where "target" for libclamav_rust tests is set to <build>/unit_tests
For example:
clamav/build/unit_tests/x86_64-pc-windows-msvc/debug/deps/clamav_rust-7e1343f8a2bff1cc.exe
Since this program isn't co-located with the rest of the libraries
we also have to set environment variables so the test program can find and
load the shared libraries:
- Windows: PATH
- macOS: DYLD_LIBRARY_PATH
We already set LD_LIBRARY_PATH when not Windows for similar reasons.
Note: In build.rs, we iterate references to LIB_ENV_LINK & Co because
older Rust versions do implement Iterator for [&str].
Add support for Rust FreeBSD, OpenBSD targets
Add support for Rust on GNU aarch64.
Add support for Rust on Alpine (musl x86_64).
Note: Current trick of checking for musl libc.so doesn't work when
cross compiling. TODO: Find a better way to check if the target is
MUSL.
Add Rust toolchain to fix Dockerfile build.
Vendoring crate dependencies is required for offline builds.
Some packaging systems require that builds can be performed offline.
This feature enabled vendoring crates at configure time which are
then included in in CPack source packaging.
If ncurses or pdcurses are static libraries, they are not properly
detected.
First, the user compiling clamav needs to specify if the include path is
for NCURSES or PDCURSES, which will differentiate the two. I've updated
the INSTALL.md file to show this.
Second, the wrong variable was being used to add the include path to the
Curses::curses target, which means that clamdtop would fail to include
ncurses.h. I fixed this.
CMake/CPack is already used to build:
- TGZ source tarball
- WiX-based installer (Windows)
- ZIP install packages (Windows)
This commit adds support for building:
- macOS PKG installer
- DEB package
- RPM package
This should also enable building FreeBSD packages, but while I was able
to build all of the static dependencies using Mussels, CMake/CPack 3.20
doesn't appear to have the the FreeBSD generator despite being in the
documentation.
The package names are will be in this format:
clamav-<version><suffix>.<os>.<arch>.<extension>
This includes changing the Windows .zip and .msi installer names.
E.g.:
- clamav-0.104.0-rc.macos.x86_64.pkg
- clamav-0.104.0-rc.win.win32.msi
- clamav-0.104.0-rc.win.win32.zip
- clamav-0.104.0-rc.win.x64.msi
- clamav-0.104.0-rc.linux.x86_64.deb
- clamav-0.104.0-rc.linux.x86_64.rpm
Notes about building the packages:
I've only tested this with building ClamAV using static dependencies that
I build using the clamav_deps "host-static" recipes from the "clamav"
Mussels cookbook. Eg:
msl build clamav_deps -t host-static
Here's an example configuration to build clam in this way, installing to
/usr/local/clamav:
```sh
cmake .. \
-D CMAKE_FIND_PACKAGE_PREFER_CONFIG=TRUE \
-D CMAKE_PREFIX_PATH=$HOME/.mussels/install/host-static \
-D CMAKE_INSTALL_PREFIX="/usr/local/clamav" \
-D CMAKE_MODULE_PATH=$HOME/.mussels/install/host-static/lib/cmake \
-D CMAKE_BUILD_TYPE=RelWithDebInfo \
-D ENABLE_EXAMPLES=OFF \
-D JSONC_INCLUDE_DIR="$HOME/.mussels/install/host-static/include/json-c" \
-D JSONC_LIBRARY="$HOME/.mussels/install/host-static/lib/libjson-c.a" \
-D ENABLE_JSON_SHARED=OFF \
-D BZIP2_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D BZIP2_LIBRARY_RELEASE="$HOME/.mussels/install/host-static/lib/libbz2_static.a" \
-D OPENSSL_ROOT_DIR="$HOME/.mussels/install/host-static" \
-D OPENSSL_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D OPENSSL_CRYPTO_LIBRARY="$HOME/.mussels/install/host-static/lib/libcrypto.a" \
-D OPENSSL_SSL_LIBRARY="$HOME/.mussels/install/host-static/lib/libssl.a" \
-D LIBXML2_INCLUDE_DIR="$HOME/.mussels/install/host-static/include/libxml2" \
-D LIBXML2_LIBRARY="$HOME/.mussels/install/host-static/lib/libxml2.a" \
-D PCRE2_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D PCRE2_LIBRARY="$HOME/.mussels/install/host-static/lib/libpcre2-8.a" \
-D CURSES_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D CURSES_LIBRARY="$HOME/.mussels/install/host-static/lib/libncurses.a" \
-D ZLIB_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D ZLIB_LIBRARY="$HOME/.mussels/install/host-static/lib/libz.a" \
-D LIBCHECK_INCLUDE_DIR="$HOME/.mussels/install/host-static/include" \
-D LIBCHECK_LIBRARY="$HOME/.mussels/install/host-static/lib/libcheck.a"
```
Set CPACK_PACKAGING_INSTALL_PREFIX to customize the resulting package's
install location. This can be different than the install prefix. E.g.:
```sh
-D CMAKE_INSTALL_PREFIX="/usr/local/clamav" \
-D CPACK_PACKAGING_INSTALL_PREFIX="/usr/local/clamav" \
```
Then `make` and then one of these, depending on the platform:
```sh
cpack # macOS: productbuild is default
cpack -G DEB # Debian-based
cpack -G RPM # RPM-based
```
On macOS you'll need to `pip3 install markdown` so that the NEWS.md file can
be converted to html so it will render in the installer.
On RPM-based systems, you'll need rpmbuild (install rpm-build)
This commit also fixes an issue where the html manual (if present) was
not correctly added to the Windows (or now other) install packages.
Fix num to hex function for Windows installer guid
Fix win32 cpack build
Fix macOS cpack build
In openSUSE Tumbleweed, this test always fails because it compiles with `-Werror=return-type` by default. Fixing this by adding a return value in the test script to keep the compiler happy.
I'd foolishly set the Iconv detection module to use uppercase to
differentiate it from the upstream (original) module. I also kind of
forgot to change the documentation to use the uppercase name. In
reviewing that, I decided it's better to match the upstream
FindIconv.cmake module as closely as possible in case they fix the issue
described here, so that we can stop carrying a custom module without
breaking the CMake options for everyone:
b26cdc5453
This reverts the CMake options for setting the Iconv include and library
paths back to match the upstream before 0.104 ships with stable CMake
support.
The fuzz targets should use malloc instead of mmap because libFuzzer
doesn't seem to track mmap leaks.
The ENABLE_FUZZ feature disables HAVE_MMAP which is supposed to
disable use of mmap in favor of malloc, but it doesn't because
checks for the ANONYMOUS_MAP feature execute even if HAVE_MMAP
is disabled, and some mmap usage in libclamav only tests if
ANONYMOUS_MAP is defined without checking HAVE_MMAP.
This commit will only define ANONYMOUS_MAP if HAVE_MMAP is set.
This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673
Changes in 0.103 to order of operations for creating fmaps and
performaing hashes of fmaps resulted errors when scanning files that are
4096M and a different (but related) error when scanning files > 4096M.
This is despite the fact that scanning is supposed to be limited to
--max-scansize (MaxScanSize) and was also apparently limited to
INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes
for a few large file formats.
(see https://bugzilla.clamav.net/show_bug.cgi?id=10960)
This last limitation was not documented, so I added it to the sample
clamd.conf.
Anyways, the main issue is that the fmap module was using "unsigned int"
and was then enforcing a limitation (verbose error messages) when that
a map length exceeded the capapacity of an unsigned int. This commit
switches the associated variables over to uint64_t, and while fmaps are
still limited to size_t in other places, the fmap module will at least
work with files > 4G on 64bit systems.
In testing this, I found that the time to hash a file, particularly when
hashing a file on an NTFS partition from Linux was really slow because
we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing
this to 10MB chunks speeds up scanning of large files.
Finally, now that hashing is performed immediately when an fmap is
created for a file, hashing of files larger than max-scansize was
occuring. This commit adds checks to bail out early if the file size
exceeds the maximum before creating an fmap. It will alert with the
Heuristics.Limits.Exceeded name if the heuristic is enabled.
Also fixed CheckFmapFeatures.cmake module that detects if
sysconf(_SC_PAGESIZE) is available.
Adds support to the pcre2 and pthreadw32 Find<Package>.cmake modules for
correctly discovering the debug versions. This change modeled after the
upstream FindBZip2.cmake module.
Also eliminated HAVE_STRUCT_TIMESPEC redefinition warnings in Windows
builds.
Updates to fix issues in the CMake install instructions.
Updates the README.md to indicate that CMake is now preferred
Adds a GitHub Actions badge, Discord badge, and logo to the README.md.
CMake:
- Renamed ENABLE_DOCS to ENABLE_MAN_PAGES.
- Fixed build issue when milter isn't enabled on Linux. Changed the
default to build milter on non-macOS, non-Windows operating systems.
- Fix LD_LIBRARY_PATH for tests including on macOS where LD_LIBRARY_PATH
and DYLD_LIBRARY_PATH must be manually propagated to subprocesses.
- Use UNKNOWN IMPORTED library instead of INTERFACE IMPORTED library for
pdcurses, but still use INTERFACE IMPORTED for ncurses.
UNKNOWN IMPORTED appears to be required so that we can use
$<TARGET_FILE_DIR:Curses::curses> to collected the pdcurses library at
install time on Windows.
- When building with vcpkg on Windows, CMake will automatically install
your app local dependencies (aka the DLL runtime dependencies).
Meanwhile, file(GET_RUNTIME_DEPENDENCIES ...) doesn't appear to work
correctly with vcpkg packages. The solution is to use a custom target
that has CMake perform a local install to the unit_tests directory
when using vcpkg.
This is in fact far easier than using GET_RUNTIME_DEPENDENCIES in the
unit_tests for assembling the test environment but we can't use this
method for the non-vcpkg install because it won't collect
checkDynamic.dll for us because we don't install our tests.
We also can't link with the static check.lib because the static
check.lib has pthreads symbols linked in and will conflict with our
pthread.dll.
TL;DR: We'll continue to use file(GET_RUNTIME_DEPENDENCIES ...) for
assembling the test enviornment on non-vcpkg builds, and use the local
install method for vcpkg builds.
testcase.py: Wrapped a Pathlib.unlink() call in exception handling as
the missing_ok optional parameter requires a Python version too new for
common use.
Remove localtime_r from win32 compat lib.
localtime_r may be present in libcheck when building with vcpkg and
while making it a static function would also solve the issue, using
localtime_s instead like we do everywhere else should work just fine.
check_clamd: Limited the max # of connections for the stress test on Mac
to 850, to address issues found testing on macos-latest on GitHub Actions.
liushuyu