I found that on macOS, setting the libncurses.a and libtinfo.a into the
same CURSES_LIBRARY variable does not find or link with libtinfo.a.
To fix this, this commit adds a separate TINFO_LIBRARY variable.
In the end, CURSES_LIBRARIES and the Curses::curses CMake TARGET will
still have both libraries set, if both are provided.
This fix is necessary if the ncurses was built with `--with-terminfo`.
I think we got away without it on Linux because of pkg-config.
I also found that Apple's ncurses is prioritized by PkgConfig over one
specified by using variables. To this end, we'll skip PkgConfig if
the include path was provided.
Add X509 certificate chain based signing with PKCS7-PEM external
signatures distributed alongside CVD's in a custom .cvd.sign format.
This new signing and verification mechanism is primarily in support
of FIPS compliance.
Fixes: https://github.com/Cisco-Talos/clamav/issues/564
Add a Rust implementation for parsing, verifying, and unpacking CVD
files.
Now installs a 'certs' directory in the app config directory
(e.g. <prefix>/etc/certs). The install location is configurable.
The CMake option to configure the CVD certs directory is:
`-D CVD_CERTS_DIRECTORY=PATH`
New options to set an alternative CVD certs directory:
- Commandline for freshclam, clamd, clamscan, and sigtool is:
`--cvdcertsdir PATH`
- Env variable for freshclam, clamd, clamscan, and sigtool is:
`CVD_CERTS_DIR`
- Config option for freshclam and clamd is:
`CVDCertsDirectory PATH`
Sigtool:
- Add sign/verify commands.
- Also verify CDIFF external digital signatures when applying CDIFFs.
- Place commonly used commands at the top of --help string.
- Fix up manpage.
Freshclam:
- Will try to download .sign files to verify CVDs and CDIFFs.
- Fix an issue where making a CLD would only include the CFG file for
daily and not if patching any other database.
libclamav.so:
- Bump version to 13:0:1 (aka 12.1.0).
- Also remove libclamav.map versioning.
Resolves: https://github.com/Cisco-Talos/clamav/issues/1304
- Add two new API's to the public clamav.h header:
```c
extern cl_error_t cl_cvdverify_ex(const char *file,
const char *certs_directory);
extern cl_error_t cl_cvdunpack_ex(const char *file,
const char *dir,
bool dont_verify,
const char *certs_directory);
```
The original `cl_cvdverify` and `cl_cvdunpack` are deprecated.
- Add `cl_engine_field` enum option `CL_ENGINE_CVDCERTSDIR`.
You may set this option with `cl_engine_set_str` and get it
with `cl_engine_get_str`, to override the compiled in default
CVD certs directory.
libfreshclam.so: Bump version to 4:0:0 (aka 4.0.0).
Add sigtool sign/verify tests and test certs.
Make it so downloadFile doesn't throw a warning if the server
doesn't have the .sign file.
Replace use of md5-based FP signatures in the unit tests with
sha256-based FP signatures because the md5 implementation used
by Python may be disabled in FIPS mode.
Fixes: https://github.com/Cisco-Talos/clamav/issues/1411
CMake: Add logic to enable the Rust openssl-sys / openssl-rs crates
to build against the same OpenSSL library as is used for the C build.
The Rust unit test application must also link directly with libcrypto
and libssl.
Fix some log messages with missing new lines.
Fix missing environment variable notes in --help messages and manpages.
Deconflict CONFDIR/DATADIR/CERTSDIR variable names that are defined in
clamav-config.h.in for libclamav from variable that had the same name
for use in clamav applications that use the optparser.
The 'clamav-test' certs for the unit tests will live for 10 years.
The 'clamav-beta.crt' public cert will only live for 120 days and will
be replaced before the stable release with a production 'clamav.crt'.
Somehow the changes from #692 did not come with a doc update. The cap at version 13 is given from #574. I did not try to build it with LLVM 14, so I don't know whether it truly doesn't compile or simply fails due to the CMake thing.
RPATH overrides the normal library search path, possibly interfering
with local policy and causing problems for multi-lib, among other issues.
Add an option to avoid setting it with letting it enabled by default.
Signed-off-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
The supported Docker files used to build the official clamav images are
now located here: https://github.com/Cisco-Talos/clamav-docker
Also fix some minor errors in the INSTALL.md file.
ClamAV requires non-default build options for TomsFastMath to support
bigger floating point numbers. Without this change, database and
Windows EXE/DLL authenticode certificate validation may fail.
The option to build with an external TomsFastMath library should be
removed. The `ENABLE_EXTERNAL_TOMSFASTMATH` build should be ignored,
with some sort of warning.
This commit removes the option. If used, the build will not fail, but
CMake will print a warning that the unused option was ignored.
Also add release notes from 0.103 and 0.104 patch versions published
during the development of 0.105.
Also Update llvm support details in the INSTALL.md file.
The current method of trying to determine the target triple based
the architecture, operating system, etc. is difficult to get right
and maintain. In particular, I found that some installations like the
Alpine package will use "alpine" in the name of the triple instead of
"unknown".
E.g. "aarch64-alpine-linux-musl" instead of "aarch64-unknown-linux-musl".
This makes it nearly impossible to figure out the exact target name
based on target system specifications.
I believe it will be better to use the default target 99% of the time
and require users that which to cross-compile to use a new CMake
variable `-D RUST_TARGET_TRIPLE=<target>` to choose the target.
This is basically how it works for C/C++ anyways.
Add support for compiling with external TomsFastMath library provided by
the system instead of compiling the vendored copy into libclamav.
The vendored source is still built directly into libclamav instead of as
a separate library the way libmspack is done.
The rationale is that:
A) it's more complicated to deal with possibly compiling as static or
dynamic, and also
B) libmspack and libunrar are compiled separately primarily because of
licensing concerns. TomsFastMath public domain, so that isn't a concern.
Resolves: https://bugzilla.clamav.net/show_bug.cgi?id=12562
In testing, I found that libclang.so/clang.dll is required by bindgen
and was not found on all of our test machines. To resolve this we will
only generate sys.rs bindings when CMake MAINTAINER_MODE option is "ON".
This is unfortunate that we have to commit generated source to version
control. But as a benefit it makes rust-analyzer happier when editing a
workspace that hasn't yet been compiled. And it makes it more reasonable
that the generated sys.rs file generated to the source directory and not
the build directory (something we hadn't resolved yet).
If ncurses or pdcurses are static libraries, they are not properly
detected.
First, the user compiling clamav needs to specify if the include path is
for NCURSES or PDCURSES, which will differentiate the two. I've updated
the INSTALL.md file to show this.
Second, the wrong variable was being used to add the include path to the
Curses::curses target, which means that clamdtop would fail to include
ncurses.h. I fixed this.
Move all step-by-step instructions for installing dependencies to
docs.clamav.net.
INSTALL.md serves to direct folks to our online documentation (or
the offline copy in the release tarball), and as a reference for
all custom config options.
Add some introductory CMake material to help people new to CMake.
Add un-install instructions.
Also fix broken links in README.md.
Updates to prepare for the 0.104 release candidate:
- Change documentation to explain current bytecode runtime situation.
- Document Python 2 pytest issue.
- Add additional contributors to acknowledgements.
- Update Install instructions to note that Autotool has been removed.
- Add *.cat SHA256 support and PDF bytecode hook bugfix to the News.
- Clarify purpose of the clamscan `--gen-json` option in the
clamscan --help.
CMake is now required to build.
The built-in LLVM is no longer available.
Also removed support for libltdl calls, which is not used in the CMake
builds, was only used when building with Autotools.
TODO: Fix CMake LLVM support & update to work with modern versions.
Adds support to the pcre2 and pthreadw32 Find<Package>.cmake modules for
correctly discovering the debug versions. This change modeled after the
upstream FindBZip2.cmake module.
Also eliminated HAVE_STRUCT_TIMESPEC redefinition warnings in Windows
builds.
Updates to fix issues in the CMake install instructions.
Updates the README.md to indicate that CMake is now preferred
Adds a GitHub Actions badge, Discord badge, and logo to the README.md.
CMake:
- Renamed ENABLE_DOCS to ENABLE_MAN_PAGES.
- Fixed build issue when milter isn't enabled on Linux. Changed the
default to build milter on non-macOS, non-Windows operating systems.
- Fix LD_LIBRARY_PATH for tests including on macOS where LD_LIBRARY_PATH
and DYLD_LIBRARY_PATH must be manually propagated to subprocesses.
- Use UNKNOWN IMPORTED library instead of INTERFACE IMPORTED library for
pdcurses, but still use INTERFACE IMPORTED for ncurses.
UNKNOWN IMPORTED appears to be required so that we can use
$<TARGET_FILE_DIR:Curses::curses> to collected the pdcurses library at
install time on Windows.
- When building with vcpkg on Windows, CMake will automatically install
your app local dependencies (aka the DLL runtime dependencies).
Meanwhile, file(GET_RUNTIME_DEPENDENCIES ...) doesn't appear to work
correctly with vcpkg packages. The solution is to use a custom target
that has CMake perform a local install to the unit_tests directory
when using vcpkg.
This is in fact far easier than using GET_RUNTIME_DEPENDENCIES in the
unit_tests for assembling the test environment but we can't use this
method for the non-vcpkg install because it won't collect
checkDynamic.dll for us because we don't install our tests.
We also can't link with the static check.lib because the static
check.lib has pthreads symbols linked in and will conflict with our
pthread.dll.
TL;DR: We'll continue to use file(GET_RUNTIME_DEPENDENCIES ...) for
assembling the test enviornment on non-vcpkg builds, and use the local
install method for vcpkg builds.
testcase.py: Wrapped a Pathlib.unlink() call in exception handling as
the missing_ok optional parameter requires a Python version too new for
common use.
Remove localtime_r from win32 compat lib.
localtime_r may be present in libcheck when building with vcpkg and
while making it a static function would also solve the issue, using
localtime_s instead like we do everywhere else should work just fine.
check_clamd: Limited the max # of connections for the stress test on Mac
to 850, to address issues found testing on macos-latest on GitHub Actions.
Also creates a ZIP for non-Admin (per-user) installs.
WIX requires the license file to have a .txt or .rtf extension so I
added the .txt extension. I've taken the opportunity to migrate the 3rd
party licenses to a COPYING subdirectory and have added licensing
details to the README.md file.
To build the installer, install WIX and simply run `cpack -C Release`
Also removed the explicit --config option from the
clamav-clamonacc.service file because it should not be required and
isn't being generated correctly when using autotools anyways, especially
after changes in this commit.
Enabled the metadata collection feature, scan heuristics, and all-match
mode when fuzzing in the interest of better code coverage.
Also remove deprecated STREAM command.
An ENABLE_TESTS CMake option is provided so that users can disable
testing if they don't want it. Instructions for how to use this
included in the INSTALL.cmake.md file.
If you run `ctest`, each testcase will write out a log file to the
<build>/unit_tests directory.
As with Autotools' make check, the test files are from test/.split
and unit_tests/.split files, but for CMake these are generated at
build time instead of at test time.
On Posix systems, sets the LD_LIBRARY_PATH so that ClamAV-compiled
libraries can be loaded when running tests.
On Windows systems, CTest will identify and collect all library
dependencies and assemble a temporarily install under the
build/unit_tests directory so that the libraries can be loaded when
running tests.
The same feature is used on Windows when using CMake to install to
collect all DLL dependencies so that users don't have to install them
manually afterwards.
Each of the CTest tests are run using a custom wrapper around Python's
unittest framework, which is also responsible for finding and inserting
valgrind into the valgrind tests on Posix systems.
Unlike with Autotools, the CMake CTest Valgrind-tests are enabled by
default, if Valgrind can be found. There's no need to set VG=1.
CTest's memcheck module is NOT supported, because we use Python to
orchestrate our tests.
Added a bunch of Windows compatibility changes to the unit tests.
These were primarily changing / to PATHSEP and making adjustments
to use Win32 C headers and ifdef out the POSIX ones which aren't
available on Windows. Also disabled a bunch of tests on Win32
that don't work on Windows, notably the mmap ones and FD-passing
(i.e. FILEDES) ones.
Add JSON_C_HAVE_INTTYPES_H definition to clamav-config.h to eliminate
warnings on Windows where json.h is included after inttypes.h because
json-c's inttypes replacement relies on it.
This is a it of a hack and may be removed if json-c fixes their
inttypes header stuff in the future.
Add preprocessor definitions on Windows to disable MSVC warnings about
CRT secure and nonstandard functions. While there may be a better
solution, this is needed to be able to see other more serious warnings.
Add missing file comment block and copyright statement for clamsubmit.c.
Also change json-c/json.h include filename to json.h in clamsubmit.c.
The directory name is not required.
Changed the hash table data integer type from long, which is poorly
defined, to size_t -- which is capable of storing a pointer. Fixed a
bunch of casts regarding this variable to eliminate warnings.
Fixed two bugs causing utf8 encoding unit tests to fail on Windows:
- The in_size variable should be the number of bytes, not the character
count. This was was causing the SHIFT_JIS (japanese codepage) to UTF8
transcoding test to only transcode half the bytes.
- It turns out that the MultiByteToWideChar() API can't transcode
UTF16-BE to UTF16-LE. The solution is to just iterate over the buffer
and flip the bytes on each uint16_t. This but was causing the UTF16-BE
to UTF8 tests to fail.
I also split up the utf8 transcoding tests into separate tests so I
could see all of the failures instead of just the first one.
Added a flags parameter to the unit test function to open testfiles
because it turns out that on Windows if a file contains the \r\n it will
replace it with just \n if you opened the file as a text file instead of
as binary. However, if we open the CBC files as binary, then a bunch of
bytecode tests fail. So I've changed the tests to open the CBC files in
the bytecode tests as text files and open all other files as binary.
Ported the feature tests from shell scripts to Python using a modified
version of our QA test-framework, which is largely compatible and will
allow us to migrate some QA tests into this repo. I'd like to add GitHub
Actions pipelines in the future so that all public PR's get some testing
before anyone has to manually review them.
The clamd --log option was missing from the help string, though it
definitely works. I've added it in this commit.
It appears that clamd.c was never clang-format'd, so this commit also
reformats clamd.c.
Some of the check_clamd tests expected the path returned by clamd to
match character for character with original path sent to clamd. However,
as we now evaluate real paths before a scan, the path returned by clamd
isn't going to match the relative (and possibly symlink-ridden) path
passed to clamdscan. I fixed this test by changing the test to search
for the basename: <signature> FOUND within the response instead of
matching the exact path.
Autotools: Link check_clamd with libclamav so we can use our utility
functions in check_clamd.c.
Flex and Bison are generally available not not particularly easy to
install and on macOS, the Bison version is relatively ancient and not
compatible. Homebrew doesn't necessarily play nice with Xcode, so to
make CMake builds work on macOS without mandating the use of Homebrew,
our best option is to make Flex & Bison optional.
Flex and Bison generated files will be kept in revision control and will
get re-generated only if you use -DMAINTAINER_MODE=ON which will
introduce the Flex and Bison tool dependencies.
CMake: don't emit fullpath in yara generated source
Autotool's ylwrap script has a hack that prevents the full path of the
bison & flex generated source from being included in the debug line
numbers and in the preprocessor include guard macros. CMake doesn't have
this, so when it sets the output file to the full path, the current
user's path is leaked into the generated source.
Added `%output "yara_grammar.c"` to yara_grammar.y and re-generated the
.c & .h file with this change. This overrides the "FILE" setting used
when generating those line numbers and include guard macro names so that
the path isn't included.
Similarly, added `%option outfile="yara_lexer.c` to yara_lexer.l and
re-generated the .c file with this change. This has the same effect but
for flex so that full filepaths are not emitted into the source.
Revert NEWS.md item regarding Flex, Bison change.
Revert placing yara grammar/lexer files in win32 compat.
The pcre2.h header dependency is propagated to the bytecode runtime,
lzma_sdk, yara, and regex build targets within the libclamav build
because it is included by matcher.h which is included all over the
place.
This patch adds the pcre2 dependency to the affected build targets so
that systems where pcre2 isn't in the standard include path can still
build.
Also removed CMake `PCRE2_DIR` from documentation, as it doesn't apply
to this PCRE2 detection logic that we settled on.
This patch adds experimental-quality CMake build tooling.
The libmspack build required a modification to use "" instead of <> for
header #includes. This will hopefully be included in the libmspack
upstream project when adding CMake build tooling to libmspack.
Removed use of libltdl when using CMake.
Flex & Bison are now required to build.
If -DMAINTAINER_MODE, then GPERF is also required, though it currently
doesn't actually do anything. TODO!
I found that the autotools build system was generating the lexer output
but not actually compiling it, instead using previously generated (and
manually renamed) lexer c source. As a consequence, changes to the .l
and .y files weren't making it into the build. To resolve this, I
removed generated flex/bison files and fixed the tooling to use the
freshly generated files. Flex and bison are now required build tools.
On Windows, this adds a dependency on the winflexbison package,
which can be obtained using Chocolatey or may be manually installed.
CMake tooling only has partial support for building with external LLVM
library, and no support for the internal LLVM (to be removed in the
future). I.e. The CMake build currently only supports the bytecode
interpreter.
Many files used include paths relative to the top source directory or
relative to the current project, rather than relative to each build
target. Modern CMake support requires including internal dependency
headers the same way you would external dependency headers (albeit
with "" instead of <>). This meant correcting all header includes to
be relative to the build targets and not relative to the workspace.
For example, ...
```c
include "../libclamav/clamav.h"
include "clamd/clamd_others.h"
```
... becomes:
```c
// libclamav
include "clamav.h"
// clamd
include "clamd_others.h"
```
Fixes header name conflicts by renaming a few of the files.
Converted the "shared" code into a static library, which depends on
libclamav. The ironically named "shared" static library provides
features common to the ClamAV apps which are not required in
libclamav itself and are not intended for use by downstream projects.
This change was required for correct modern CMake practices but was
also required to use the automake "subdir-objects" option.
This eliminates warnings when running autoreconf which, in the next
version of autoconf & automake are likely to break the build.
libclamav used to build in multiple stages where an earlier stage is
a static library containing utils required by the "shared" code.
Linking clamdscan and clamdtop with this libclamav utils static lib
allowed these two apps to function without libclamav. While this is
nice in theory, the practical gains are minimal and it complicates
the build system. As such, the autotools and CMake tooling was
simplified for improved maintainability and this feature was thrown
out. clamdtop and clamdscan now require libclamav to function.
Removed the nopthreads version of the autotools
libclamav_internal_utils static library and added pthread linking to
a couple apps that may have issues building on some platforms without
it, with the intention of removing needless complexity from the
source. Kept the regular version of libclamav_internal_utils.la
though it is no longer used anywhere but in libclamav.
Added an experimental doxygen build option which attempts to build
clamav.h and libfreshclam doxygen html docs.
The CMake build tooling also may build the example program(s), which
isn't a feature in the Autotools build system.
Changed C standard to C90+ due to inline linking issues with socket.h
when linking libfreshclam.so on Linux.
Generate common.rc for win32.
Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef
from misc.c.
Add CMake files to the automake dist, so users can try the new
CMake tooling w/out having to build from a git clone.
clamonacc changes:
- Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other
similar macros.
- Added a new clamav-clamonacc.service systemd unit file, based on
the work of ChadDevOps & Aaron Brighton.
- Added missing clamonacc man page.
Updates to clamdscan man page, add missing options.
Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use
libclamav).
Rename Windows mspack.dll to libmspack.dll so all ClamAV-built
libraries have the lib-prefix with Visual Studio as with CMake.
Val Snyder