15 Commits (c5b91947c6cf98c6048eb98994ef2db6496158e5)

Author	SHA1	Message	Date
Micah Snyder	4a9cff9214	CMake: support Xcode builds ... Xcode (and perhaps some other generators?) do not like targets that have only object files. See: https://cmake.org/cmake/help/latest/command/add_library.html#object-libraries And: https://cmake.org/pipermail/cmake/2016-May/063479.html This issue manifests when using `-G Xcode` on macOS as the library dylibs being missing when linking with other binaries. This commit removes the object libraries for libclamav, libfreshclam, libclamunrar_iface, libclamunrar, libclammspack, and (lib)common because they were used by static or shared libs that didn't themselves have any added sources. Add getter & setter for the debug flag, so it isn't referenced by unit tests or other code that links with libclamav. This is needed because global variables are exported symbols on Windows.	4 years ago
Andy Ragusa	c4af06c317	Fix ENABLE_UNRAR=off build ... Cmake errors out when the ENABLE_UNRAR=off option is used. This commit addresses that.	4 years ago
Micah Snyder	201e1b12a7	XOR test files; clean up tests directory ... The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.	4 years ago
Yasuhiro Kimura	fb479870cc	Fix configuration error when ENABLE_EXTERNAL_MSPACK is ON ... This fixes https://bugzilla.clamav.net/show_bug.cgi?id=12759	4 years ago
Micah Snyder	da23b1ceab	CMake: Fix support for external LLVM (3.6) ... Have to manually link libtinfo (`-ltinfo`) because our FindLLVM didn't add it to the LLVM_LIBRARIES variable for us. See: - https://stackoverflow.com/questions/21477407/llvm-3-5-fails-to-link Have to remove the CXX_STANDARD setting at the top of CMakeLists.txt because of c++90 / c++11 ABI compatibility issues w/ LLVM. See: - https://maleadt.github.io/LLVM.jl/dev/man/troubleshooting/ Rename "llvm/Config/config.h" "llvm/Config/llvm-config.h" because LLVM renamed it in 2.8. Have to link LLVM manually with the test binaries that use the clamav object library instead of libclamav shared library. CMake does not propagate library dependencies from object files. I tested on ubuntu:16.04 with LLVM 3.6 built from source using: ``` /usr/local/bin/cmake .. -D CMAKE_INSTALL_PREFIX=/opt/llvm/3.6 \ -D LLVM_ENABLE_RTTI=ON ``` Then built clamav w/: ``` /usr/local/bin/cmake .. -D CMAKE_INSTALL_PREFIX=`pwd`/install \ -D BYTECODE_RUNTIME="llvm" \ -D LLVM_ROOT_DIR="/opt/llvm/3.6" \ -D LLVM_FIND_VERSION="3.6.0" && make && make install ```	4 years ago
Micah Snyder	473496c3c7	tests: Use correct pytest command ... If pytest is available in the path, but wasn't found in the Python 3 installation discovered by CMake, then the test suite will fail. This can happen when two different Python 3 installations exist, eg 3.6 and 3.7 and when the newest one (prefered by CMake) doesn't have pytest. This commit will use the correct command depending on where pytest was (or wasn't) found. It will also fail more gracefully if neither pytest nor unittest was found (this happens with some wierd python installs).	4 years ago
Micah Snyder	c025afd683	Rename "shared" library to "common" ... The named "shared" is confusing, especially now that these features are built as a static library instead of being directly compiled into the various applications.	4 years ago
Micah Snyder	1cc8c2dce3	Test: fix check_clamd failure on Windows ... Also add support for using pytest instead of python's unittest to make it easier to find & read failed test results. Clean up the log output in check_clamd when printing "wrong reply" results. Switch from using collections.namedtuple() to defining our own NamedTuple subclass, which adds member variable typing.	4 years ago
Micah Snyder	d26b71d02d	cmake: Fix vcpkg debug build issues ... Adds support to the pcre2 and pthreadw32 Find<Package>.cmake modules for correctly discovering the debug versions. This change modeled after the upstream FindBZip2.cmake module. Also eliminated HAVE_STRUCT_TIMESPEC redefinition warnings in Windows builds.	4 years ago
Micah Snyder	05a8d589e7	CMake: improve multiarch support ... In testing on Alpine, I found that most libs were installing to <prefix>/lib while libclamav installed to <prefix>/lib64. Those who like multiarch will advocate for lib64, though I only actually noticed it because clamscan failed to find libclamav.so! Anyways, they should all install to lib64 by default if that's what how the system is set up. Using ${CMAKE_INSTALL_FULL_LIBDIR} instead of <prefix>/lib will do that.	4 years ago
Micah Snyder	5bb8aa8a37	tests: test in a tmp-dir; w. relative clamd socket ... The clamd socket path was changed be an absolute path when adding CTest support. This quietly broke the check_clamd libcheck program when building with autotools because a relative path was expected. I failed to notice because the autotools `make check` doesn't actually care if check_clamd works! It turns out that a relative path is required because the max length for a socket path is *very* short. This commit changes check_clamd and the associated CMake test to also use a relative path for the clamd socket. Notably it also modifies the testcase.py framework switch to the cls.path_tmp (generated) directory before the tests and restore the CWD after the tests so as to ensure that the socket file is dropped in somewhere in that tmp directory.	4 years ago
Micah Snyder	c81968d3a7	GitHub Actions testing on Ubuntu, Mac, & Windows ... Updates to fix issues in the CMake install instructions. Updates the README.md to indicate that CMake is now preferred Adds a GitHub Actions badge, Discord badge, and logo to the README.md. CMake: - Renamed ENABLE_DOCS to ENABLE_MAN_PAGES. - Fixed build issue when milter isn't enabled on Linux. Changed the default to build milter on non-macOS, non-Windows operating systems. - Fix LD_LIBRARY_PATH for tests including on macOS where LD_LIBRARY_PATH and DYLD_LIBRARY_PATH must be manually propagated to subprocesses. - Use UNKNOWN IMPORTED library instead of INTERFACE IMPORTED library for pdcurses, but still use INTERFACE IMPORTED for ncurses. UNKNOWN IMPORTED appears to be required so that we can use $<TARGET_FILE_DIR:Curses::curses> to collected the pdcurses library at install time on Windows. - When building with vcpkg on Windows, CMake will automatically install your app local dependencies (aka the DLL runtime dependencies). Meanwhile, file(GET_RUNTIME_DEPENDENCIES ...) doesn't appear to work correctly with vcpkg packages. The solution is to use a custom target that has CMake perform a local install to the unit_tests directory when using vcpkg. This is in fact far easier than using GET_RUNTIME_DEPENDENCIES in the unit_tests for assembling the test environment but we can't use this method for the non-vcpkg install because it won't collect checkDynamic.dll for us because we don't install our tests. We also can't link with the static check.lib because the static check.lib has pthreads symbols linked in and will conflict with our pthread.dll. TL;DR: We'll continue to use file(GET_RUNTIME_DEPENDENCIES ...) for assembling the test enviornment on non-vcpkg builds, and use the local install method for vcpkg builds. testcase.py: Wrapped a Pathlib.unlink() call in exception handling as the missing_ok optional parameter requires a Python version too new for common use. Remove localtime_r from win32 compat lib. localtime_r may be present in libcheck when building with vcpkg and while making it a static function would also solve the issue, using localtime_s instead like we do everywhere else should work just fine. check_clamd: Limited the max # of connections for the stress test on Mac to 850, to address issues found testing on macos-latest on GitHub Actions.	4 years ago
Micah Snyder	840b559c7a	Remove Visual Studio project files ... Visual Studio projects removed in favor of CMake because it's far easier to build and maintain. Also removed the old InnoSetup installer now that CMake's CPack provides installer creation. While working on this I found that the THIS_IS_CLAMAV macro was missing, resulting in warnings for the `have_rar` and `have_clamjit` exported global variables. I also stumbled across some code duplication and more cl_error_t / int type issues in the pcre code, so this commit includes a little cleanup.	4 years ago
Micah Snyder	451279876e	CMake: Add fuzz support. ... Enabled the metadata collection feature, scan heuristics, and all-match mode when fuzzing in the interest of better code coverage. Also remove deprecated STREAM command.	4 years ago
Micah Snyder	2552cfd0d1	CMake: Add CTest support to match Autotools checks ... An ENABLE_TESTS CMake option is provided so that users can disable testing if they don't want it. Instructions for how to use this included in the INSTALL.cmake.md file. If you run `ctest`, each testcase will write out a log file to the <build>/unit_tests directory. As with Autotools' make check, the test files are from test/.split and unit_tests/.split files, but for CMake these are generated at build time instead of at test time. On Posix systems, sets the LD_LIBRARY_PATH so that ClamAV-compiled libraries can be loaded when running tests. On Windows systems, CTest will identify and collect all library dependencies and assemble a temporarily install under the build/unit_tests directory so that the libraries can be loaded when running tests. The same feature is used on Windows when using CMake to install to collect all DLL dependencies so that users don't have to install them manually afterwards. Each of the CTest tests are run using a custom wrapper around Python's unittest framework, which is also responsible for finding and inserting valgrind into the valgrind tests on Posix systems. Unlike with Autotools, the CMake CTest Valgrind-tests are enabled by default, if Valgrind can be found. There's no need to set VG=1. CTest's memcheck module is NOT supported, because we use Python to orchestrate our tests. Added a bunch of Windows compatibility changes to the unit tests. These were primarily changing / to PATHSEP and making adjustments to use Win32 C headers and ifdef out the POSIX ones which aren't available on Windows. Also disabled a bunch of tests on Win32 that don't work on Windows, notably the mmap ones and FD-passing (i.e. FILEDES) ones. Add JSON_C_HAVE_INTTYPES_H definition to clamav-config.h to eliminate warnings on Windows where json.h is included after inttypes.h because json-c's inttypes replacement relies on it. This is a it of a hack and may be removed if json-c fixes their inttypes header stuff in the future. Add preprocessor definitions on Windows to disable MSVC warnings about CRT secure and nonstandard functions. While there may be a better solution, this is needed to be able to see other more serious warnings. Add missing file comment block and copyright statement for clamsubmit.c. Also change json-c/json.h include filename to json.h in clamsubmit.c. The directory name is not required. Changed the hash table data integer type from long, which is poorly defined, to size_t -- which is capable of storing a pointer. Fixed a bunch of casts regarding this variable to eliminate warnings. Fixed two bugs causing utf8 encoding unit tests to fail on Windows: - The in_size variable should be the number of bytes, not the character count. This was was causing the SHIFT_JIS (japanese codepage) to UTF8 transcoding test to only transcode half the bytes. - It turns out that the MultiByteToWideChar() API can't transcode UTF16-BE to UTF16-LE. The solution is to just iterate over the buffer and flip the bytes on each uint16_t. This but was causing the UTF16-BE to UTF8 tests to fail. I also split up the utf8 transcoding tests into separate tests so I could see all of the failures instead of just the first one. Added a flags parameter to the unit test function to open testfiles because it turns out that on Windows if a file contains the \r\n it will replace it with just \n if you opened the file as a text file instead of as binary. However, if we open the CBC files as binary, then a bunch of bytecode tests fail. So I've changed the tests to open the CBC files in the bytecode tests as text files and open all other files as binary. Ported the feature tests from shell scripts to Python using a modified version of our QA test-framework, which is largely compatible and will allow us to migrate some QA tests into this repo. I'd like to add GitHub Actions pipelines in the future so that all public PR's get some testing before anyone has to manually review them. The clamd --log option was missing from the help string, though it definitely works. I've added it in this commit. It appears that clamd.c was never clang-format'd, so this commit also reformats clamd.c. Some of the check_clamd tests expected the path returned by clamd to match character for character with original path sent to clamd. However, as we now evaluate real paths before a scan, the path returned by clamd isn't going to match the relative (and possibly symlink-ridden) path passed to clamdscan. I fixed this test by changing the test to search for the basename: <signature> FOUND within the response instead of matching the exact path. Autotools: Link check_clamd with libclamav so we can use our utility functions in check_clamd.c.	4 years ago