Disable the HTTPUserAgent config option if the DatabaseMirror uses
clamav.net. This will prevent users from being inadvertently blocked and
will ensure that we can keep better metrics on which clamav versions are
being used.
This change is needed because we observed some users being blocked by
the new CDN rules when they set custom user-agents.
Improvements to use modern block list and allow list verbiage.
blacklist -> block list
whitelist -> allow listed
blacklisted -> blocked
whitelisted -> allowed
In the case of certificate verification, use "trust" or "verify" when
something is allowed.
Also changed domainlist -> domain list (or DomainList) to match.
CMake is now required to build.
The built-in LLVM is no longer available.
Also removed support for libltdl calls, which is not used in the CMake
builds, was only used when building with Autotools.
TODO: Fix CMake LLVM support & update to work with modern versions.
Added special warning messages for 403 and 429 HTTP codes.
For 403, FreshClam will fail (non-zero exit code) if not in daemon-mode.
For 429, FreshClam will succeed (exit 0) if not in daemon-mode.
Adds If-Modified-Since header for CVD downloads (not just CVD-head)
which should reduce data usage if DNS is advertising a newer version
than is actually available, which seems to happen sometimes due to
caching issues, it should still fail out when this happens - it just
won't have to download the older CVD, and should detect the HTTP 304
(Not-Modified) response instead.
Also replaced "Freshclam" with "FreshClam" in a few places, for
consistency.
This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673
Changes in 0.103 to order of operations for creating fmaps and
performaing hashes of fmaps resulted errors when scanning files that are
4096M and a different (but related) error when scanning files > 4096M.
This is despite the fact that scanning is supposed to be limited to
--max-scansize (MaxScanSize) and was also apparently limited to
INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes
for a few large file formats.
(see https://bugzilla.clamav.net/show_bug.cgi?id=10960)
This last limitation was not documented, so I added it to the sample
clamd.conf.
Anyways, the main issue is that the fmap module was using "unsigned int"
and was then enforcing a limitation (verbose error messages) when that
a map length exceeded the capapacity of an unsigned int. This commit
switches the associated variables over to uint64_t, and while fmaps are
still limited to size_t in other places, the fmap module will at least
work with files > 4G on 64bit systems.
In testing this, I found that the time to hash a file, particularly when
hashing a file on an NTFS partition from Linux was really slow because
we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing
this to 10MB chunks speeds up scanning of large files.
Finally, now that hashing is performed immediately when an fmap is
created for a file, hashing of files larger than max-scansize was
occuring. This commit adds checks to bail out early if the file size
exceeds the maximum before creating an fmap. It will alert with the
Heuristics.Limits.Exceeded name if the heuristic is enabled.
Also fixed CheckFmapFeatures.cmake module that detects if
sysconf(_SC_PAGESIZE) is available.
VirusEvent commands may use %v to get the signature name (virus name)
for the alert but do not have a format option to get the file name.
This commit adds %f to get the file name.
The VirusEvent feature does provide two environment variables,
$CLAM_VIRUSEVENT_FILENAME and $CLAM_VIRUSEVENT_VIRUSNAME which provide
file and virus names, but they weren't documented in the sample configs.
This commit also adds these environment variables to the sample configs.
Added a new scan option to alert on broken media (graphics) file
formats. This feature mitigates the risk of malformed media files
intended to exploit vulnerabilities in other software. At present
media validation exists for JPEG, TIFF, PNG, and GIF files.
To enable this feature, set `AlertBrokenMedia yes` in clamd.conf, or
use the `--alert-broken-media` option when using `clamscan`.
These options are disabled by default for now.
Application developers may enable this scan option by enabling
`CL_SCAN_HEURISTIC_BROKEN_MEDIA` for the `heuristic` scan option bit
field.
Fixed PNG parser logic bugs that caused an excess of parsing errors
and fixed a stack exhaustion issue affecting some systems when
scanning PNG files. PNG file type detection was disabled via
signature database update for 0.103.0 to mitigate effects from these
bugs.
Fixed an issue where PNG and GIF files no longer work with Target:5
(graphics) signatures if detected as CL_TYPE_PNG/GIF rather than as
CL_TYPE_GRAPHICS. Target types now support up to 10 possible file
types to make way for additional graphics types in future releases.
Scanning JPEG, TIFF, PNG, and GIF files will no longer return "parse"
errors when file format validation fails. Instead, the scan will alert
with the "Heuristics.Broken.Media" signature prefix and a descriptive
suffix to indicate the issue, provided that the "alert broken media"
feature is enabled.
GIF format validation will no longer fail if the GIF image is missing
the trailer byte, as this appears to be a relatively common issue in
otherwise functional GIF files.
Added a TIFF dynamic configuration (DCONF) option, which was missing.
This will allow us to disable TIFF format validation via signature
database update in the event that it proves to be problematic.
This feature already exists for many other file types.
Added CL_TYPE_JPEG and CL_TYPE_TIFF types.
When using --on-update-execute=EXIT_1 freshclam doesn't clean up the
temporary directory where it downloaded and tested the new database.
This patch moves the command execution to happen after temp-cleanup.
Drop privileges in the parent process before waiting for the signal
from the child process, so that the parent properly responds to
the signal and terminates.
Verify that the log file will be owned by user that the deamon will
run as.
Explicitly set PID file ownership to root when starting the daemon
as root with the PID file enabled.
There is an autotools variable and clamav-config.h macro BUILD_CLAMD
which dates back 17 years to a time where libpthread support was
optional and users could build clamscan without building clamd,
clamdscan, clamav-milter, etc. We don't live in that world anymore and
certainly don't test that configuration.
To get the NotifyClamd freshclam feature to work with CMake builds we
either have to define BUILD_CLAMD from CMake as well, or remove the
feature entirely. I chose to remove it, because we don't test builds
without BUILD_CLAMD enabled and it adds needless complexity to our build
system and source.
This patch adds experimental-quality CMake build tooling.
The libmspack build required a modification to use "" instead of <> for
header #includes. This will hopefully be included in the libmspack
upstream project when adding CMake build tooling to libmspack.
Removed use of libltdl when using CMake.
Flex & Bison are now required to build.
If -DMAINTAINER_MODE, then GPERF is also required, though it currently
doesn't actually do anything. TODO!
I found that the autotools build system was generating the lexer output
but not actually compiling it, instead using previously generated (and
manually renamed) lexer c source. As a consequence, changes to the .l
and .y files weren't making it into the build. To resolve this, I
removed generated flex/bison files and fixed the tooling to use the
freshly generated files. Flex and bison are now required build tools.
On Windows, this adds a dependency on the winflexbison package,
which can be obtained using Chocolatey or may be manually installed.
CMake tooling only has partial support for building with external LLVM
library, and no support for the internal LLVM (to be removed in the
future). I.e. The CMake build currently only supports the bytecode
interpreter.
Many files used include paths relative to the top source directory or
relative to the current project, rather than relative to each build
target. Modern CMake support requires including internal dependency
headers the same way you would external dependency headers (albeit
with "" instead of <>). This meant correcting all header includes to
be relative to the build targets and not relative to the workspace.
For example, ...
```c
include "../libclamav/clamav.h"
include "clamd/clamd_others.h"
```
... becomes:
```c
// libclamav
include "clamav.h"
// clamd
include "clamd_others.h"
```
Fixes header name conflicts by renaming a few of the files.
Converted the "shared" code into a static library, which depends on
libclamav. The ironically named "shared" static library provides
features common to the ClamAV apps which are not required in
libclamav itself and are not intended for use by downstream projects.
This change was required for correct modern CMake practices but was
also required to use the automake "subdir-objects" option.
This eliminates warnings when running autoreconf which, in the next
version of autoconf & automake are likely to break the build.
libclamav used to build in multiple stages where an earlier stage is
a static library containing utils required by the "shared" code.
Linking clamdscan and clamdtop with this libclamav utils static lib
allowed these two apps to function without libclamav. While this is
nice in theory, the practical gains are minimal and it complicates
the build system. As such, the autotools and CMake tooling was
simplified for improved maintainability and this feature was thrown
out. clamdtop and clamdscan now require libclamav to function.
Removed the nopthreads version of the autotools
libclamav_internal_utils static library and added pthread linking to
a couple apps that may have issues building on some platforms without
it, with the intention of removing needless complexity from the
source. Kept the regular version of libclamav_internal_utils.la
though it is no longer used anywhere but in libclamav.
Added an experimental doxygen build option which attempts to build
clamav.h and libfreshclam doxygen html docs.
The CMake build tooling also may build the example program(s), which
isn't a feature in the Autotools build system.
Changed C standard to C90+ due to inline linking issues with socket.h
when linking libfreshclam.so on Linux.
Generate common.rc for win32.
Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef
from misc.c.
Add CMake files to the automake dist, so users can try the new
CMake tooling w/out having to build from a git clone.
clamonacc changes:
- Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other
similar macros.
- Added a new clamav-clamonacc.service systemd unit file, based on
the work of ChadDevOps & Aaron Brighton.
- Added missing clamonacc man page.
Updates to clamdscan man page, add missing options.
Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use
libclamav).
Rename Windows mspack.dll to libmspack.dll so all ClamAV-built
libraries have the lib-prefix with Visual Studio as with CMake.
Removed all autotools generates files. Autotools (autoconf, automake,
libtool, pkg-config, m4) will be required from now on for builds from
git clones.
Added autogen.sh to be run before ./configure.
Significant update to main .gitignore file.
Removed extraneous .gitignore files. A Git repository only needs one
.gitignore file.
Add notices to man pages and help strings cautioning against running
bytecode signatures from untrusted sources.
Also adds missing BytecodeUnsigned option to clamd.conf.sample files.
Add clamd config option to force blocking clamd database reload to
conserve RAM. Users may set `ConcurrentDatabaseReload no` in their
clamd.conf config file to force a blocking reload.
The blocking mode will still perform the reload in a new thread, but
will first free the current database, wait for scans targeting that
database to complete, and then load the new database in the new thread
and wait (`pthread_join()`) on that thread. Once loaded, any pending
scans will continue. This is effectively the same behavior as how
clamd reloads worked before the multi-threaded database reload feature
was added.
Add Data-Loss-Prevention option to detect credit cards only, excluding
debit and private label cards where possible.
You can select the credit card-only DLP mode for clamscan with the
`--structured-cc-mode` command-line option.
You can select the credit card-only DLP mode for clamd with the
`StructuredCCOnly` clamd.conf config option.
This patch also adds credit card matching for additional vendors:
- Mastercard 2016
- China Union Pay
- Discover 2009
This fixes issues in cvd download when network speed is slow.
Setting is passed to libcurl CURLOPT_TIMEOUT. Original default of 60s
was not enough if network speed is limited. Curl handles this as
total time for http(s) transfer.
https://curl.haxx.se/libcurl/c/CURLOPT_TIMEOUT.html
Also change commented out setting of ReceiveTimeout on example configs
to somewhat sensible value (1800s).
Signed-off-by: Tuomo Soini <tis@foobar.fi>
New features added to freshclam:
- Update signature definitions over HTTPS.
- Support for HTTP protocol v1.1 (formerly v1.0).
- New libfreshclam library with an all new API and versioning separate from libclamav (v2.0.0). This library is now build and installed alongside libclamav as a hard dependency of freshclam.
- The ability to opt-in and opt-out of standard and optional official ClamAV databases (ExtraDatabase, ExcludeDatabase)
- The option to specify the protocol and port number of official and private mirror servers.
- Support for additional types of proxy servers beyond plain HTTP (SOCKS 4, SOCKS 5).
Features removed from freshclam:
- Mirror management (mirrors.dat) file. This feature is no longer needed as official signature databases are distributed using a paid content delivery network (Cloudflare).
This commit also adds the following features for Windows users:
- The clamsubmit tool.
- The json-c library dependency, which will enable the --gen-json option in clamscan.
- Third party libraries under the win32/3rdparty directory have been removed. Developers will need to build the libraries separately from ClamAV and provide the headers and lib/dll library files the same way they do for OpenSSL. This includes libxml2, pthread-win32, bzip2, zlib, pcre2 as well as new dependencies: curl, json-c. Developers are encouraged to use the build tool Mussels to simplify this task.
Micah Snyder