New features added to freshclam:
- Update signature definitions over HTTPS.
- Support for HTTP protocol v1.1 (formerly v1.0).
- New libfreshclam library with an all new API and versioning separate from libclamav (v2.0.0). This library is now build and installed alongside libclamav as a hard dependency of freshclam.
- The ability to opt-in and opt-out of standard and optional official ClamAV databases (ExtraDatabase, ExcludeDatabase)
- The option to specify the protocol and port number of official and private mirror servers.
- Support for additional types of proxy servers beyond plain HTTP (SOCKS 4, SOCKS 5).
Features removed from freshclam:
- Mirror management (mirrors.dat) file. This feature is no longer needed as official signature databases are distributed using a paid content delivery network (Cloudflare).
This commit also adds the following features for Windows users:
- The clamsubmit tool.
- The json-c library dependency, which will enable the --gen-json option in clamscan.
- Third party libraries under the win32/3rdparty directory have been removed. Developers will need to build the libraries separately from ClamAV and provide the headers and lib/dll library files the same way they do for OpenSSL. This includes libxml2, pthread-win32, bzip2, zlib, pcre2 as well as new dependencies: curl, json-c. Developers are encouraged to use the build tool Mussels to simplify this task.
It is possible for bm->offset to be negative and (offset + bm->offset)
to be positive, in which case the bounds check was incorrectly skipped,
which could result in a segfault.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fea90598db0 in cli_bcomp_compare_check (
f_buffer=0x7fea5c9e3a3e <error: Cannot access memory at address 0x7fea5c9e3a3e>, f_buffer@entry=0x7fea5c98c1ba "\001\030\001\030",
buffer_length=buffer_length@entry=2590, offset=<optimized out>,
bm=bm@entry=0x7fea7289f9c8) at matcher-byte-comp.c:720
- If cross-compiling platform like ARM, the mmap support will be
disabled directly, which indirectly disabled mempool function, too.
Without mempool, the engine initialization time will be very long
due to memory fragmentation. The fragmentation problem will make
heap grow very fast, and results in using a lot of swap while
running on low RAM machines. It will slow down initialization
and scanning process.
- Test result:
The initialization time of using memory pool on 256MB ARM machine is
faster than without mempool by 5-hour.
This addresses a regression with sample 848092559:
LDB sig (Win.Virus.Virut-5898123-1) that uses 'NumberOfSections:3-3'
started matching on a PE that has 4 sections, but one is totally outside
of the file and gets removed. Previously, two of the ClamAV PE header
parsing implementations handled this case differently, and the NDB/LDB
matching code would be told there were 4 sections while the bytecode
and unpacking code would only see 3 sections. When consolidating the
PE header parsing code, I made it so that the section always gets
removed.
For now we just replicate the original behavior by providing a new
flag to the PE header parsing code. We should re-evaluate the effects
that this has later, once we have better tests for the bytecode API
and we have test samples for each of the hardcoded detection cases in
cli_scanpe.
Also, fixes some memory leaks based on the changes in my last commit x_x
Changes include:
- Fixing several memory leaks noticed when running with ASan
- Adds documentation for several functions and structs
- Simplifies the interface for using cli_targetinfo_init/destroy
and cli_exe_info_init/destroy
- A few other minor changes
.crb rules are needed to validate .cat files before they get loaded
in, but when running clamscan with '-d <dir>' there wasn't any logic
to ensure that .cat files got loaded after the .crb files. This
commit changes that, and refactors the code a bit to make it easier
to add new ordering requirements and to make error handling cleaner.
Also, move the cert-related DCONF cfg checks to more
appropriate locations. One change in behavior:
PE_CONF_CATALOG will disable loading trusted hashes from
.cat files, but won't disable Authenticode hash checking
completely (PE_CONF_CERTS does this).
Instead of checking the Authenticode header as an FP prevention
mechanism, we now check it in the beginning if it exists. Also,
we can now do actual blacklisting with .crb rules (previously, a
blacklist rule just let you override a whitelist rule).
cli_checkfp_pe is now effectively the function that just checks
the Authenticode hash. This makes the code less complicated,
and adds some minor improvements:
- section hashes are no longer computed if there is no stats
callback function (at least in that part of the code)
- We now actually set the len field in the stats_section_t
structure
- If an error occurs when computing a section hash, we skip
that section instead of not computing any hashes
For certain aspack versions, the code wasn't restoring the correct
entrypoint. This had ramifications for unpacked file scanning,
since now an EP that points outside of a file makes the PE not
have .MDB sigs evaluated against it (a change from earlier versions,
but one which the MRT is OK with).
Consolidate the PE parsing code into one function. I tried to preserve all existing functionality from the previous, distinct implementations to a large extent (with the exceptions mentioned below). If I noticed potential bugs/improvements, I added a TODO statement about those so that they can be fixed in a smaller commit later. Also, there are more TODOs in places where I'm not entirely sure why certain actions are performed - more research is needed for these.
I'm submitting a pull request now so that regression testing can be done, and because merging what I have thus far now will likely have fewer conflicts than if I try to merge later
PE parsing code improvements:
- PEs without all 16 data directories are parsed more appropriately now
- Added lots more debug statements
Also:
- Allow MAX_BC and MAX_TRACKED_PCRE to be specified via CFLAGS
When doing performance testing with the latest CVD, MAX_BC and
MAX_TRACKED_PCRE need to be raised to track all the events.
Allow these to be specified via CFLAGS by not redefining them
if they are already defined
- Fix an issue preventing wildcard sizes in .MDB/.MSB rules
I'm not sure what the original intent of the check I removed was,
but it prevents using wildcard sizes in .MDB/.MSB rules. AFAICT
these wildcard sizes should be handled appropriately by the MD5
section hash computation code, so I don't think a check on that
is needed.
- Fix several issues related to db loading
- .imp files will now get loaded if they exist in a directory passed
via clamscan's '-d' flag
- .pwdb files will now get loaded if they exist in a directory passed
via clamscan's '-d' flag even when compiling without yara support
- Changes to .imp, .ign, and .ign2 files will now be reflected in calls
to cl_statinidir and cl_statchkdir (and also .pwdb files, even when
compiling without yara support)
- The contents of .sfp files won't be included in some of the signature
counts, and the contents of .cud files will be
- Any local.gdb files will no longer be loaded twice
- For .imp files, you are no longer required to specify a minimum flevel for wildcard rules, since this isn't needed
With FILE_OFFSET_BITS=64, st_size is 8-byte.
In this case, casting st_size to size_t, which is 4-byte on 32-bit machine,
will make the comparison go wrong.
Signed-off-by: rickwang <rickwang@synology.com>
Micah Snyder