Reviewing Coverity bug reports we found that the return value to this
filter_search call was effectively being ignored, causing no filtering
to occur. Fixing this issue resulted in a unit test that uses the
following match list regex to fail when searching for `ebay.com`.:
.+\\.paypal\\.(com|de|fr|it)([/?].*)?:.+\\.ebay\\.(at|be|ca|ch|co\\.uk|de|es|fr|ie|in|it|nl|ph|pl|com(\\.(au|cn|hk|my|sg))?)/
After investigating further, this is because the regex_list_add_pattern
call, which parses the regex for suffixes and attempts to add these to
the filter, can't handle the `com(\\.(au|cn|hk|my|sg))?` portion of
the regex. As a result, it only adds `ebay.at`, `ebay.be`, `ebay.ca`, up
through `ebay.pl` into the filter). With the code returning if no filter match
is found, the `ebay.com` suffix not existing in the filter causes incoming URLs
to be treated as if there are no corresponding regexes for ebay.com, which results
in no regex rules being evaluated against it.
We should get the regex parsing code working (and ensure it handles any
other complex cases in daily.cdb) before re-enabling this code. The code
has had no effect for 12+ years at this point, though, so it's probably
safe to wait a bit longer without it.
I had some time over the weekend and implemented fixes for warnings
in clamdtop/clamdtop.c:
- 279008 - In make_connection_real: Code can never be reached because
of a logical contradiction (CWE-561). host and port are freed and set
to NULL before they are used to set conn->remote. The fix is to
cleanup host and port after conn->remote is set.
- 147369 - In get_ip: Code can never be reached because of a logical
contradiction (CWE-561). Removed an unnecessary return statement that
could never be reached.
- 147624 - In get_port: Leak of memory or pointers to system resources
(CWE-404). dupip wasn’t being freed in the case where no port could
be found.
- 279007 - In make_connection_real: Pointer is checked against null
but then dereferenced anyway (CWE-476). This function checked soname
for NULL but wouldn’t fail in this case, and subsequent code would
dereference it. The NULL check could just be removed, though, because
the only calling function of this static function ensured that soname
is not NULL.
- 147316 - In cleanup: Value returned from a library function is not
checked for errors before being used. This value may indicate an error
condition. (CWE-252). The code ignores the value of send when exiting
because at that point it doesn’t really matter if the send succeeds.
The fix is to cast this function call to void to explicitly ignore
the return value.
- 147318 - In recv_line: Value returned from a library function is not
checked for errors before being used. This value may indicate an error
condition. (CWE-252) The code ignores the value of send when exiting
because at that point it doesn’t really matter if the send succeeds.
The fix is to cast this function call to void to explicitly ignore
the return value.
Fixed the following Coverity issues:
- 225236 - In cli_egg_extract_file: Dereference of an explicit
null value (CWE-476). The first fail case checked handle for
NULL and then dereferenced it in the done block
- 225209 - In executeIfNewVersion: Leak of memory or pointers
to system resources (CWE-404). modifiedCommand was defined
twice, with the inner instance being assigned to and the
outer instance being freed
- 225201 - In regex_list_match: Code can never be reached
because of a logical contradiction (CWE-561). The code had
logic off to the side that may have been missed:
filter_search_rc = filter_search(&matcher->filter, (const unsigned char *)bufrev, buffer_len) != -1;
if (filter_search_rc == -1) {
- 225198 - In phishingCheck: Leak of memory or pointers to
system resources (CWE-404). A fail case caused by malloc
failing would leak previously allocated memory.
- 225197 - In updatecustomdb: A pointer to freed memory
is dereferenced, used as a function argument, or otherwise
used (CWE-416). In a fail case, a pointer was freed and
then used in a debug print statement
- 225190 - In updatedb: A pointer to freed memory is
dereferenced, used as a function argument, or otherwise used
(CWE-416). In a fail case, a pointer was freed and then used
in a debug print statement
- 225195 - In cli_egg_open: The sizeof operator is used on a
wrong argument that incidentally has the same size (CWE-467).
sizeof(char **) was being used instead of sizeof(char *)
- 225193 - In egg_parse_comment_header: Code can never be
reached because of a logical contradiction (CWE-561).
A cleanup case for variable comment was unnecessary, and
to fix comment was removed entirely.
- 225147 - In get_server_node: Code can never be reached
because of a logical contradiction (CWE-561). A cleanup
case for variable url was unnecessary
- 225168 - In download_complete_callback: Missing break
statement between cases in switch statement (CWE-484).
In the case where forking failed, freshclam would check
the database without forking but then continue on to
execute the code intended to be done in the child process
because of a missing break statement
- 225152 - In cli_egg_lzma_decompress: Use of an
uninitialized variable (CWE-457). Certain fail cases
would call cli_LzmaShutdown on an uninitialized stream.
Now it’s only called after initialization occurs.
Looking through the list of issues, I spotted some easy ones and submitted
some fixes:
- 225229 - In cli_rarload: Leak of memory or pointers to system resources.
If finding the necessary libunrar functions fails (should be rare),we now
dlclose libunrar.
225224 - In main (freshclam.c): A copied piece of code is inconsistent with
the original (CWE-398). A minor copy-paste error was present, and optOutList
could be cleaned up in one of the failure edge cases.
225228 - In decodecdb: Out-of-bounds access to a buffer (CWE-119). Off by one
error when tokenizing certain CDB sig fields for printing with sigtool. Ex:
$ cat test.cdb
a:CL_TYPE_7Z:1-2-3:/.*/:1-2-3:1-2-3:0:1-2-3::
$ cat test.cdb | ../installed/bin/sigtool --decode
VIRUS NAME: a
CONTAINER TYPE: CL_TYPE_7Z
CONTAINER SIZE: WITHIN RANGE 1 to 2
FILENAME REGEX: /.*/
COMPRESSED FILESIZE: WITHIN RANGE 1 to 2
UNCOMPRESSED FILESIZE: WITHIN RANGE 1 to 2
ENCRYPTION: NO
FILE POSITION: =================================================================
==17245==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffe3136d10 at pc 0x7f0f31c3f414 bp 0x7fffe3136c70 sp 0x7fffe3136c60
WRITE of size 8 at 0x7fffe3136d10 thread T0
#0 0x7f0f31c3f413 in cli_strtokenize ../../libclamav/str.c:524
#1 0x559e9797dc91 in decodecdb ../../sigtool/sigtool.c:2929
#2 0x559e9797ea66 in decodesig ../../sigtool/sigtool.c:3058
#3 0x559e9797f31e in decodesigs ../../sigtool/sigtool.c:3162
#4 0x559e97981fbc in main ../../sigtool/sigtool.c:3638
#5 0x7f0f3100fb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
#6 0x559e9795a1d9 in _start (/home/zelda/workspace/clamav-devel/installed/bin/sigtool+0x381d9)
Address 0x7fffe3136d10 is located in stack of thread T0 at offset 48 in frame
#0 0x559e9797d113 in decodecdb ../../sigtool/sigtool.c:2840
This frame has 1 object(s):
[32, 48) 'range' <== Memory access at offset 48 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow ../../libclamav/str.c:524 in cli_strtokenize
- 225223 - In cli_egg_deflate_decompress: Reads an uninitialized pointer or
its target (CWE-457). Certain fail cases would call inflateEnd on an
uninitialized stream. Now it’s only called after initialization occurs.
- 225220 - In buildcld: Use of an uninitialized variable (CWE-457). Certain
fail cases would result in oldDir being used before initialization. It now
gets zeroed before the first fail case.
- 225219 - In cli_egg_open: Leak of memory or pointers to system resources
(CWE-404). If certain realloc’s failed, several structures would not be cleaned up
- 225218 - In cli_scanhwpml: Code block is unreachable because of the syntactic
structure of the code (CWE-561). With certain macros set, there could be two
consecutive return statements.
This commit includes bug fixes and minor modifications based on
warnings generated by Coverity. These include:
- 287096 - In cli_xlm_extract_macros: Leak of memory or pointers
to system resources (CWE-404). This was a legitimate leak of a
generated temp filename and could occur frequently.
- 287095 - In scan_for_xlm_macros: Use of an uninitialized
variable. The uninitialized value (state.length) was likely
never used unitialized, but we now initialize it just in case.
- 287094 - In cli_vba_readdir_new: Out-of-bounds access to a
buffer (CWE-119). This looks like a copy-paste error and was
a legitimate read past the bounds of a buffer in an error case.
- 284479 - In hfsplus_walk_catalog: All paths that lead to this
null pointer comparison already dereference the pointer earlier
(CWE-476). In certain cases a NULL pointer could be returned
in the success case of hfsplus_scanfile, which was not handled
correctly. This case may have been prevented in practice by
an earlier check, but adding a check for NULL just in case.
- 284478 - In hfsplus_walk_catalog: A value assigned to a
variable is never used. ret would be set if zlib's inflateEnd
function fails. The fix is to just not set ret in this case,
since the error doesn't seem fatal (although would result in
a memory leak by the zlib code...).
- 284477 - In hfsplus_check_attribute: Pointer is checked against
null but then dereferenced anyway. I just took out the NULL check
of record and recordSize, since the code requires these values
to not be NULL elsewhere and there's no way an error could
occur as currently used (stack var addresses are passed via these
parameters).
I also fixed up some of the function identifiers in debug print
messages.
- Moved calculation of remtime so it will only be done if
fractiondownloaded is greater than 0.0, to avoid division by zero.
- Optimized the printing of bytes values by removing handling of
upload values and removing the if statements altogether; the rest
of the function does not handle uploads and it already returns
early if TotalToDownload isn't greater than 0.0.
- Progress display now utilizes format string field widths for
consistent alignment (no shifting around as information changes),
a shorter progress bar, and more intuitive Xm XXs (or Xh XXm) time
displays for values of 60 seconds or more. It is now practically
assured that the progress display will not spill over to the next
line in an 80-column console.
- Simplified a few calculations. Bytes display now changes units at the
proper 1024 B/KiB instead of 2048 B/KiB.
- Fixed the progress bar not showing anything when numDots == 1.
- Replaced an instance of printf(" ") in progress function with
fprintf(stdout, " ") to be consistent with the other output calls.
Currently, freshclam's "OnUpdateExecute", "OnErrorExecute", and
"OnOutdatedExecute" options will fail when running on Windows in daemon
mode if the supplied command contains any arguments.
This patch fixes execute() function on Windows in daemon mode by
replacing spawnlp() with system().
Some detections, like phishing, are considered heuristic alerts because
they match based on behavior more than on content. A subset of these
are considered "potentially unwanted" (low-severity). These
low-severity alerts include:
- phishing
- PDFs with obfuscated object names
- bytecode signature alerts that start with "BC.Heuristics"
The concept is that unless you enable "heuristic precedence" (a method
of lowing the threshold to immediateley alert on low-severity
detections), the scan should continue after a match in case a higher
severity match is found. Only at the end will it print the low-severity
match if nothing else was found.
The current implementation is buggy though. Scanning of archives does
not correctly bail out for the entire archive if one email contains a
phishing link. Instead, it sets the "heuristic found" flag then and
alerts for every subsequent file in the archive because it doesn't know
if the heuristic was found in an embedded file or the target file.
Because it's just a heuristic and the status is "clean", it keeps
scanning.
This patch corrects the behavior by checking if a low-severity alerts
were found at the end of scanning the target file, instead of at the end
of each embedded file.
Additionally, this patch fixes an in issue with phishing alerts wherein
heuristic precedence mode did not cause a scan to stop after the first
alert.
The above changes required restructuring to create an fmap inside of
cl_scandesc_callback() so that scan_common() could be modified to
require an fmap and set up so that the current *ctx->fmap pointer is
never NULL when scan_common() evaluates match results.
Also fixed a couple minor bugs in the phishing unit tests and cleaned up
the test code for improved legitibility and type safety.
In regression testing we observed occasional errors creating files or
directories. It appears as though these are caused by tmp files that use
identical prefixes. Such prefixed tmp file names currently have a
5-character hash suffix. Though these temporary files or directories are
deleted when no longer needed, there is still the possibility that there
could be a hash collision which causes the scan to error out.
This patch doubles the size of the short-hash to 10-characters to reduce
the chance of a collision.
If using a bytecode signature that makes use of the BC_PRECLASS hook and
if it alerts, a NULL dereference may occur. This change fixes that.
Also fixed unrelated memory leaks introduced recently when adding file
name extraction to the zip parser and rar parser.
scan_common must either be passed an fmap (map) or a file
descriptor (desc) corresponding to the file being scanned.
In the case where map is NULL, scan_common will create an
fmap in order to execute the BC_PRECLASS bytecode hook, and
this fmap wasn't being unmapped afterward
Fixed copypaste bug with duplicated fmap names being assigned to the
parent instead of the dup/child fmap.
Fixed file descriptor initialization issue in the HTML normalizer.
Disables run time warning messages emitted by libxml2 when parsing
HTML email content for JSON metadata feature.
Fixed compile time warning caused by libjson-c API changes from int to
size_t.
At present many parsers create tmp subdirectories to store extracted
files. For parsers like the vba parser, this is required as the
directory is later scanned. For other parsers, these subdirectories are
probably not helpful now that we provide recursive sub-dirs when
--leave-temps is enabled. It's not quite as simple as removing the extra
subdirectories, however. Certain parsers, like autoit, don't create very
unique filenames and would result in file name collisions when
--leave-temps is not enabled.
The best thing to do would be to make sure each parser uses unique
filenames and doesn't rely on cli_magic_scan_dir() to scan extracted
content before removing the extra subdirectory. In the meantime, this
commit gives the extra subdirectories meaningful names to improve
readability.
This commit also:
- Provides the 'bmp' prefix for extracted PE icons.
- Removes empty tmp subdirs when extracting rtf files, to eliminate
clutter.
- The PDF parser sometimes creates tmp files when decompressing streams
before it knows if there is actually any content to decompress. This
resulted in a large number of empty files. While it would be best to
avoid creating empty files in the first place, that's not quite as
as it sounds. This commit does the next best thing and deletes the
tmp files if nothing was actually extracted, even if --leave-temps is
enabled.
- Removes the "scantemp" prefix for unnamed fmaps scanned with
cli_magic_scan(). The 5-character hashes given to tmp files with
prefixes resulted in occasional file name collisions when extracting
certain file types with thousands of embedded files.
- The VBA and TAR parsers mistakenly used NAME_MAX instead of PATH_MAX,
resulting in truncated file paths and failed extraction when
--leave-temps is enabled and a lot of recursion is in play. This commit
switches them from NAME_MAX to PATH_MAX.
HTML normalization creates a tmp directory for storing rfc2397 style
links. The vast majority of html does not make use of rfc2397 and thus
an excess of empty tmp directories are generated. This commit alters
behavior to only create the rfc2397 directory when required if it does
not already exist.
Many of the core scanning functions' names no longer represent their
specific purpose or arguments. This commit aims to make the names more
intuitive. Names are now prefixed with "magic" if they involve
file-typing and file-type parsing. In addition, each function now
includes the type of input being scanned whether its "desc", "fmap", or
"buff". Some of the APIs also now specify "type" to indicate that a type
other than "ANY" may be passed in to select the type rather than use
file type magic for type recognition.
| current name | new name |
| ------------------------- | --------------------------------- |
| magic_scandesc() | cli_magic_scan() |
| cli_magic_scandesc_type() | <delete> |
| cli_magic_scandesc() | cli_magic_scan_desc() |
| cli_base_scandesc() | cli_magic_scan_desc_type() |
| cli_partition_scandesc() | <delete> |
| cli_map_scandesc() | magic_scan_nested_fmap_type() |
| cli_map_scan() | cli_magic_scan_nested_fmap_type() |
| cli_mem_scandesc() | cli_magic_scan_buff() |
| cli_scanbuff() | cli_scan_buff() |
| cli_scandesc() | cli_scan_desc() |
| cli_fmap_scandesc() | cli_scan_fmap() |
| cli_scanfile() | cli_magic_scan_file() |
| cli_scandir() | cli_magic_scan_dir() |
| cli_filetype2() | cli_determine_fmap_type() |
| cli_filetype() | cli_compare_ftm_file() |
| cli_partitiontype() | cli_compare_ftm_partition() |
| cli_scanraw() | scanraw() |
The metadata projecties JSON structure isn't recording file types found
embedded within a file such as self-extracting (SFX) types and office
document types (DOCX, PPTX, etc). This presents a problem...
At present there's no way to know if the current file has ended and a
few file is found tacked on to the end of the first file. If there
were, we could simply check if the type found by the raw-scan exists
within the first file, or after.
If within the first, and the type is an archive then it's reasonable to
conclude we're either observing zip headers (for SFXZIP detections) or
other files that are not compressed.
If the type ISN'T found within the first file, then we definitely have
whole new file to parse and we should do so with cli_magic_scan()
rather than only using these embedded type scanners.
At present we can't ignore SFXZIP detections even if the original file
type is a ZIP because we may have found two ZIPs appended together to
evade detection (a legitimate trick). As a consequence, we will
effectively parse every zip entry twice. The same issue applies to
types found within non-compressed archives.
This commit adds an EmbeddedObjects list to the metadata JSON object so
that the existance of these types is noted.
Additionally, this commit removes the two-part int64 cli_jsonint64()
implementation as json_object_new_int64() should be available
everywhere and the macro to detect such support was never set.
A way is needed to record scanned file names for two purposes:
1. File names (and extensions) must be stored in the json metadata
properties recorded when using the --gen-json clamscan option. Future
work may use this to compare file extensions with detected file types.
2. File names are useful when interpretting tmp directory output when
using the --leave-temps option.
This commit enables file name retention for later use by storing file
names in the fmap header structure, if a file name exists.
To store the names in fmaps, an optional name argument has been added to
any internal scan API's that create fmaps and every call to these APIs
has been modified to pass a file name or NULL if a file name is not
required. The zip and gpt parsers required some modification to record
file names. The NSIS and XAR parsers fail to collect file names at all
and will require future work to support file name extraction.
Also:
- Added recursive extraction to the tmp directory when the
--leave-temps option is enabled. When not enabled, the tmp directory
structure remains flat so as to prevent the likelihood of exceeding
MAX_PATH. The current tmp directory is stored in the scan context.
- Made the cli_scanfile() internal API non-static and added it to
scanners.h so it would be accessible outside of scanners.c in order to
remove code duplication within libmspack.c.
- Added function comments to scanners.h and matcher.h
- Converted a TDB-type macros and LSIG-type macros to enums for improved
type safey.
- Converted more return status variables from `int` to `cl_error_t` for
improved type safety, and corrected ooxml file typing functions so
they use `cli_file_t` exclusively rather than mixing types with
`cl_error_t`.
- Restructured the magic_scandesc() function to use goto's for error
handling and removed the early_ret_from_magicscan() macro and
magic_scandesc_cleanup() function. This makes the code easier to
read and made it easier to add the recursive tmp directory cleanup to
magic_scandesc().
- Corrected zip, egg, rar filename extraction issues.
- Removed use of extra sub-directory layer for zip, egg, and rar file
extraction. For Zip, this also involved changing the extracted
filenames to be randomly generated rather than using the "zip.###"
file name scheme.
This commit improves the layout of the tmp file output and the JSON
metadata output when using the --leave-temps and --gen-json options.
For all scans, each scan target will get a unique tmp sub-directory. If
using --leave-temps, that subdir will include the basename of the
original file to make it easier to identify. Additionally, when using
--leave-temps option, all extracted objects will have their
subdirectories extracted in recursive subdirectories including filename
prefixes where available. When not using the --leave-temps option, the
layout of the tmp sub-directory will remain flat, so as to alleviate the
possibility of exceeding PATH_MAX.
The JSON metadata generated by the --gen-json option is now generated
for all file types, not just a select few. The format is also
pretty-printed for readability and now includes filenames and file paths
when available.
Also:
- Added missing ALLMATCH check when determining if bytecode hooks should
be run.
- Added cl_engine_get_str API to windows libclamav symbol export file.
A missing return statement in png.c for a function that should return a
status code is resulting in undefined behavior.
In this patch I also added ".PNG" to one of the new heuristic signatures
to match others.
Add missing size checks to validate size data parsed from a VBA file.
This fixes a possible buffer overflow read that was caught by oss-fuzz
before it made it into any release.
Fix for an out-of-bounds read in the PDF parser when initializing
aes crypto routines that may result in a crash.
Bug found by OSS-Fuzz.
Also added checks for the arc4 init routine to mitigate the risk of a
similar issue.
Fix for an out-of-bounds read in the ARJ parser accidentally introduced
when adding text normalization and bound checking when parsing filename
and comment fields from file headers.
On some systems, the VirusEvent feature doubles the amount of RAM being used
because fork() duplicates the loaded signature database in the new process.
This commit changes fork() to vfork() so that VirusEvent won't fail if these systems
don't have enough memory.
Fixes a shell compatibility issue with string comparisons in the
clamonacc and libclamav-only M4 files:
test(1) uses `=` for string equality. (`==` is a bashism)
XLM is a macro language in Excel that was used before VBA (before
1996). It is still parsed and executed by modern Excel and is gaining
popularity with malware authors.
This patch adds rudimentary support for detecting and extracting
Excel 4.0 (XLM) macros.
The code is based on Didier Steven's plugin_biff for oletools.py.
Fixes a bug in the PtrVerifier pass when using LLVM >= v3.5 for the
bytecode signature runtime.
LLVM 3.5 changed the meaning of "use" and introduced "user". This fix
swaps out "use" keywords for "user" so the code functions correctly when
using LLVM 3.5+.
Add the credit card-only DLP option "StructuredCCOnly" to the win32
sample clamd config.
Also update NEWS.md to credit John Schember and Alexander Sulfrian for
the DLP CC-only mode contribution.
An integer overflow causes an out-of-bounds read that results in
a crash. The crash may occur when using the optional
Data-Loss-Prevention (DLP) feature to block content that contains credit
card numbers. This commit fixes the issue by using a signed index variable.
Add Data-Loss-Prevention option to detect credit cards only, excluding
debit and private label cards where possible.
You can select the credit card-only DLP mode for clamscan with the
`--structured-cc-mode` command-line option.
You can select the credit card-only DLP mode for clamd with the
`StructuredCCOnly` clamd.conf config option.
This patch also adds credit card matching for additional vendors:
- Mastercard 2016
- China Union Pay
- Discover 2009
Adds LZMA and BZip2 decompression routines to the bytecode API.
The ability to decompress LZMA and BZip2 streams is particularly
useful for bytecode signatures that extend clamav executable
unpacking capabilities.
Of note, the LZMA format is not well standardized. This API
expects the stream to start with the LZMA_Alone header.
Also fixed a bug in LZMA dictionary size setting.
- Existing VBA extraction code uses undocumented cache structures.
This code uses the documented way of accessing VBA projects.
- Adds additional detail to the dumped information:
Project name, Project doc string, ...
All VBA projects are dumped into a single file.
- Malware authors are currently evading detection by spreading
malicious code over several projects. It is hard to write
signatures if only part of the malicious code is visible.
Fixes an fmap leak in the bytecode switch_input() API. The
switch_input() API provides a way to read from an extracted file instead
of reading from the current file. The issue is that the current
implementation fails to free the fmap created to read from the extracted
file on cleanup or when switching back to the original fmap. In
addition, it fails to use the cli_bytecode_context_setfile() function
to restore the file_size in the context for the current fmap.
Fixes a couple fmap leaks in the unit tests.
Specifically this fixes use of cli_map_scandesc().
The cli_map_scandesc() function used to override the current fmap
settings with a new size and offset, performing a scan of the embedded
content. This broke the ability to iterate backwards through the fmap
recursion array when an alert occurs to check each map's hash for
whitelist matches.
In order to fix this issue, it needed to be possible to duplicate an
fmap header for the scan of the embedded file without duplicating the
actual map/data. This wasn't feasible with the posix fmap handle
implementation where the fmap header, bitmap array, and memory map
were all contiguouus. This commit makes it possible by extracting the
fmap header and bitmap array from the mmap region, using instead a
pointer for both the bitmap array and mmap/data. The resulting posix
fmap handle implementation as a result ended up working more similarly
to existing the Windows implementation.
In addition to the above changes, this commit fixes:
- fmap recursion tracking for cli_scandesc()
- a recursion tracking issue in cli_scanembpe() error handling
Andrew