mirror of https://github.com/postgres/postgres
generated text files. Fix build of that file, too. Put the text files in the right place during make dist, so there are no extra manual steps required anymore.REL8_2_STABLE
Peter Eisentraut
20 years ago
parent
9e0eff05ca
commit
0bc2a8ca65
@ -1,278 +0,0 @@ |
||||
|
||||
Regression Tests |
||||
|
||||
The regression tests are a comprehensive set of tests for the SQL |
||||
implementation in PostgreSQL. They test standard SQL operations as |
||||
well as the extended capabilities of PostgreSQL. |
||||
_________________________________________________________________ |
||||
|
||||
Running the Tests |
||||
|
||||
The regression tests can be run against an already installed and |
||||
running server, or using a temporary installation within the build |
||||
tree. Furthermore, there is a "parallel" and a "sequential" mode for |
||||
running the tests. The sequential method runs each test script in |
||||
turn, whereas the parallel method starts up multiple server processes |
||||
to run groups of tests in parallel. Parallel testing gives confidence |
||||
that interprocess communication and locking are working correctly. For |
||||
historical reasons, the sequential test is usually run against an |
||||
existing installation and the parallel method against a temporary |
||||
installation, but there are no technical reasons for this. |
||||
|
||||
To run the regression tests after building but before installation, |
||||
type |
||||
gmake check |
||||
|
||||
in the top-level directory. (Or you can change to "src/test/regress" |
||||
and run the command there.) This will first build several auxiliary |
||||
files, such as some sample user-defined trigger functions, and then |
||||
run the test driver script. At the end you should see something like |
||||
====================== |
||||
All 98 tests passed. |
||||
====================== |
||||
|
||||
or otherwise a note about which tests failed. See the section called |
||||
Test Evaluation below before assuming that a "failure" represents a |
||||
serious problem. |
||||
|
||||
Because this test method runs a temporary server, it will not work |
||||
when you are the root user (since the server will not start as root). |
||||
If you already did the build as root, you do not have to start all |
||||
over. Instead, make the regression test directory writable by some |
||||
other user, log in as that user, and restart the tests. For example |
||||
root# chmod -R a+w src/test/regress |
||||
root# chmod -R a+w contrib/spi |
||||
root# su - joeuser |
||||
joeuser$ cd top-level build directory |
||||
joeuser$ gmake check |
||||
|
||||
(The only possible "security risk" here is that other users might be |
||||
able to alter the regression test results behind your back. Use common |
||||
sense when managing user permissions.) |
||||
|
||||
Alternatively, run the tests after installation. |
||||
|
||||
If you have configured PostgreSQL to install into a location where an |
||||
older PostgreSQL installation already exists, and you perform gmake |
||||
check before installing the new version, you may find that the tests |
||||
fail because the new programs try to use the already-installed shared |
||||
libraries. (Typical symptoms are complaints about undefined symbols.) |
||||
If you wish to run the tests before overwriting the old installation, |
||||
you'll need to build with configure --disable-rpath. It is not |
||||
recommended that you use this option for the final installation, |
||||
however. |
||||
|
||||
The parallel regression test starts quite a few processes under your |
||||
user ID. Presently, the maximum concurrency is twenty parallel test |
||||
scripts, which means sixty processes: there's a server process, a |
||||
psql, and usually a shell parent process for the psql for each test |
||||
script. So if your system enforces a per-user limit on the number of |
||||
processes, make sure this limit is at least seventy-five or so, else |
||||
you may get random-seeming failures in the parallel test. If you are |
||||
not in a position to raise the limit, you can cut down the degree of |
||||
parallelism by setting the MAX_CONNECTIONS parameter. For example, |
||||
gmake MAX_CONNECTIONS=10 check |
||||
|
||||
runs no more than ten tests concurrently. |
||||
|
||||
On some systems, the default Bourne-compatible shell ("/bin/sh") gets |
||||
confused when it has to manage too many child processes in parallel. |
||||
This may cause the parallel test run to lock up or fail. In such |
||||
cases, specify a different Bourne-compatible shell on the command |
||||
line, for example: |
||||
gmake SHELL=/bin/ksh check |
||||
|
||||
If no non-broken shell is available, you may be able to work around |
||||
the problem by limiting the number of connections, as shown above. |
||||
|
||||
To run the tests after installation, initialize a data area and start |
||||
the server, then type |
||||
gmake installcheck |
||||
|
||||
or for a parallel test |
||||
gmake installcheck-parallel |
||||
|
||||
The tests will expect to contact the server at the local host and the |
||||
default port number, unless directed otherwise by PGHOST and PGPORT |
||||
environment variables. |
||||
_________________________________________________________________ |
||||
|
||||
Test Evaluation |
||||
|
||||
Some properly installed and fully functional PostgreSQL installations |
||||
can "fail" some of these regression tests due to platform-specific |
||||
artifacts such as varying floating-point representation and message |
||||
wording. The tests are currently evaluated using a simple "diff" |
||||
comparison against the outputs generated on a reference system, so the |
||||
results are sensitive to small system differences. When a test is |
||||
reported as "failed", always examine the differences between expected |
||||
and actual results; you may well find that the differences are not |
||||
significant. Nonetheless, we still strive to maintain accurate |
||||
reference files across all supported platforms, so it can be expected |
||||
that all tests pass. |
||||
|
||||
The actual outputs of the regression tests are in files in the |
||||
"src/test/regress/results" directory. The test script uses "diff" to |
||||
compare each output file against the reference outputs stored in the |
||||
"src/test/regress/expected" directory. Any differences are saved for |
||||
your inspection in "src/test/regress/regression.diffs". (Or you can |
||||
run "diff" yourself, if you prefer.) |
||||
|
||||
If for some reason a particular platform generates a "failure" for a |
||||
given test, but inspection of the output convinces you that the result |
||||
is valid, you can add a new comparison file to silence the failure |
||||
report in future test runs. See the section called Variant Comparison |
||||
Files for details. |
||||
_________________________________________________________________ |
||||
|
||||
Error message differences |
||||
|
||||
Some of the regression tests involve intentional invalid input values. |
||||
Error messages can come from either the PostgreSQL code or from the |
||||
host platform system routines. In the latter case, the messages may |
||||
vary between platforms, but should reflect similar information. These |
||||
differences in messages will result in a "failed" regression test that |
||||
can be validated by inspection. |
||||
_________________________________________________________________ |
||||
|
||||
Locale differences |
||||
|
||||
If you run the tests against an already-installed server that was |
||||
initialized with a collation-order locale other than C, then there may |
||||
be differences due to sort order and follow-up failures. The |
||||
regression test suite is set up to handle this problem by providing |
||||
alternative result files that together are known to handle a large |
||||
number of locales. |
||||
_________________________________________________________________ |
||||
|
||||
Date and time differences |
||||
|
||||
Most of the date and time results are dependent on the time zone |
||||
environment. The reference files are generated for time zone PST8PDT |
||||
(Berkeley, California), and there will be apparent failures if the |
||||
tests are not run with that time zone setting. The regression test |
||||
driver sets environment variable PGTZ to PST8PDT, which normally |
||||
ensures proper results. |
||||
_________________________________________________________________ |
||||
|
||||
Floating-point differences |
||||
|
||||
Some of the tests involve computing 64-bit floating-point numbers |
||||
(double precision) from table columns. Differences in results |
||||
involving mathematical functions of double precision columns have been |
||||
observed. The float8 and geometry tests are particularly prone to |
||||
small differences across platforms, or even with different compiler |
||||
optimization options. Human eyeball comparison is needed to determine |
||||
the real significance of these differences which are usually 10 places |
||||
to the right of the decimal point. |
||||
|
||||
Some systems display minus zero as -0, while others just show 0. |
||||
|
||||
Some systems signal errors from pow() and exp() differently from the |
||||
mechanism expected by the current PostgreSQL code. |
||||
_________________________________________________________________ |
||||
|
||||
Row ordering differences |
||||
|
||||
You might see differences in which the same rows are output in a |
||||
different order than what appears in the expected file. In most cases |
||||
this is not, strictly speaking, a bug. Most of the regression test |
||||
scripts are not so pedantic as to use an ORDER BY for every single |
||||
SELECT, and so their result row orderings are not well-defined |
||||
according to the letter of the SQL specification. In practice, since |
||||
we are looking at the same queries being executed on the same data by |
||||
the same software, we usually get the same result ordering on all |
||||
platforms, and so the lack of ORDER BY isn't a problem. Some queries |
||||
do exhibit cross-platform ordering differences, however. When testing |
||||
against an already-installed server, ordering differences can also be |
||||
caused by non-C locale settings or non-default parameter settings, |
||||
such as custom values of work_mem or the planner cost parameters. |
||||
|
||||
Therefore, if you see an ordering difference, it's not something to |
||||
worry about, unless the query does have an ORDER BY that your result |
||||
is violating. But please report it anyway, so that we can add an ORDER |
||||
BY to that particular query and thereby eliminate the bogus "failure" |
||||
in future releases. |
||||
|
||||
You might wonder why we don't order all the regression test queries |
||||
explicitly to get rid of this issue once and for all. The reason is |
||||
that that would make the regression tests less useful, not more, since |
||||
they'd tend to exercise query plan types that produce ordered results |
||||
to the exclusion of those that don't. |
||||
_________________________________________________________________ |
||||
|
||||
The "random" test |
||||
|
||||
The random test script is intended to produce random results. In rare |
||||
cases, this causes the random regression test to fail. Typing |
||||
diff results/random.out expected/random.out |
||||
|
||||
should produce only one or a few lines of differences. You need not |
||||
worry unless the random test fails repeatedly. |
||||
_________________________________________________________________ |
||||
|
||||
Variant Comparison Files |
||||
|
||||
Since some of the tests inherently produce environment-dependent |
||||
results, we have provided ways to specify alternative "expected" |
||||
result files. Each regression test can have several comparison files |
||||
showing possible results on different platforms. There are two |
||||
independent mechanisms for determining which comparison file is used |
||||
for each test. |
||||
|
||||
The first mechanism allows comparison files to be selected for |
||||
specific platforms. There is a mapping file, |
||||
"src/test/regress/resultmap", that defines which comparison file to |
||||
use for each platform. To eliminate bogus test "failures" for a |
||||
particular platform, you first choose or make a variant result file, |
||||
and then add a line to the "resultmap" file. |
||||
|
||||
Each line in the mapping file is of the form |
||||
testname/platformpattern=comparisonfilename |
||||
|
||||
The test name is just the name of the particular regression test |
||||
module. The platform pattern is a pattern in the style of the Unix |
||||
tool "expr" (that is, a regular expression with an implicit ^ anchor |
||||
at the start). It is matched against the platform name as printed by |
||||
"config.guess" followed by :gcc or :cc, depending on whether you use |
||||
the GNU compiler or the system's native compiler (on systems where |
||||
there is a difference). The comparison file name is the base name of |
||||
the substitute result comparison file. |
||||
|
||||
For example: some systems interpret very small floating-point values |
||||
as zero, rather than reporting an underflow error. This causes a few |
||||
differences in the "float8" regression test. Therefore, we provide a |
||||
variant comparison file, "float8-small-is-zero.out", which includes |
||||
the results to be expected on these systems. To silence the bogus |
||||
"failure" message on OpenBSD platforms, "resultmap" includes |
||||
float8/i.86-.*-openbsd=float8-small-is-zero |
||||
|
||||
which will trigger on any machine for which the output of |
||||
"config.guess" matches i.86-.*-openbsd. Other lines in "resultmap" |
||||
select the variant comparison file for other platforms where it's |
||||
appropriate. |
||||
|
||||
The second selection mechanism for variant comparison files is much |
||||
more automatic: it simply uses the "best match" among several supplied |
||||
comparison files. The regression test driver script considers both the |
||||
standard comparison file for a test, testname.out, and variant files |
||||
named testname_digit.out (where the "digit" is any single digit 0-9). |
||||
If any such file is an exact match, the test is considered to pass; |
||||
otherwise, the one that generates the shortest diff is used to create |
||||
the failure report. (If "resultmap" includes an entry for the |
||||
particular test, then the base "testname" is the substitute name given |
||||
in "resultmap".) |
||||
|
||||
For example, for the char test, the comparison file "char.out" |
||||
contains results that are expected in the C and POSIX locales, while |
||||
the file "char_1.out" contains results sorted as they appear in many |
||||
other locales. |
||||
|
||||
The best-match mechanism was devised to cope with locale-dependent |
||||
results, but it can be used in any situation where the test results |
||||
cannot be predicted easily from the platform name alone. A limitation |
||||
of this mechanism is that the test driver cannot tell which variant is |
||||
actually "correct" for the current environment; it will just pick the |
||||
variant that seems to work best. Therefore it is safest to use this |
||||
mechanism only for variant results that you are willing to consider |
||||
equally valid in all contexts. |
||||
Loading…
Reference in new issue