mirror of https://github.com/postgres/postgres
generated text files. Fix build of that file, too. Put the text files in the right place during make dist, so there are no extra manual steps required anymore.REL8_2_STABLE
Peter Eisentraut
20 years ago
parent
9e0eff05ca
commit
0bc2a8ca65
@ -1,278 +0,0 @@ |
|||||||
|
|
||||||
Regression Tests |
|
||||||
|
|
||||||
The regression tests are a comprehensive set of tests for the SQL |
|
||||||
implementation in PostgreSQL. They test standard SQL operations as |
|
||||||
well as the extended capabilities of PostgreSQL. |
|
||||||
_________________________________________________________________ |
|
||||||
|
|
||||||
Running the Tests |
|
||||||
|
|
||||||
The regression tests can be run against an already installed and |
|
||||||
running server, or using a temporary installation within the build |
|
||||||
tree. Furthermore, there is a "parallel" and a "sequential" mode for |
|
||||||
running the tests. The sequential method runs each test script in |
|
||||||
turn, whereas the parallel method starts up multiple server processes |
|
||||||
to run groups of tests in parallel. Parallel testing gives confidence |
|
||||||
that interprocess communication and locking are working correctly. For |
|
||||||
historical reasons, the sequential test is usually run against an |
|
||||||
existing installation and the parallel method against a temporary |
|
||||||
installation, but there are no technical reasons for this. |
|
||||||
|
|
||||||
To run the regression tests after building but before installation, |
|
||||||
type |
|
||||||
gmake check |
|
||||||
|
|
||||||
in the top-level directory. (Or you can change to "src/test/regress" |
|
||||||
and run the command there.) This will first build several auxiliary |
|
||||||
files, such as some sample user-defined trigger functions, and then |
|
||||||
run the test driver script. At the end you should see something like |
|
||||||
====================== |
|
||||||
All 98 tests passed. |
|
||||||
====================== |
|
||||||
|
|
||||||
or otherwise a note about which tests failed. See the section called |
|
||||||
Test Evaluation below before assuming that a "failure" represents a |
|
||||||
serious problem. |
|
||||||
|
|
||||||
Because this test method runs a temporary server, it will not work |
|
||||||
when you are the root user (since the server will not start as root). |
|
||||||
If you already did the build as root, you do not have to start all |
|
||||||
over. Instead, make the regression test directory writable by some |
|
||||||
other user, log in as that user, and restart the tests. For example |
|
||||||
root# chmod -R a+w src/test/regress |
|
||||||
root# chmod -R a+w contrib/spi |
|
||||||
root# su - joeuser |
|
||||||
joeuser$ cd top-level build directory |
|
||||||
joeuser$ gmake check |
|
||||||
|
|
||||||
(The only possible "security risk" here is that other users might be |
|
||||||
able to alter the regression test results behind your back. Use common |
|
||||||
sense when managing user permissions.) |
|
||||||
|
|
||||||
Alternatively, run the tests after installation. |
|
||||||
|
|
||||||
If you have configured PostgreSQL to install into a location where an |
|
||||||
older PostgreSQL installation already exists, and you perform gmake |
|
||||||
check before installing the new version, you may find that the tests |
|
||||||
fail because the new programs try to use the already-installed shared |
|
||||||
libraries. (Typical symptoms are complaints about undefined symbols.) |
|
||||||
If you wish to run the tests before overwriting the old installation, |
|
||||||
you'll need to build with configure --disable-rpath. It is not |
|
||||||
recommended that you use this option for the final installation, |
|
||||||
however. |
|
||||||
|
|
||||||
The parallel regression test starts quite a few processes under your |
|
||||||
user ID. Presently, the maximum concurrency is twenty parallel test |
|
||||||
scripts, which means sixty processes: there's a server process, a |
|
||||||
psql, and usually a shell parent process for the psql for each test |
|
||||||
script. So if your system enforces a per-user limit on the number of |
|
||||||
processes, make sure this limit is at least seventy-five or so, else |
|
||||||
you may get random-seeming failures in the parallel test. If you are |
|
||||||
not in a position to raise the limit, you can cut down the degree of |
|
||||||
parallelism by setting the MAX_CONNECTIONS parameter. For example, |
|
||||||
gmake MAX_CONNECTIONS=10 check |
|
||||||
|
|
||||||
runs no more than ten tests concurrently. |
|
||||||
|
|
||||||
On some systems, the default Bourne-compatible shell ("/bin/sh") gets |
|
||||||
confused when it has to manage too many child processes in parallel. |
|
||||||
This may cause the parallel test run to lock up or fail. In such |
|
||||||
cases, specify a different Bourne-compatible shell on the command |
|
||||||
line, for example: |
|
||||||
gmake SHELL=/bin/ksh check |
|
||||||
|
|
||||||
If no non-broken shell is available, you may be able to work around |
|
||||||
the problem by limiting the number of connections, as shown above. |
|
||||||
|
|
||||||
To run the tests after installation, initialize a data area and start |
|
||||||
the server, then type |
|
||||||
gmake installcheck |
|
||||||
|
|
||||||
or for a parallel test |
|
||||||
gmake installcheck-parallel |
|
||||||
|
|
||||||
The tests will expect to contact the server at the local host and the |
|
||||||
default port number, unless directed otherwise by PGHOST and PGPORT |
|
||||||
environment variables. |
|
||||||
_________________________________________________________________ |
|
||||||
|
|
||||||
Test Evaluation |
|
||||||
|
|
||||||
Some properly installed and fully functional PostgreSQL installations |
|
||||||
can "fail" some of these regression tests due to platform-specific |
|
||||||
artifacts such as varying floating-point representation and message |
|
||||||
wording. The tests are currently evaluated using a simple "diff" |
|
||||||
comparison against the outputs generated on a reference system, so the |
|
||||||
results are sensitive to small system differences. When a test is |
|
||||||
reported as "failed", always examine the differences between expected |
|
||||||
and actual results; you may well find that the differences are not |
|
||||||
significant. Nonetheless, we still strive to maintain accurate |
|
||||||
reference files across all supported platforms, so it can be expected |
|
||||||
that all tests pass. |
|
||||||
|
|
||||||
The actual outputs of the regression tests are in files in the |
|
||||||
"src/test/regress/results" directory. The test script uses "diff" to |
|
||||||
compare each output file against the reference outputs stored in the |
|
||||||
"src/test/regress/expected" directory. Any differences are saved for |
|
||||||
your inspection in "src/test/regress/regression.diffs". (Or you can |
|
||||||
run "diff" yourself, if you prefer.) |
|
||||||
|
|
||||||
If for some reason a particular platform generates a "failure" for a |
|
||||||
given test, but inspection of the output convinces you that the result |
|
||||||
is valid, you can add a new comparison file to silence the failure |
|
||||||
report in future test runs. See the section called Variant Comparison |
|
||||||
Files for details. |
|
||||||
_________________________________________________________________ |
|
||||||
|
|
||||||
Error message differences |
|
||||||
|
|
||||||
Some of the regression tests involve intentional invalid input values. |
|
||||||
Error messages can come from either the PostgreSQL code or from the |
|
||||||
host platform system routines. In the latter case, the messages may |
|
||||||
vary between platforms, but should reflect similar information. These |
|
||||||
differences in messages will result in a "failed" regression test that |
|
||||||
can be validated by inspection. |
|
||||||
_________________________________________________________________ |
|
||||||
|
|
||||||
Locale differences |
|
||||||
|
|
||||||
If you run the tests against an already-installed server that was |
|
||||||
initialized with a collation-order locale other than C, then there may |
|
||||||
be differences due to sort order and follow-up failures. The |
|
||||||
regression test suite is set up to handle this problem by providing |
|
||||||
alternative result files that together are known to handle a large |
|
||||||
number of locales. |
|
||||||
_________________________________________________________________ |
|
||||||
|
|
||||||
Date and time differences |
|
||||||
|
|
||||||
Most of the date and time results are dependent on the time zone |
|
||||||
environment. The reference files are generated for time zone PST8PDT |
|
||||||
(Berkeley, California), and there will be apparent failures if the |
|
||||||
tests are not run with that time zone setting. The regression test |
|
||||||
driver sets environment variable PGTZ to PST8PDT, which normally |
|
||||||
ensures proper results. |
|
||||||
_________________________________________________________________ |
|
||||||
|
|
||||||
Floating-point differences |
|
||||||
|
|
||||||
Some of the tests involve computing 64-bit floating-point numbers |
|
||||||
(double precision) from table columns. Differences in results |
|
||||||
involving mathematical functions of double precision columns have been |
|
||||||
observed. The float8 and geometry tests are particularly prone to |
|
||||||
small differences across platforms, or even with different compiler |
|
||||||
optimization options. Human eyeball comparison is needed to determine |
|
||||||
the real significance of these differences which are usually 10 places |
|
||||||
to the right of the decimal point. |
|
||||||
|
|
||||||
Some systems display minus zero as -0, while others just show 0. |
|
||||||
|
|
||||||
Some systems signal errors from pow() and exp() differently from the |
|
||||||
mechanism expected by the current PostgreSQL code. |
|
||||||
_________________________________________________________________ |
|
||||||
|
|
||||||
Row ordering differences |
|
||||||
|
|
||||||
You might see differences in which the same rows are output in a |
|
||||||
different order than what appears in the expected file. In most cases |
|
||||||
this is not, strictly speaking, a bug. Most of the regression test |
|
||||||
scripts are not so pedantic as to use an ORDER BY for every single |
|
||||||
SELECT, and so their result row orderings are not well-defined |
|
||||||
according to the letter of the SQL specification. In practice, since |
|
||||||
we are looking at the same queries being executed on the same data by |
|
||||||
the same software, we usually get the same result ordering on all |
|
||||||
platforms, and so the lack of ORDER BY isn't a problem. Some queries |
|
||||||
do exhibit cross-platform ordering differences, however. When testing |
|
||||||
against an already-installed server, ordering differences can also be |
|
||||||
caused by non-C locale settings or non-default parameter settings, |
|
||||||
such as custom values of work_mem or the planner cost parameters. |
|
||||||
|
|
||||||
Therefore, if you see an ordering difference, it's not something to |
|
||||||
worry about, unless the query does have an ORDER BY that your result |
|
||||||
is violating. But please report it anyway, so that we can add an ORDER |
|
||||||
BY to that particular query and thereby eliminate the bogus "failure" |
|
||||||
in future releases. |
|
||||||
|
|
||||||
You might wonder why we don't order all the regression test queries |
|
||||||
explicitly to get rid of this issue once and for all. The reason is |
|
||||||
that that would make the regression tests less useful, not more, since |
|
||||||
they'd tend to exercise query plan types that produce ordered results |
|
||||||
to the exclusion of those that don't. |
|
||||||
_________________________________________________________________ |
|
||||||
|
|
||||||
The "random" test |
|
||||||
|
|
||||||
The random test script is intended to produce random results. In rare |
|
||||||
cases, this causes the random regression test to fail. Typing |
|
||||||
diff results/random.out expected/random.out |
|
||||||
|
|
||||||
should produce only one or a few lines of differences. You need not |
|
||||||
worry unless the random test fails repeatedly. |
|
||||||
_________________________________________________________________ |
|
||||||
|
|
||||||
Variant Comparison Files |
|
||||||
|
|
||||||
Since some of the tests inherently produce environment-dependent |
|
||||||
results, we have provided ways to specify alternative "expected" |
|
||||||
result files. Each regression test can have several comparison files |
|
||||||
showing possible results on different platforms. There are two |
|
||||||
independent mechanisms for determining which comparison file is used |
|
||||||
for each test. |
|
||||||
|
|
||||||
The first mechanism allows comparison files to be selected for |
|
||||||
specific platforms. There is a mapping file, |
|
||||||
"src/test/regress/resultmap", that defines which comparison file to |
|
||||||
use for each platform. To eliminate bogus test "failures" for a |
|
||||||
particular platform, you first choose or make a variant result file, |
|
||||||
and then add a line to the "resultmap" file. |
|
||||||
|
|
||||||
Each line in the mapping file is of the form |
|
||||||
testname/platformpattern=comparisonfilename |
|
||||||
|
|
||||||
The test name is just the name of the particular regression test |
|
||||||
module. The platform pattern is a pattern in the style of the Unix |
|
||||||
tool "expr" (that is, a regular expression with an implicit ^ anchor |
|
||||||
at the start). It is matched against the platform name as printed by |
|
||||||
"config.guess" followed by :gcc or :cc, depending on whether you use |
|
||||||
the GNU compiler or the system's native compiler (on systems where |
|
||||||
there is a difference). The comparison file name is the base name of |
|
||||||
the substitute result comparison file. |
|
||||||
|
|
||||||
For example: some systems interpret very small floating-point values |
|
||||||
as zero, rather than reporting an underflow error. This causes a few |
|
||||||
differences in the "float8" regression test. Therefore, we provide a |
|
||||||
variant comparison file, "float8-small-is-zero.out", which includes |
|
||||||
the results to be expected on these systems. To silence the bogus |
|
||||||
"failure" message on OpenBSD platforms, "resultmap" includes |
|
||||||
float8/i.86-.*-openbsd=float8-small-is-zero |
|
||||||
|
|
||||||
which will trigger on any machine for which the output of |
|
||||||
"config.guess" matches i.86-.*-openbsd. Other lines in "resultmap" |
|
||||||
select the variant comparison file for other platforms where it's |
|
||||||
appropriate. |
|
||||||
|
|
||||||
The second selection mechanism for variant comparison files is much |
|
||||||
more automatic: it simply uses the "best match" among several supplied |
|
||||||
comparison files. The regression test driver script considers both the |
|
||||||
standard comparison file for a test, testname.out, and variant files |
|
||||||
named testname_digit.out (where the "digit" is any single digit 0-9). |
|
||||||
If any such file is an exact match, the test is considered to pass; |
|
||||||
otherwise, the one that generates the shortest diff is used to create |
|
||||||
the failure report. (If "resultmap" includes an entry for the |
|
||||||
particular test, then the base "testname" is the substitute name given |
|
||||||
in "resultmap".) |
|
||||||
|
|
||||||
For example, for the char test, the comparison file "char.out" |
|
||||||
contains results that are expected in the C and POSIX locales, while |
|
||||||
the file "char_1.out" contains results sorted as they appear in many |
|
||||||
other locales. |
|
||||||
|
|
||||||
The best-match mechanism was devised to cope with locale-dependent |
|
||||||
results, but it can be used in any situation where the test results |
|
||||||
cannot be predicted easily from the platform name alone. A limitation |
|
||||||
of this mechanism is that the test driver cannot tell which variant is |
|
||||||
actually "correct" for the current environment; it will just pick the |
|
||||||
variant that seems to work best. Therefore it is safest to use this |
|
||||||
mechanism only for variant results that you are willing to consider |
|
||||||
equally valid in all contexts. |
|
||||||
Loading…
Reference in new issue