mirror of https://github.com/postgres/postgres
Follow on patch in the multi-variate statistics patch series.
CREATE STATISTICS s1 WITH (dependencies) ON (a, b) FROM t;
ANALYZE;
will collect dependency stats on (a, b) and then use the measured
dependency in subsequent query planning.
Commit 7b504eb282 added
CREATE STATISTICS with n-distinct coefficients. These are now
specified using the mutually exclusive option WITH (ndistinct).
Author: Tomas Vondra, David Rowley
Reviewed-by: Kyotaro HORIGUCHI, Álvaro Herrera, Dean Rasheed, Robert Haas
and many other comments and contributions
Discussion: https://postgr.es/m/56f40b20-c464-fad2-ff39-06b668fac47c@2ndquadrant.com
pull/3/merge
Simon Riggs
9 years ago
parent
00b6b6feb1
commit
2686ee1b7c
@ -0,0 +1,119 @@ |
||||
Soft functional dependencies |
||||
============================ |
||||
|
||||
Functional dependencies are a concept well described in relational theory, |
||||
particularly in the definition of normalization and "normal forms". Wikipedia |
||||
has a nice definition of a functional dependency [1]: |
||||
|
||||
In a given table, an attribute Y is said to have a functional dependency |
||||
on a set of attributes X (written X -> Y) if and only if each X value is |
||||
associated with precisely one Y value. For example, in an "Employee" |
||||
table that includes the attributes "Employee ID" and "Employee Date of |
||||
Birth", the functional dependency |
||||
|
||||
{Employee ID} -> {Employee Date of Birth} |
||||
|
||||
would hold. It follows from the previous two sentences that each |
||||
{Employee ID} is associated with precisely one {Employee Date of Birth}. |
||||
|
||||
[1] https://en.wikipedia.org/wiki/Functional_dependency |
||||
|
||||
In practical terms, functional dependencies mean that a value in one column |
||||
determines values in some other column. Consider for example this trivial |
||||
table with two integer columns: |
||||
|
||||
CREATE TABLE t (a INT, b INT) |
||||
AS SELECT i, i/10 FROM generate_series(1,100000) s(i); |
||||
|
||||
Clearly, knowledge of the value in column 'a' is sufficient to determine the |
||||
value in column 'b', as it's simply (a/10). A more practical example may be |
||||
addresses, where the knowledge of a ZIP code (usually) determines city. Larger |
||||
cities may have multiple ZIP codes, so the dependency can't be reversed. |
||||
|
||||
Many datasets might be normalized not to contain such dependencies, but often |
||||
it's not practical for various reasons. In some cases, it's actually a conscious |
||||
design choice to model the dataset in a denormalized way, either because of |
||||
performance or to make querying easier. |
||||
|
||||
|
||||
Soft dependencies |
||||
----------------- |
||||
|
||||
Real-world data sets often contain data errors, either because of data entry |
||||
mistakes (user mistyping the ZIP code) or perhaps issues in generating the |
||||
data (e.g. a ZIP code mistakenly assigned to two cities in different states). |
||||
|
||||
A strict implementation would either ignore dependencies in such cases, |
||||
rendering the approach mostly useless even for slightly noisy data sets, or |
||||
result in sudden changes in behavior depending on minor differences between |
||||
samples provided to ANALYZE. |
||||
|
||||
For this reason, the statistics implements "soft" functional dependencies, |
||||
associating each functional dependency with a degree of validity (a number |
||||
between 0 and 1). This degree is then used to combine selectivities in a |
||||
smooth manner. |
||||
|
||||
|
||||
Mining dependencies (ANALYZE) |
||||
----------------------------- |
||||
|
||||
The current algorithm is fairly simple - generate all possible functional |
||||
dependencies, and for each one count the number of rows consistent with it. |
||||
Then use the fraction of rows (supporting/total) as the degree. |
||||
|
||||
To count the rows consistent with the dependency (a => b): |
||||
|
||||
(a) Sort the data lexicographically, i.e. first by 'a' then 'b'. |
||||
|
||||
(b) For each group of rows with the same 'a' value, count the number of |
||||
distinct values in 'b'. |
||||
|
||||
(c) If there's a single distinct value in 'b', the rows are consistent with |
||||
the functional dependency, otherwise they contradict it. |
||||
|
||||
The algorithm also requires a minimum size of the group to consider it |
||||
consistent (currently 3 rows in the sample). Small groups make it less likely |
||||
to break the consistency. |
||||
|
||||
|
||||
Clause reduction (planner/optimizer) |
||||
------------------------------------ |
||||
|
||||
Applying the functional dependencies is fairly simple - given a list of |
||||
equality clauses, we compute selectivities of each clause and then use the |
||||
degree to combine them using this formula |
||||
|
||||
P(a=?,b=?) = P(a=?) * (d + (1-d) * P(b=?)) |
||||
|
||||
Where 'd' is the degree of functional dependence (a=>b). |
||||
|
||||
With more than two equality clauses, this process happens recursively. For |
||||
example for (a,b,c) we first use (a,b=>c) to break the computation into |
||||
|
||||
P(a=?,b=?,c=?) = P(a=?,b=?) * (d + (1-d)*P(b=?)) |
||||
|
||||
and then apply (a=>b) the same way on P(a=?,b=?). |
||||
|
||||
|
||||
Consistency of clauses |
||||
---------------------- |
||||
|
||||
Functional dependencies only express general dependencies between columns, |
||||
without referencing particular values. This assumes that the equality clauses |
||||
are in fact consistent with the functional dependency, i.e. that given a |
||||
dependency (a=>b), the value in (b=?) clause is the value determined by (a=?). |
||||
If that's not the case, the clauses are "inconsistent" with the functional |
||||
dependency and the result will be over-estimation. |
||||
|
||||
This may happen, for example, when using conditions on the ZIP code and city |
||||
name with mismatching values (ZIP code for a different city), etc. In such a |
||||
case, the result set will be empty, but we'll estimate the selectivity using |
||||
the ZIP code condition. |
||||
|
||||
In this case, the default estimation based on AVIA principle happens to work |
||||
better, but mostly by chance. |
||||
|
||||
This issue is the price for the simplicity of functional dependencies. If the |
||||
application frequently constructs queries with clauses inconsistent with |
||||
functional dependencies present in the data, the best solution is not to |
||||
use functional dependencies, but one of the more complex types of statistics. |
||||
File diff suppressed because it is too large
Load Diff
Loading…
Reference in new issue