This is a replacement for the existing --with-openssl, extending the
logic to make easier the addition of new SSL libraries. The grammar is
chosen to be similar to --with-uuid, where multiple values can be
chosen, with "openssl" as the only supported value for now.
The original switch, --with-openssl, is kept for compatibility.
Author: Daniel Gustafsson, Michael Paquier
Reviewed-by: Jacob Champion
Discussion: https://postgr.es/m/FAB21FC8-0F62-434F-AA78-6BD9336D630A@yesql.se
With this commit, SHA1 goes through the implementation provided by
OpenSSL via EVP when building the backend with it, and uses as fallback
implementation KAME which was located in pgcrypto and already shaped for
an integration with a set of init, update and final routines.
Structures and routines have been renamed to make things consistent with
the fallback implementations of MD5 and SHA2.
uuid-ossp has used for ages a shortcut with pgcrypto to fetch a copy of
SHA1 if needed. This was built depending on the build options within
./configure, so this cleans up some code and removes the build
dependency between pgcrypto and uuid-ossp.
Note that this will help with the refactoring of HMAC, as pgcrypto
offers the option to use MD5, SHA1 or SHA2, so only the second option
was missing to make that possible.
Author: Michael Paquier
Reviewed-by: Heikki Linnakangas
Discussion: https://postgr.es/m/X9HXKTgrvJvYO7Oh@paquier.xyz
This commit heavily reorganizes the MD5 implementations that exist in
the tree in various aspects.
First, MD5 is added to the list of options available in cryptohash.c and
cryptohash_openssl.c. This means that if building with OpenSSL, EVP is
used for MD5 instead of the fallback implementation that Postgres had
for ages. With the recent refactoring work for cryptohash functions,
this change is straight-forward. If not building with OpenSSL, a
fallback implementation internal to src/common/ is used.
Second, this reduces the number of MD5 implementations present in the
tree from two to one, by moving the KAME implementation from pgcrypto to
src/common/, and by removing the implementation that existed in
src/common/. KAME was already structured with an init/update/final set
of routines by pgcrypto (see original pgcrypto/md5.h) for compatibility
with OpenSSL, so moving it to src/common/ has proved to be a
straight-forward move, requiring no actual manipulation of the internals
of each routine. Some benchmarking has not shown any performance gap
between both implementations.
Similarly to the fallback implementation used for SHA2, the fallback
implementation of MD5 is moved to src/common/md5.c with an internal
header called md5_int.h for the init, update and final routines. This
gets then consumed by cryptohash.c.
The original routines used for MD5-hashed passwords are moved to a
separate file called md5_common.c, also in src/common/, aimed at being
shared between all MD5 implementations as utility routines to keep
compatibility with any code relying on them.
Like the SHA2 changes, this commit had its round of tests on both Linux
and Windows, across all versions of OpenSSL supported on HEAD, with and
even without OpenSSL.
Author: Michael Paquier
Reviewed-by: Daniel Gustafsson
Discussion: https://postgr.es/m/20201106073434.GA4961@paquier.xyz
Andres Freund pointed out that allowing non-superusers to run
"CREATE EXTENSION ... FROM unpackaged" has security risks, since
the unpackaged-to-1.0 scripts don't try to verify that the existing
objects they're modifying are what they expect. Just attaching such
objects to an extension doesn't seem too dangerous, but some of them
do more than that.
We could have resolved this, perhaps, by still requiring superuser
privilege to use the FROM option. However, it's fair to ask just what
we're accomplishing by continuing to lug the unpackaged-to-1.0 scripts
forward. None of them have received any real testing since 9.1 days,
so they may not even work anymore (even assuming that one could still
load the previous "loose" object definitions into a v13 database).
And an installation that's trying to go from pre-9.1 to v13 or later
in one jump is going to have worse compatibility problems than whether
there's a trivial way to convert their contrib modules into extension
style.
Hence, let's just drop both those scripts and the core-code support
for "CREATE EXTENSION ... FROM".
Discussion: https://postgr.es/m/20200213233015.r6rnubcvl4egdh5r@alap3.anarazel.de
When maintaining or merging patches, one of the most common sources
for conflicts are the list of objects in makefiles. Especially when
the split across lines has been changed on both sides, which is
somewhat common due to attempting to stay below 80 columns, those
conflicts are unnecessarily laborious to resolve.
By splitting, and alphabetically sorting, OBJS style lines into one
object per line, conflicts should be less frequent, and easier to
resolve when they still occur.
Author: Andres Freund
Discussion: https://postgr.es/m/20191029200901.vww4idgcxv74cwes@alap3.anarazel.de
Upstream fixed bugs over the years, but none are confirmed to have
affected pgcrypto. We're better off naively tracking upstream than
reactively maintaining a twelve-year-old snapshot of upstream. Add a
header comment describing the synchronization procedure. Discard use of
INVERT_COMPARE_RESULT(); the domain of the comparisons in question is
{-1,0,1}, controlled entirely by code in imath.c.
Andrew Gierth reviewed the Makefile change. Tom Lane reviewed the
synchronization procedure description.
Discussion: https://postgr.es/m/20190203035704.GA6226@rfd.leadboat.com
This introduces a new generic SASL authentication method, similar to the
GSS and SSPI methods. The server first tells the client which SASL
authentication mechanism to use, and then the mechanism-specific SASL
messages are exchanged in AuthenticationSASLcontinue and PasswordMessage
messages. Only SCRAM-SHA-256 is supported at the moment, but this allows
adding more SASL mechanisms in the future, without changing the overall
protocol.
Support for channel binding, aka SCRAM-SHA-256-PLUS is left for later.
The SASLPrep algorithm, for pre-processing the password, is not yet
implemented. That could cause trouble, if you use a password with
non-ASCII characters, and a client library that does implement SASLprep.
That will hopefully be added later.
Authorization identities, as specified in the SCRAM-SHA-256 specification,
are ignored. SET SESSION AUTHORIZATION provides more or less the same
functionality, anyway.
If a user doesn't exist, perform a "mock" authentication, by constructing
an authentic-looking challenge on the fly. The challenge is derived from
a new system-wide random value, "mock authentication nonce", which is
created at initdb, and stored in the control file. We go through these
motions, in order to not give away the information on whether the user
exists, to unauthenticated users.
Bumps PG_CONTROL_VERSION, because of the new field in control file.
Patch by Michael Paquier and Heikki Linnakangas, reviewed at different
stages by Robert Haas, Stephen Frost, David Steele, Aleksander Alekseev,
and many others.
Discussion: https://www.postgresql.org/message-id/CAB7nPqRbR3GmFYdedCAhzukfKrgBLTLtMvENOmPrVWREsZkF8g%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CAB7nPqSMXU35g%3DW9X74HVeQp0uvgJxvYOuA4A-A3M%2B0wfEBv-w%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/55192AFE.6080106@iki.fi
This way both frontend and backends can use them. The functions are taken
from pgcrypto, which now fetches the source files it needs from
src/common/.
A new interface is designed for the SHA2 functions, which allow linking
to either OpenSSL or the in-core stuff taken from KAME as needed.
Michael Paquier, reviewed by Robert Haas.
Discussion: https://www.postgresql.org/message-id/CAB7nPqTGKuTM5jiZriHrNaQeVqp5e_iT3X4BFLWY_HyHxLvySQ%40mail.gmail.com
This adds a new routine, pg_strong_random() for generating random bytes,
for use in both frontend and backend. At the moment, it's only used in
the backend, but the upcoming SCRAM authentication patches need strong
random numbers in libpq as well.
pg_strong_random() is based on, and replaces, the existing implementation
in pgcrypto. It can acquire strong random numbers from a number of sources,
depending on what's available:
- OpenSSL RAND_bytes(), if built with OpenSSL
- On Windows, the native cryptographic functions are used
- /dev/urandom
Unlike the current pgcrypto function, the source is chosen by configure.
That makes it easier to test different implementations, and ensures that
we don't accidentally fall back to a less secure implementation, if the
primary source fails. All of those methods are quite reliable, it would be
pretty surprising for them to fail, so we'd rather find out by failing
hard.
If no strong random source is available, we fall back to using erand48(),
seeded from current timestamp, like PostmasterRandom() was. That isn't
cryptographically secure, but allows us to still work on platforms that
don't have any of the above stronger sources. Because it's not very secure,
the built-in implementation is only used if explicitly requested with
--disable-strong-random.
This replaces the more complicated Fortuna algorithm we used to have in
pgcrypto, which is unfortunate, but all modern platforms have /dev/urandom,
so it doesn't seem worth the maintenance effort to keep that. pgcrypto
functions that require strong random numbers will be disabled with
--disable-strong-random.
Original patch by Magnus Hagander, tons of further work by Michael Paquier
and me.
Discussion: https://www.postgresql.org/message-id/CAB7nPqRy3krN8quR9XujMVVHYtXJ0_60nqgVc6oUk8ygyVkZsA@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CAB7nPqRWkNYRRPJA7-cF+LfroYV10pvjdz6GNvxk-Eee9FypKA@mail.gmail.com
This reverts commit 9e083fd468. That was a
few bricks shy of a load:
* Query cancel stopped working
* Buildfarm member pademelon stopped working, because the box doesn't have
/dev/urandom nor /dev/random.
This clearly needs some more discussion, and a quite different patch, so
revert for now.
This adds a new routine, pg_strong_random() for generating random bytes,
for use in both frontend and backend. At the moment, it's only used in
the backend, but the upcoming SCRAM authentication patches need strong
random numbers in libpq as well.
pg_strong_random() is based on, and replaces, the existing implementation
in pgcrypto. It can acquire strong random numbers from a number of sources,
depending on what's available:
- OpenSSL RAND_bytes(), if built with OpenSSL
- On Windows, the native cryptographic functions are used
- /dev/urandom
- /dev/random
Original patch by Magnus Hagander, with further work by Michael Paquier
and me.
Discussion: <CAB7nPqRy3krN8quR9XujMVVHYtXJ0_60nqgVc6oUk8ygyVkZsA@mail.gmail.com>
This add a new pgp_armor_headers function to extract armor headers from an
ASCII-armored blob, and a new overloaded variant of the armor function, for
constructing an ASCII-armor with extra headers.
Marko Tiikkaja and me.
ws2_32 is the new version of the library that should be used, as
it contains the require functionality from wsock32 as well as some
more (which is why some binaries were already using ws2_32).
Michael Paquier, reviewed by MauMau
Prominent binaries already had this metadata. A handful of minor
binaries, such as pg_regress.exe, still lack it; efforts to eliminate
such exceptions are welcome.
Michael Paquier, reviewed by MauMau.
This function provides a way of generating version 4 (pseudorandom) UUIDs
based on pgcrypto's PRNG. The main reason for doing this is that the
OSSP UUID library depended on by contrib/uuid-ossp is becoming more and
more of a porting headache, so we need an alternative for people who can't
install that. A nice side benefit though is that this implementation is
noticeably faster than uuid-ossp's uuid_generate_v4() function.
Oskari Saarenmaa, reviewed by Emre Hasegeli
This isn't fully tested as yet, in particular I'm not sure that the
"foo--unpackaged--1.0.sql" scripts are OK. But it's time to get some
buildfarm cycles on it.
sepgsql is not converted to an extension, mainly because it seems to
require a very nonstandard installation process.
Dimitri Fontaine and Tom Lane
installations whose pg_config program does not appear first in the PATH.
Per gripe from Eddie Stanley and subsequent discussions with Fabien Coelho
and others.
Few cleanups and couple of new things:
- add SHA2 algorithm to older OpenSSL
- add BIGNUM math to have public-key cryptography work on non-OpenSSL
build.
- gen_random_bytes() function
The status of SHA2 algoritms and public-key encryption can now be
changed to 'always available.'
That makes pgcrypto functionally complete and unless there will be new
editions of AES, SHA2 or OpenPGP standards, there is no major changes
planned.
the pubkey functions a bit. The actual RSA-specific code
there is tiny, most of the patch consists of reorg of the
pubkey code, as lots of it was written as elgamal-only.
---------------------------------------------------------------------------
The SHLIB section was copy-pasted from somewhere and contains
several unnecessary libs. This cleans it up a bit.
-lcrypt
we don't use system crypt()
-lssl, -lssleay32
no SSL here
-lz in win32 section
already added on previous line
-ldes
The chance anybody has it is pretty low.
And the chance pgcrypto works with it is even lower.
Also trim the win32 section.
---------------------------------------------------------------------------
It is already disabled in Makefile, remove code too.
---------------------------------------------------------------------------
I was bit hasty making the random exponent 'k' a prime. Further researh
shows that Elgamal encryption has no specific needs in respect to k,
any random number is fine.
It is bit different for signing, there it needs to be 'relatively prime'
to p - 1, that means GCD(k, p-1) == 1, which is also a lot lighter than
full primality. As we don't do signing, this can be ignored.
This brings major speedup to Elgamal encryption.
---------------------------------------------------------------------------
o pgp_mpi_free: Accept NULLs
o pgp_mpi_cksum: result should be 16bit
o Remove function name from error messages - to be similar to other
SQL functions, and it does not match anyway the called function
o remove couple junk lines
---------------------------------------------------------------------------
o Support for RSA encryption
o Big reorg to better separate generic and algorithm-specific code.
o Regression tests for RSA.
---------------------------------------------------------------------------
o Tom stuck a CVS id into file. I doubt the usefulness of it,
but if it needs to be in the file then rather at the end.
Also tag it as comment for asciidoc.
o Mention bytea vs. text difference
o Couple clarifications
---------------------------------------------------------------------------
There is a choice whether to update it with pgp functions or
remove it. I decided to remove it, updating is pointless.
I've tried to keep the core of pgcrypto relatively independent
from main PostgreSQL, to make it easy to use externally if needed,
and that is good. Eg. that made development of PGP functions much
nicer.
But I have no plans to release it as generic library, so keeping such
doc
up-to-date is waste of time. If anyone is interested in using it in
other products, he can probably bother to read the source too.
Commented source is another thing - I'll try to make another pass
over code to see if there is anything non-obvious that would need
more comments.
---------------------------------------------------------------------------
Marko Kreen
of password-based encryption from RFC2440 (OpenPGP).
The goal of this code is to be more featureful encryption solution
than current encrypt(), which only functionality is running cipher
over data.
Compared to encrypt(), pgp_encrypt() does following:
* It uses the equvialent of random Inital Vector to get cipher
into random state before it processes user data
* Stores SHA-1 of the data into result so any modification
will be detected.
* Remembers if data was text or binary - thus it can decrypt
to/from text data. This was a major nuisance for encrypt().
* Stores info about used algorithms with result, so user needs
not remember them - more user friendly!
* Uses String2Key algorithms (similar to crypt()) with random salt
to generate full-length binary key to be used for encrypting.
* Uses standard format for data - you can feed it to GnuPG, if needed.
Optional features (off by default):
* Can use separate session key - user data will be encrypted
with totally random key, which will be encrypted with S2K
generated key and attached to result.
* Data compression with zlib.
* Can convert between CRLF<->LF line-endings - to get fully
RFC2440-compliant behaviour. This is off by default as
pgcrypto does not know the line-endings of user data.
Interface is simple:
pgp_encrypt(data text, key text) returns bytea
pgp_decrypt(data text, key text) returns text
pgp_encrypt_bytea(data bytea, key text) returns bytea
pgp_decrypt_bytea(data bytea, key text) returns bytea
To change parameters (cipher, compression, mdc):
pgp_encrypt(data text, key text, parms text) returns bytea
pgp_decrypt(data text, key text, parms text) returns text
pgp_encrypt_bytea(data bytea, key text, parms text) returns bytea
pgp_decrypt_bytea(data bytea, key text, parms text) returns bytea
Parameter names I lifted from gpg:
pgp_encrypt('message', 'key', 'compress-algo=1,cipher-algo=aes256')
For text data, pgp_encrypt simply encrypts the PostgreSQL internal data.
This maps to RFC2440 data type 't' - 'extenally specified encoding'.
But this may cause problems if data is dumped and reloaded into database
which as different internal encoding. My next goal is to implement data
type 'u' - which means data is in UTF-8 encoding by converting internal
encoding to UTF-8 and back. And there wont be any compatibility
problems with current code, I think its ok to submit this without UTF-8
encoding by converting internal encoding to UTF-8 and back. And there
wont be any compatibility problems with current code, I think its ok to
submit this without UTF-8 support.
Here is v4 of PGP encrypt. This depends on previously sent
Fortuna-patch, as it uses the px_add_entropy function.
- New function: pgp_key_id() for finding key id's.
- Add SHA1 of user data and key into RNG pools. We need to get
randomness from somewhere, and it is in user best interests
to contribute.
- Regenerate pgp-armor test for SQL_ASCII database.
- Cleanup the key handling so that the pubkey support is less
hackish.
Marko Kreen
- Move openssl random provider to openssl.c and builtin provider
to internal.c
- Make px_random_bytes use Fortuna, instead of giving error.
- Retarget random.c to aquiring system randomness, for initial seeding
of Fortuna. There is ATM 2 functions for Windows,
reader from /dev/urandom and the regular time()/getpid() silliness.
Marko Kreen
to make. We ship the table file in the tarball and so this dependency
just opens file timestamp skew problems without doing anything useful.
(Not that it should hurt, either ... except for cross-compile builds.)
* test error handling
* add tests for des, 3des, cast5
* add some tests to blowfish, rijndael
* Makefile: ability to specify different tests for different crypto
libraries, so we can skip des, 3des and cast5 for builtin.
Marko Kreen
libmcrypt seems to dead, maintainer address bounces,
and cast-128 fails on 2 of the 3 test vectors from RFC2144.
So I see no reason to keep around stuff I don't trust
anymore.
Support for several crypto libraries is probably only
confusing to users, although it was good for initial
developing - it helped to find hidden assumptions and
forced me to create regression tests for all functionality.
Marko Kreen
>
> The patch adds missing the "libpgport.a" file to the installation under
> "install-all-headers". It is needed by some contribs. I install the
> library in "pkglibdir", but I was wondering whether it should be "libdir"?
> I was wondering also whether it would make sense to have a "libpgport.so"?
>
> It fixes various macros which are used by contrib makefiles, especially
> libpq_*dir and LDFLAGS when used under PGXS. It seems to me that they are
> needed to
>
> It adds the ability to test and use PGXS with contribs, with "make
> USE_PGXS=1". Without the macro, this is exactly as before, there should be
> no difference, esp. wrt the vpath feature that seemed broken by previous
> submission. So it should not harm anybody, and it is useful at least to me.
>
> It fixes some inconsistencies in various contrib makefiles
> (useless override, ":=" instead of "=").
Fabien COELHO
Converted pgcrypto one too.
* Changed default randomness source to libc random()
That way pgcrypto does not have any external dependencies
and should work everywhere.
* Re-enabled pgcrypto build in contrib/makefile
* contrib/README update - there is more stuff than
only 'hash functions'
* Noted the libc random fact in README.pgcrypto
Marko Kreen
salt generation code. He also urged using better random source
and making possible to choose using bcrypt and xdes rounds more
easily. So, here's patch:
* For all salt generation, use Solar Designer's own code. This
is mostly due fact that his code is more fit for get_random_bytes()
style interface.
* New function: gen_salt(type, rounds). This lets specify iteration
count for algorithm.
* random.c: px_get_random_bytes() function.
Supported randomness soure: /dev/urandom, OpenSSL PRNG, libc random()
Default: /dev/urandom.
* Draft description of C API for pgcrypto functions.
New files: API, crypt-gensalt.c, random.c
Marko Kreen
under libdir, for a cleaner separation in the installation layout
and compatibility with binary packaging standards. Point backend's
default search location there. The contrib modules are also
installed in the said location, giving them the benefit of the
default search path as well. No changes in user interface
nevertheless.
* remove support for encode() as it is in main tree now
* remove krb5.c
* new 'PX library' architecture
* remove BSD license from my code to let the general
PostgreSQL one to apply
* md5, sha1: ANSIfy, use const where appropriate
* various other formatting and clarity changes
* hmac()
* UN*X-like crypt() - system or internal crypt
* Internal crypt: DES, Extended DES, MD5, Blowfish
crypt-des.c, crypt-md5.c from FreeBSD
crypt-blowfish.c from Solar Designer
* gen_salt() for crypt() - Blowfish, MD5, DES, Extended DES
* encrypt(), decrypt(), encrypt_iv(), decrypt_iv()
* Cipher support in mhash.c, openssl.c
* internal: Blowfish, Rijndael-128 ciphers
* blf.[ch], rijndael.[ch] from OpenBSD
* there will be generated file rijndael-tbl.inc.
Marko Kreen
Cygwin with the possible exception of mSQL-interface. Since I don't
have mSQL installed, I skipped this tool.
Except for dealing with a missing getopt.h (oid2name) and HUGE (seg),
the bulk of the patch uses the standard PostgreSQL approach to deal with
Windows DLL issues.
I tested the build aspect of this patch under Cygwin and Linux without
any ill affects. Note that I did not actually attempt to test the code
for functionality.
The procedure to apply the patch is as follows:
$ # save the attachment as /tmp/contrib.patch
$ # change directory to the top of the PostgreSQL source tree
$ patch -p0 </tmp/contrib.patch
Jason
* reverse the change #include <> -> "" in krb.c.
It _must not_ include files in "."
* Makefile update. Inconsistent var usage and SHLIB was
not set.
Now it should work with all external libs.
arko Kreen
timing, I know :)) At the moment the digest() function returns
hexadecimal coded hash, but I want it to return pure binary. I
have also included functions encode() and decode() which support
'base64' and 'hex' encodings, so if anyone needs digest() in hex
he can do encode(digest(...), 'hex').
Main reason for it is "to do one thing and do it well" :)
Another reason is if someone needs really lot of digesting, in
the end he wants to store the binary not the hexadecimal result.
It is really silly to convert it to hex then back to binary
again. As I said if someone needs hex he can get it.
Well, and the real reason that I am doing encrypt()/decrypt()
functions and _they_ return binary. For testing I like to see
it in hex occasionally, but it is really wrong to let them
return hex. Only now it caught my eye that hex-coding in
digest() is wrong. When doing digest() I thought about 'common
case' but hacking with psql is probably _not_ the common case :)
Marko Kreen
Michael Paquier