Add columns with volatile DEFAULTs without an ACCESS_EXCLUSIVE table lock

[andrew-farries]

Ensure that adding a new column (with an add_column operation) does not take an ACCESS_EXCLUSIVE lock on the table even when the column has a volatile DEFAULT expression.

Adding a column with a DEFAULT expression in Postgres can have one of two different locking behaviours depending on the expression used as the DEFAULT:

• For non-volatile DEFAULT expressions, the column addition can use the fast-path optimization to add the column default; this is a catalog-only operation and so holds a lock only for a very short time.
• For volatile DEFAULT expressions, the fast-path optimization cannot be used; the change requires a full table rewrite - this means holding an ACCESS_EXCLUSIVE lock on the table until the rewrite is complete.

This PR makes add_column operations aware of this distinction and able to add columns with volatile defaults without taking an ACCESS_EXCLUSIVE lock on the table.

When adding a column with a DEFAULT:

• create a temporary schema-only copy of the table.
• add the column to this temporary table
• read the pg_attribute.atthasmissing system catalog field to determine whether the fast-path optimization was used to add the column DEFAULT.
• drop the temporary table

Now the operation knows whether the column can be added using the fast-path DEFAULT optimization. In the case where the column can use the optimization, the column can be added with a DEFAULT directly. In the case where the fast-path optimization can't be used:

• Add the column without a DEFAULT value
• Create an up trigger to backfill the column using the column DEFAULT expression
• Add the DEFAULT to the column on migration completion

This avoids the ACCESS_EXCLUSIVE lock; effectively replacing the table rewrite that would have caused it with a backfill operation instead.

If the column to be added has a volatile DEFAULT and a NOT NULL constraint, the column must be added with a NOT NULL ... NOT VALID constraint to allow the column to be added without a DEFAULT; the constraint is validated and upgraded on migration completion.

For a non-volatile DEFAULTs, the operation must set up to the same expression as is set in default. The operation will fail and roll back if they differ.

References

Volatile vs non-volatile functions:

• https://www.postgresql.org/docs/current/xfunc-volatility.html

The fast-path optimization for non-volatile column DEFAULT expressions is explained here:

• https://brandur.org/postgres-default
• https://www.enterprisedb.com/blog/adding-new-table-columns-default-values-postgresql-11

Fixes #643