users/andresfreund/postgres.git
5 years agofixup! heapam: WIP: cacheline prefetching for hot pruning. aio-before-rebase-2020-06-24
Andres Freund [Wed, 24 Jun 2020 23:26:26 +0000 (16:26 -0700)]
fixup! heapam: WIP: cacheline prefetching for hot pruning.

5 years agotmp
Andres Freund [Thu, 28 May 2020 21:57:20 +0000 (14:57 -0700)]
tmp

5 years agoWIP: Change instr_time to just store nanoseconds, that's cheaper.
Andres Freund [Thu, 28 May 2020 21:52:48 +0000 (14:52 -0700)]
WIP: Change instr_time to just store nanoseconds, that's cheaper.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoheapam: WIP: cacheline prefetching for hot pruning.
Andres Freund [Thu, 28 May 2020 21:51:58 +0000 (14:51 -0700)]
heapam: WIP: cacheline prefetching for hot pruning.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoheapam: Don't re-inquire block number for each tuple during pruning.
Andres Freund [Thu, 28 May 2020 21:50:50 +0000 (14:50 -0700)]
heapam: Don't re-inquire block number for each tuple during pruning.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoexecutor: Reduce overhead of instrumentation a bit by allowing more inlining.
Andres Freund [Thu, 28 May 2020 21:49:45 +0000 (14:49 -0700)]
executor: Reduce overhead of instrumentation a bit by allowing more inlining.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: Back PrefetchSharedBuffer with aio.
Andres Freund [Thu, 28 May 2020 21:48:01 +0000 (14:48 -0700)]
aio: Back PrefetchSharedBuffer with aio.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: prewarm: remove error message that was just a test.
Andres Freund [Thu, 28 May 2020 21:46:52 +0000 (14:46 -0700)]
aio: prewarm: remove error message that was just a test.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: bitmap heap scan: Minimal and hacky improvement for prefetching.
Andres Freund [Thu, 28 May 2020 21:46:05 +0000 (14:46 -0700)]
aio: bitmap heap scan: Minimal and hacky improvement for prefetching.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoFix compliation with tracepoints
Dmitrii Dolgov [Thu, 28 May 2020 17:36:27 +0000 (19:36 +0200)]
Fix compliation with tracepoints

I guess they were skipped, and issues were not causing compilation
errors.

5 years agoaio: Separate out variables controlling different counts of IOs.
Andres Freund [Tue, 26 May 2020 02:10:44 +0000 (19:10 -0700)]
aio: Separate out variables controlling different counts of IOs.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: heapam: Use the pg_streaming_read logic for both parallel and normal scans.
Andres Freund [Tue, 26 May 2020 02:07:57 +0000 (19:07 -0700)]
aio: heapam: Use the pg_streaming_read logic for both parallel and normal scans.

The previous stuff for parallel scans was a dirty hack.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoUse larger step sizes for Parallel Seq Scan.
Thomas Munro [Mon, 18 May 2020 21:16:07 +0000 (09:16 +1200)]
Use larger step sizes for Parallel Seq Scan.

Instead of handing out a single block at a time, which
confuses the read ahead heuristics of many systems, use
an increasing step size.

XXX Proof of concept grade code only, needs better ramp
down algorithm and contains a magic number 16

5 years agoaio: Significantly improve prefetcher using the new callbacks.
Andres Freund [Mon, 25 May 2020 23:35:36 +0000 (16:35 -0700)]
aio: Significantly improve prefetcher using the new callbacks.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: Add AIO completion callbacks that execute in the issuing backend.
Andres Freund [Mon, 25 May 2020 23:33:41 +0000 (16:33 -0700)]
aio: Add AIO completion callbacks that execute in the issuing backend.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: Improve naming consistency.
Andres Freund [Mon, 25 May 2020 10:30:49 +0000 (03:30 -0700)]
aio: Improve naming consistency.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: Remove iovecs from PgAioInProgress.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: Remove iovecs from PgAioInProgress.

This is pretty nice, because due to combine support this was the
majority of struct PgAioInProgress size.

As it turns out, io_uring always copies the IOV during submit, making
it unnecessary to have a long live iovecs.

There's more to do to shrink PgAioInProgress to something sensible,
but not now...

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: Add comment to explain better design for pgaio_wait_for_io().
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: Add comment to explain better design for pgaio_wait_for_io().

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: Significantly improve shared memory state, aio state handling.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: Significantly improve shared memory state, aio state handling.

In particular this prepared for per-backend unused IO lists, as well
as local callbacks.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agofixup! aio: Hacky improvement to pg_streaming_write
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
fixup! aio: Hacky improvement to pg_streaming_write

5 years agoaio: Merging support for PGAIO_WRITE_WAL.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: Merging support for PGAIO_WRITE_WAL.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agofixup! aio: Make use of O_DIRECT configurable.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
fixup! aio: Make use of O_DIRECT configurable.

5 years agoaio: xlog: HEAVY WIP: Start of a better flushing infrastructure.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: xlog: HEAVY WIP: Start of a better flushing infrastructure.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agowalwriter: Separate sleep from delay.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
walwriter: Separate sleep from delay.

We should probably instead wake walwriter when wal_writer_flush_after
has passed, even if wal_writer_delay wouldn't yet trigger.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: Hacky improvement to pg_streaming_write
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: Hacky improvement to pg_streaming_write

Should instead use local completion callbacks (which do not exist
yet).

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: xlog: Small XLogFileInit() improvements.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: xlog: Small XLogFileInit() improvements.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agofixup! aio: Make use of O_DIRECT configurable.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
fixup! aio: Make use of O_DIRECT configurable.

5 years agoaio: vacuum: Make first scan use prefetching.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: vacuum: Make first scan use prefetching.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: Make use of O_DIRECT configurable.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: Make use of O_DIRECT configurable.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: xlog: aio xlog file init.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: xlog: aio xlog file init.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: xlog: small XLogWrite fixes for the aio part.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: xlog: small XLogWrite fixes for the aio part.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agolwlock, xlog: Report caller wait event for LWLockWaitForVar.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
lwlock, xlog: Report caller wait event for LWLockWaitForVar.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio: prewarm: bugfix for empty relations
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: prewarm: bugfix for empty relations

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agofixup! xlog: Avoid repeated scans of all xlog insert locks when replacing contents.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
fixup! xlog: Avoid repeated scans of all xlog insert locks when replacing contents.

5 years agoaio: xlog: small fixes
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio: xlog: small fixes

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoxlog: Avoid repeated scans of all xlog insert locks when replacing contents.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
xlog: Avoid repeated scans of all xlog insert locks when replacing contents.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agov2 of bulk extension
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
v2 of bulk extension

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoaio improvements, particularly error handling / retries.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
aio improvements, particularly error handling / retries.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agominor logging & formatting stuff.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
minor logging & formatting stuff.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoinitial (crappy) prototype of bulk extension.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
initial (crappy) prototype of bulk extension.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoasync writeback for bgwriter.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
async writeback for bgwriter.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agowrite combining, better io error handling.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
write combining, better io error handling.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agotmp: work around instability in tuplesort test.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
tmp: work around instability in tuplesort test.

Not really clear why this changes right now.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoilist.h debugging improvements.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
ilist.h debugging improvements.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agocombining and lots of other hacking.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
combining and lots of other hacking.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoHack hack hack - including readahead for seqscans.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Hack hack hack - including readahead for seqscans.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agofixup! basic streaming writes
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
fixup! basic streaming writes

5 years agoFix races around recycling of IOs.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Fix races around recycling of IOs.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoAllow PgAioInProgress to be reused.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Allow PgAioInProgress to be reused.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoMake streaming write helper more generic.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Make streaming write helper more generic.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agominor cleanups.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
minor cleanups.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoRefactor PgAioInProgress acquisition to callers.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Refactor PgAioInProgress acquisition to callers.

This allows to associate more information with aios before they've
been submitted. Will make retries easier. And more.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoBounce buffer management for checksum'ed pages.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Bounce buffer management for checksum'ed pages.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agobasic streaming writes
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
basic streaming writes

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoFix: Don't add postmaster dead check to WES when not under postmaster.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Fix: Don't add postmaster dead check to WES when not under post

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agominimal aio write
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
minimal aio write

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoWIP: Fix (non-aio) bug leading to checkpointer not shutting down etc.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
WIP: Fix (non-aio) bug leading to checkpointer not shutting down etc.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/20200517010005.jzaf2245w4rrgs2o@alap3.anarazel.de
Backpatch:

5 years agoHacky around condition variable initialization for single process setup.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Hacky around condition variable initialization for single process setup.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoMinor logging improvements.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Minor logging improvements.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoFix stupid race condition.
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
Fix stupid race condition.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agohack some more
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
hack some more

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agohackety hack
Andres Freund [Mon, 25 May 2020 10:02:56 +0000 (03:02 -0700)]
hackety hack

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agotmp
Andres Freund [Mon, 25 May 2020 10:02:55 +0000 (03:02 -0700)]
tmp

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agofixup! things are kinda working
Andres Freund [Mon, 25 May 2020 10:02:55 +0000 (03:02 -0700)]
fixup! things are kinda working

5 years agoAdd allocator support for larger allocation alignment & use for IO.
Andres Freund [Mon, 25 May 2020 10:02:55 +0000 (03:02 -0700)]
Add allocator support for larger allocation alignment & use for IO.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agotmp
Andres Freund [Mon, 25 May 2020 10:02:55 +0000 (03:02 -0700)]
tmp

5 years agopgaio: reorder
Andres Freund [Mon, 25 May 2020 10:02:55 +0000 (03:02 -0700)]
pgaio: reorder

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoWIP: Increase default ringbuffer size.
Andres Freund [Mon, 25 May 2020 10:02:55 +0000 (03:02 -0700)]
WIP: Increase default ringbuffer size.

Otherwise they're too small to usefully prefetch into.

5 years agothings are kinda working
Andres Freund [Wed, 24 Jun 2020 23:20:49 +0000 (16:20 -0700)]
things are kinda working

5 years agoAdd --with-liburing configure option.
Andres Freund [Mon, 25 May 2020 10:02:55 +0000 (03:02 -0700)]
Add --with-liburing configure option.

Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:

5 years agoReplace buffer I/O locks with condition variables.
Andres Freund [Mon, 25 May 2020 10:02:55 +0000 (03:02 -0700)]
Replace buffer I/O locks with condition variables.

Author: Robert Haas
Discussion: https://postgr.es/m/CA+Tgmoaj2aPti0yho7FeEf2qt-JgQPRWb0gci_o1Hfr=C56Xng@mail.gmail.com

5 years agoDoc fixup for hashagg_avoid_disk_plan GUC.
Jeff Davis [Mon, 22 Jun 2020 19:14:55 +0000 (12:14 -0700)]
Doc fixup for hashagg_avoid_disk_plan GUC.

Reported-by: Justin Pryzby
Discussion: https://postgr.es/m/20200620220402.GZ17995@telsasoft.com
Backport-through: 13

5 years agoUndo double-quoting of index names in non-text EXPLAIN output formats.
Tom Lane [Mon, 22 Jun 2020 15:46:41 +0000 (11:46 -0400)]
Undo double-quoting of index names in non-text EXPLAIN output formats.

explain_get_index_name() applied quote_identifier() to the index name.
This is fine for text output, but the non-text output formats all have
their own quoting conventions and would much rather start from the
actual index name.  For example in JSON you'd get something like

       "Index Name": "\"My Index\"",

which is surely not desirable, especially when the same does not
happen for table names.  Hence, move the responsibility for applying
quoting out to the callers, where it can go into already-existing
special code paths for text format.

This changes the API spec for users of explain_get_index_name_hook:
before, they were supposed to apply quote_identifier() if necessary,
now they should not.  Research suggests that the only publicly
available user of the hook is hypopg, and it actually forgot to
apply quoting anyway, so it's fine.  (In any case, there's no
behavioral change for the output of a hook as seen in non-text
EXPLAIN formats, so this won't break any case that programs should
be relying on.)

Digging in the commit logs, it appears that quoting was included in
explain_get_index_name's duties when commit 604ffd280 invented it;
and that was fine at the time because we only had text output format.
This should have been rethought when non-text formats were invented,
but it wasn't.

This is a fairly clear bug for users of non-text EXPLAIN formats,
so back-patch to all supported branches.

Per bug #16502 from Maciek Sakrejda.  Patch by me (based on
investigation by Euler Taveira); thanks to Julien Rouhaud for review.

Discussion: https://postgr.es/m/16502-57bd1c9f913ed1d1@postgresql.org

5 years agoFix inconsistent markups in catalogs.sgml
Michael Paquier [Mon, 22 Jun 2020 04:40:04 +0000 (13:40 +0900)]
Fix inconsistent markups in catalogs.sgml

Some fields related to pg_opclass and pg_opfamily were using incorrect
markups, listing them as structname instead of structfield.

Author: Fabien Coelho
Discussion: https://postgr.es/m/alpine.DEB.2.22.394.2006210903560.859381@pseudo

5 years agoAdd --no-index-cleanup and --no-truncate to vacuumdb.
Michael Paquier [Mon, 22 Jun 2020 04:23:38 +0000 (13:23 +0900)]
Add --no-index-cleanup and --no-truncate to vacuumdb.

Both INDEX_CLEANUP and TRUNCATE have been available since v12, and are
enabled by default except if respectively vacuum_index_cleanup and
vacuum_truncate are disabled for a given relation.  This change adds
support for disabling these options from vacuumdb.

Author: Nathan Bossart
Reviewed-by: Michael Paquier, Masahiko Sawada
Discussion: https://postgr.es/m/6F7F17EF-B1F2-4681-8D03-BA96365717C0@amazon.com

5 years agoLanguage fixes for docs related to opclass options
Alexander Korotkov [Sun, 21 Jun 2020 01:48:03 +0000 (04:48 +0300)]
Language fixes for docs related to opclass options

Discussion: https://postgr.es/m/20200620232145.GB17995%40telsasoft.com
Author: Justin Pryzby
Backpatch-through: 13

5 years agoDoc: Tweak description of B-Tree duplicate tuples.
Peter Geoghegan [Sun, 21 Jun 2020 00:34:07 +0000 (17:34 -0700)]
Doc: Tweak description of B-Tree duplicate tuples.

Defining duplicates as "close by" to each other was unclear.  Simplify
the definition.

Backpatch: 13-, where deduplication was introduced (by commit 0d861bbb)

5 years agoMinor corrections to docs related to opclass options
Alexander Korotkov [Sat, 20 Jun 2020 21:35:42 +0000 (00:35 +0300)]
Minor corrections to docs related to opclass options

Reported-by: Peter Geoghegan
Discussion: https://postgr.es/m/CAH2-WzmwhYbxuoL0WjTLaiCxW3gj6qadeNpBhWAo_KZsE5-FGw%40mail.gmail.com

5 years agoFix masking of SP-GiST pages during xlog consistency check
Alexander Korotkov [Sat, 20 Jun 2020 14:34:51 +0000 (17:34 +0300)]
Fix masking of SP-GiST pages during xlog consistency check

spg_mask() didn't take into account that pd_lower equal to SizeOfPageHeaderData
is still valid value.  This commit fixes that.  Backpatch to 11, where
spg_mask() pg_lower check was introduced.

Reported-by: Michael Paquier
Discussion: https://postgr.es/m/20200615131405.GM52676%40paquier.xyz
Backpatch-through: 11

5 years agoAdd documentation for opclass options
Alexander Korotkov [Sat, 20 Jun 2020 10:34:54 +0000 (13:34 +0300)]
Add documentation for opclass options

911e7020770 added opclass options and adjusted documentation for each
particular affected opclass.  However, documentation for extendability was
not adjusted.  This commit adjusts documentation for interfaces of index AMs
and opclasses.

Discussion: https://postgr.es/m/CAH2-WzmQnW6%2Bz5F9AW%2BSz%2BzEcEvXofTwh_A9J3%3D_WA-FBP0wYg%40mail.gmail.com
Author: Alexander Korotkov
Reported-by: Peter Geoghegan
Reviewed-by: Peter Geoghegan
5 years agoRemove dead forceSync parameter of XactLogCommitRecord().
Noah Misch [Sat, 20 Jun 2020 08:25:40 +0000 (01:25 -0700)]
Remove dead forceSync parameter of XactLogCommitRecord().

The function has been reading global variable forceSyncCommit, mirroring
the intent of the caller that passed forceSync=forceSyncCommit.  The
other caller, RecordTransactionCommitPrepared(), passed false.  Since
COMMIT PREPARED can't share a transaction with any command, it certainly
doesn't share a transaction with a command that sets forceSyncCommit.

Reviewed by Michael Paquier.

Discussion: https://postgr.es/m/20200617032615.GC2916904@rfd.leadboat.com

5 years agoRemoval unused function parameter in CopyReadBinaryAttribute.
Amit Kapila [Sat, 20 Jun 2020 03:48:57 +0000 (09:18 +0530)]
Removal unused function parameter in CopyReadBinaryAttribute.

The function parameter column_no is not used in CopyReadBinaryAttribute,
this can be removed.

Commit 0e319c7ad7 removed the usage of column_no parameter in function
CopyReadBinaryAttribute but forgot to remove the parameter.

Reported-by: Vignesh C
Author: Vignesh C
Discussion: https://postgr.es/m/CALDaNm1TYSNTfqx_jfz9_mwEZ2Er=dZnu++duXpC1uQo1cG=WA@mail.gmail.com

5 years agoEnsure write failure reports no-disk-space
Alvaro Herrera [Fri, 19 Jun 2020 20:46:07 +0000 (16:46 -0400)]
Ensure write failure reports no-disk-space

A few places calling fwrite and gzwrite were not setting errno to ENOSPC
when reporting errors, as is customary; this led to some failures being
reported as
"could not write file: Success"
which makes us look silly.  Make a few of these places in pg_dump and
pg_basebackup use our customary pattern.

Backpatch-to: 9.5
Author: Justin Pryzby <pryzby@telsasoft.com>
Author: Tom Lane <tgl@sss.pgh.pa.us>
Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Discussion: https://postgr.es/m/20200611153753.GU14879@telsasoft.com

5 years agoFuture-proof regression tests against possibly-missing posixrules file.
Tom Lane [Fri, 19 Jun 2020 17:55:21 +0000 (13:55 -0400)]
Future-proof regression tests against possibly-missing posixrules file.

The IANA time zone folk have deprecated use of a "posixrules" file in
the tz database.  While for now it's our choice whether to keep
supplying one in our own builds, installations built with
--with-system-tzdata will soon be needing to cope with that file not
being present, at least on some platforms.

This causes a problem for the horology test, which expected the
nonstandard POSIX zone spec "CST7CDT" to apply pre-2007 US daylight
savings rules.  That does happen if the posixrules file supplies such
information, but otherwise the test produces undesired results.
To fix, add an explicit transition date rule that matches 2005 practice.
(We could alternatively have switched the test to use some real time
zone, but it seems useful to have coverage of this type of zone spec.)

While at it, update a documentation example that also relied on
"CST7CDT"; use a real-world zone name instead.  Also, document why
the zone names EST5EDT, CST6CDT, MST7MDT, PST8PDT aren't subject to
similar failures when "posixrules" is missing.

Back-patch to all supported branches, since the hazard is the same
for all.

Discussion: https://postgr.es/m/1665379.1592581287@sss.pgh.pa.us

5 years agoAdjust some glossary terms
Alvaro Herrera [Fri, 19 Jun 2020 16:55:43 +0000 (12:55 -0400)]
Adjust some glossary terms

Mostly in response to Jürgen Purtz critique of previous definitions,
though I added many other changes.

Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Jürgen Purtz <juergen@purtz.de>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Erik Rijkers <er@xs4all.nl>
Discussion: https://postgr.es/m/c1e06008-2132-30f4-9b38-877e8683d418@purtz.de

5 years agoFix deduplication "single value" strategy bug.
Peter Geoghegan [Fri, 19 Jun 2020 15:57:24 +0000 (08:57 -0700)]
Fix deduplication "single value" strategy bug.

It was possible for deduplication's single value strategy to mistakenly
believe that a very small duplicate tuple counts as one of the six large
tuples that it aims to leave behind after the page finally splits.  This
could cause slightly suboptimal space utilization with very low
cardinality indexes, though only under fairly narrow conditions.

To fix, be particular about what kind of tuple counts as a
maxpostingsize-capped tuple.  This avoids confusion in the event of a
small tuple that gets "wedged" between two large tuples, where all
tuples on the page are duplicates of the same value.

Discussion: https://postgr.es/m/CAH2-Wz=Y+sgSFc-O3LpiZX-POx2bC+okec2KafERHuzdVa7-rQ@mail.gmail.com
Backpatch: 13-, where deduplication was introduced (by commit 0d861bbb)

5 years agoFix issues in invalidation of obsolete replication slots.
Fujii Masao [Fri, 19 Jun 2020 08:15:52 +0000 (17:15 +0900)]
Fix issues in invalidation of obsolete replication slots.

This commit fixes the following issues.

1. There is the case where the slot is dropped while trying to invalidate it.
    InvalidateObsoleteReplicationSlots() did not handle this case, and
    which could cause checkpoint to fail.

2. InvalidateObsoleteReplicationSlots() could emit the same log message
    multiple times unnecessary. It should be logged only once.

3. When marking the slot as used, we always searched the target slot from
    all the replication slots even if we already found it. This could cause
    useless waste of cycles.

Back-patch to v13 where these issues were added as a part of
max_slot_wal_keep_size code.

Author: Fujii Masao
Reviewed-by: Kyotaro Horiguchi, Alvaro Herrera
Discussion: https://postgr.es/m/66c05b67-3396-042c-1b41-bfa6c3ddcf82@oss.nttdata.com

5 years agoFix EXPLAIN ANALYZE for parallel HashAgg plans
David Rowley [Fri, 19 Jun 2020 05:24:27 +0000 (17:24 +1200)]
Fix EXPLAIN ANALYZE for parallel HashAgg plans

Since 1f39bce02, HashAgg nodes have had the ability to spill to disk when
memory consumption exceeds work_mem. That commit added new properties to
EXPLAIN ANALYZE to show the maximum memory usage and disk usage, however,
it didn't quite go as far as showing that information for parallel
workers.  Since workers may have experienced something very different from
the main process, we should show this information per worker, as is done
in Sort.

Reviewed-by: Justin Pryzby
Reviewed-by: Jeff Davis
Discussion: https://postgr.es/m/CAApHDvpEKbfZa18mM1TD7qV6PG+w97pwCWq5tVD0dX7e11gRJw@mail.gmail.com
Backpatch-through: 13, where the hashagg spilling code was added.

5 years agoClean up includes of s_lock.h.
Andres Freund [Fri, 19 Jun 2020 02:40:09 +0000 (19:40 -0700)]
Clean up includes of s_lock.h.

Users of spinlocks should use spin.h, not s_lock.h. And lwlock.h
hasn't utilized spinlocks for quite a while.

Discussion: https://postgr.es/m/20200618183041.upyrd25eosecyf3x@alap3.anarazel.de

5 years agoFix deadlock danger when atomic ops are done under spinlock.
Andres Freund [Mon, 8 Jun 2020 23:50:37 +0000 (16:50 -0700)]
Fix deadlock danger when atomic ops are done under spinlock.

This was a danger only for --disable-spinlocks in combination with
atomic operations unsupported by the current platform.

While atomics.c was careful to signal that a separate semaphore ought
to be used when spinlock emulation is active, spin.c didn't actually
implement that mechanism. That's my (Andres') fault, it seems to have
gotten lost during the development of the atomic operations support.

Fix that issue and add test for nesting atomic operations inside a
spinlock.

Author: Andres Freund
Discussion: https://postgr.es/m/20200605023302.g6v3ydozy5txifji@alap3.anarazel.de
Backpatch: 9.5-

5 years agoAdd basic spinlock tests to regression tests.
Andres Freund [Mon, 8 Jun 2020 23:36:51 +0000 (16:36 -0700)]
Add basic spinlock tests to regression tests.

As s_lock_test, the already existing test for spinlocks, isn't run in
an automated fashion (and doesn't test a normal backend environment),
adding tests that are run as part of a normal regression run is a good
idea. Particularly in light of several recent and upcoming spinlock
related fixes.

Currently the new tests are run as part of the pre-existing
test_atomic_ops() test. That perhaps can be quibbled about, but for
now seems ok.

The only operations that s_lock_test tests but the new tests don't are
the detection of a stuck spinlock and S_LOCK_FREE (which is otherwise
unused, not implemented on all platforms, and will be removed).

This currently contains a test for more than INT_MAX spinlocks (only
run with --disable-spinlocks), to ensure the recent commit fixing a
bug with more than INT_MAX spinlock initializations is correct. That
test is somewhat slow, so we might want to disable it after a few
days.

It might be worth retiring s_lock_test after this. The added coverage
of a stuck spinlock probably isn't worth the added complexity?

Author: Andres Freund
Discussion: https://postgr.es/m/20200606023103.avzrctgv7476xj7i@alap3.anarazel.de

5 years agoDoc: document POSIX-style time zone specifications in full.
Tom Lane [Thu, 18 Jun 2020 20:27:18 +0000 (16:27 -0400)]
Doc: document POSIX-style time zone specifications in full.

We'd glossed over most of this complexity for years, but it's hard
to avoid writing it all down now, so that we can explain what happens
when there's no "posixrules" file in the IANA time zone database.
That was at best a tiny minority situation till now, but it's likely
to become quite common in the future, so we'd better explain it.

Nonetheless, we don't really encourage people to use POSIX zone specs;
picking a named zone is almost always what you really want, unless
perhaps you're stuck with an out-of-date zone database.  Therefore,
let's shove all this detail into an appendix.

Patch by me; thanks to Robert Haas for help with some awkward wording.

Discussion: https://postgr.es/m/1390.1562258309@sss.pgh.pa.us

5 years agoFix oldest xmin and LSN computation across repslots after advancing
Michael Paquier [Thu, 18 Jun 2020 07:34:59 +0000 (16:34 +0900)]
Fix oldest xmin and LSN computation across repslots after advancing

Advancing a replication slot did not recompute the oldest xmin and LSN
values across replication slots, preventing resource removal like
segments not recycled at checkpoint time.  The original commit that
introduced the slot advancing in 9c7d06d never did the update of those
oldest values, and b0afdca removed this code.

This commit adds a TAP test to check segment recycling with advancing
for physical slots, enforcing an extra segment switch before advancing
to check if the segment gets correctly recycled after a checkpoint.

Reported-by: Andres Freund
Reviewed-by: Alexey Kondratov, Kyptaro Horiguchi
Discussion: https://postgr.es/m/20200609171904.kpltxxvjzislidks@alap3.anarazel.de
Backpatch-through: 11

5 years agoDisallow factorial of negative numbers
Peter Eisentraut [Thu, 18 Jun 2020 06:41:31 +0000 (08:41 +0200)]
Disallow factorial of negative numbers

The previous implementation returned 1 for all negative numbers, which
is not sensible under any definition.

Discussion: https://www.postgresql.org/message-id/flat/6ce1df0e-86a3-e544-743a-f357ff663f68%402ndquadrant.com

5 years agoExpand tests for factorial
Peter Eisentraut [Thu, 18 Jun 2020 06:41:31 +0000 (08:41 +0200)]
Expand tests for factorial

Move from int4 to numeric test.  (They were originally int4 functions,
but were reimplemented for numeric in
04a4821adef38155b7920ba9eb83c4c3c29156f8.)  Add some tests for edge
cases.

Discussion: https://www.postgresql.org/message-id/flat/6ce1df0e-86a3-e544-743a-f357ff663f68%402ndquadrant.com

5 years agoRemove reset of testtablespace from pg_regress on Windows
Michael Paquier [Thu, 18 Jun 2020 01:40:10 +0000 (10:40 +0900)]
Remove reset of testtablespace from pg_regress on Windows

testtablespace is an extra path used as tablespace location in the main
regression test suite, computed from --outputdir as defined by the
caller of pg_regress (current directory if undefined).

This special handling was introduced as of f10589e to be specific to
MSVC, as we let pg_regress' Makefile handle this cleanup in other
environments.  This moves the cleanup to the MSVC script running
regression tests instead where needed: check, installcheck and
upgradecheck.  I have also checked this patch on MSVC with repeated runs
of each target.

Author: Kyotaro Horiguchi, Michael Paquier
Discussion: https://postgr.es/m/20200219.142519.437573253063431435.horikyota.ntt@gmail.com

5 years agoSync our copy of the timezone library with IANA release tzcode2020a.
Tom Lane [Wed, 17 Jun 2020 22:29:29 +0000 (18:29 -0400)]
Sync our copy of the timezone library with IANA release tzcode2020a.

This absorbs a leap-second-related bug fix in localtime.c, and
teaches zic to handle an expiration marker in the leapseconds file.
Neither are of any interest to us (for the foreseeable future
anyway), but we need to stay more or less in sync with upstream.

Also adjust some over-eager changes in the README from commit 957338418.
I have no intention of making changes that require C99 in this code,
until such time as all the live back branches require C99.  Otherwise
back-patching will get too exciting.

For the same reason, absorb assorted whitespace and other cosmetic
changes from HEAD into the back branches; mostly this reflects use of
improved versions of pgindent.

All in all then, quite a boring update.  But I figured I'd get it
done while I was looking at this code.

5 years agoFix nbtree.h dedup state comment.
Peter Geoghegan [Wed, 17 Jun 2020 22:23:55 +0000 (15:23 -0700)]
Fix nbtree.h dedup state comment.

Oversight in commit 0d861bbb.

5 years agospinlock emulation: Fix bug when more than INT_MAX spinlocks are initialized.
Andres Freund [Mon, 8 Jun 2020 22:25:49 +0000 (15:25 -0700)]
spinlock emulation: Fix bug when more than INT_MAX spinlocks are initialized.

Once the counter goes negative we ended up with spinlocks that errored
out on first use (due to check in tas_sema).

Author: Andres Freund
Reviewed-By: Robert Haas
Discussion: https://postgr.es/m/20200606023103.avzrctgv7476xj7i@alap3.anarazel.de
Backpatch: 9.5-

5 years agoAvoid potential spinlock in a signal handler as part of global barriers.
Andres Freund [Tue, 16 Jun 2020 01:23:10 +0000 (18:23 -0700)]
Avoid potential spinlock in a signal handler as part of global barriers.

On platforms without support for 64bit atomic operations where we also
cannot rely on 64bit reads to have single copy atomicity, such atomics
are implemented using a spinlock based fallback. That means it's not
safe to even read such atomics from within a signal handler (since the
signal handler might run when the spinlock already is held).

To avoid this issue defer global barrier processing out of the signal
handler. Instead of checking local / shared barrier generation to
determine whether to set ProcSignalBarrierPending, introduce
PROCSIGNAL_BARRIER and always set ProcSignalBarrierPending when
receiving such a signal. Additionally avoid redundant work in
ProcessProcSignalBarrier if ProcSignalBarrierPending is unnecessarily.

Also do a small amount of other polishing.

Author: Andres Freund
Reviewed-By: Robert Haas
Discussion: https://postgr.es/m/20200609193723.eu5ilsjxwdpyxhgz@alap3.anarazel.de
Backpatch: 13-, where the code was introduced.