wal_decoding: Introduce wal decoding via catalog timetravel
authorAndres Freund <andres@anarazel.de>
Thu, 5 Apr 2012 09:07:33 +0000 (11:07 +0200)
committerAndres Freund <andres@anarazel.de>
Fri, 17 May 2013 16:40:21 +0000 (18:40 +0200)
commite8eddd40d941dd6bb15505dda975b830005bc006
tree3ad42506c7ca7aa6bacbcf56ec2172db06eb4e5a
parent536530c9dca336d6296be90b3319fe575029b5eb
wal_decoding: Introduce wal decoding via catalog timetravel

This introduces several things:
* 'reorderbuffer' module which reassembles transactions from a stream of interspersed changes
* 'snapbuilder' which builds catalog snapshots so that tuples from wal can be understood
* logging more data into wal to facilitate logical decoding
* wal decoding into an reorderbuffer
* shared library output plugins with 5 callbacks
 * init
 * begin
 * change
 * commit
* walsender infrastructur to stream out changes and to keep the global xmin low enough
 * INIT_LOGICAL_REPLICATION $plugin; waits till a consistent snapshot is built and returns
   * initial LSN
   * replication slot identifier
   * id of a pg_export() style snapshot
 * START_LOGICAL_REPLICATION $id $lsn; streams out changes
 * uses named output plugins for output specification

Todo:
* better integrated testing infrastructure
* more frequent lowering of xmins
* more docs about the internals
* support for user declared catalog tables

Lowlevel:
* REINDEX/VACUUM FULL support for timetravel txns
* resource owner handling is suboptimal
* invalidations from uninteresting transactions (e.g. from other databases, old ones)
  need to be processed anyway
* error handling in walsender is suboptimal
* walsender slots need to be marked as catchup
* pg_receivellog needs to send a reply immediately when postgres is shutting down

Input, Testing and Review by:
Heikki Linnakangas
Kevin Grittner
Michael Paquier
Abhijit Menon-Sen
Peter Gheogegan
Robert Haas
Simon Riggs
Steve Singer

Code By:
Andres Freund

With Contributions by:
Abhijit Menon-Sen
Craig Ringer
Alvaro Herrera
57 files changed:
src/backend/access/common/reloptions.c
src/backend/access/heap/heapam.c
src/backend/access/heap/pruneheap.c
src/backend/access/index/indexam.c
src/backend/access/rmgrdesc/heapdesc.c
src/backend/access/rmgrdesc/xlogdesc.c
src/backend/access/transam/twophase.c
src/backend/access/transam/xact.c
src/backend/access/transam/xlog.c
src/backend/catalog/index.c
src/backend/catalog/system_views.sql
src/backend/commands/trigger.c
src/backend/commands/vacuumlazy.c
src/backend/postmaster/postmaster.c
src/backend/replication/Makefile
src/backend/replication/logical/Makefile [new file with mode: 0644]
src/backend/replication/logical/decode.c [new file with mode: 0644]
src/backend/replication/logical/logical.c [new file with mode: 0644]
src/backend/replication/logical/logicalfuncs.c [new file with mode: 0644]
src/backend/replication/logical/reorderbuffer.c [new file with mode: 0644]
src/backend/replication/logical/snapbuild.c [new file with mode: 0644]
src/backend/replication/repl_gram.y
src/backend/replication/repl_scanner.l
src/backend/replication/walsender.c
src/backend/storage/ipc/ipci.c
src/backend/storage/ipc/procarray.c
src/backend/storage/ipc/standby.c
src/backend/utils/cache/inval.c
src/backend/utils/cache/relcache.c
src/backend/utils/misc/guc.c
src/backend/utils/misc/postgresql.conf.sample
src/backend/utils/time/snapmgr.c
src/backend/utils/time/tqual.c
src/bin/initdb/initdb.c
src/bin/pg_controldata/pg_controldata.c
src/include/access/heapam_xlog.h
src/include/access/transam.h
src/include/access/xlog.h
src/include/access/xlogreader.h
src/include/catalog/pg_proc.h
src/include/nodes/nodes.h
src/include/nodes/replnodes.h
src/include/replication/decode.h [new file with mode: 0644]
src/include/replication/logical.h [new file with mode: 0644]
src/include/replication/logicalfuncs.h [new file with mode: 0644]
src/include/replication/output_plugin.h [new file with mode: 0644]
src/include/replication/reorderbuffer.h [new file with mode: 0644]
src/include/replication/snapbuild.h [new file with mode: 0644]
src/include/replication/walsender_private.h
src/include/storage/itemptr.h
src/include/storage/lwlock.h
src/include/storage/sinval.h
src/include/utils/rel.h
src/include/utils/relcache.h
src/include/utils/snapmgr.h
src/include/utils/tqual.h
src/test/regress/expected/rules.out