Greg Sabino Mullane [Sat, 12 Apr 2014 18:33:50 +0000 (14:33 -0400)]
sqlite: do a BEGIN EXCLUSIVE TRANSACTION in force-lock mode.
Greg Sabino Mullane [Sat, 12 Apr 2014 18:25:42 +0000 (14:25 -0400)]
Put the force-lock-file in the piddir, rather than the globally writeable /tmp
Greg Sabino Mullane [Sat, 12 Apr 2014 17:58:52 +0000 (13:58 -0400)]
Take out the session level isolation level for mysql - our per transaction calls should be good enough.
Greg Sabino Mullane [Sat, 12 Apr 2014 17:50:57 +0000 (13:50 -0400)]
Use READ WRITE isolation levels wherever we can.
Greg Sabino Mullane [Sat, 12 Apr 2014 17:31:05 +0000 (13:31 -0400)]
Have kid_pid_start notification happen on the main channel, not per database
Greg Sabino Mullane [Sat, 12 Apr 2014 17:22:55 +0000 (13:22 -0400)]
Gather the maximum transaction time for all tables in a single UNION query,
rather than making separate SQL calls for each table.
Per suggestion on mailing list from Hans van der Riet
Joshua Tolley [Thu, 10 Apr 2014 17:18:42 +0000 (11:18 -0600)]
Add error NOTIFY when reload fails
Greg Sabino Mullane [Fri, 11 Apr 2014 17:28:17 +0000 (13:28 -0400)]
Exit right away if the delta test fails, for better debugging.
Greg Sabino Mullane [Fri, 11 Apr 2014 17:26:04 +0000 (13:26 -0400)]
When debug is on, write out a small text file to help connect to the test databases via psql.
Greg Sabino Mullane [Fri, 11 Apr 2014 11:39:08 +0000 (07:39 -0400)]
Clean up confusing comment
Greg Sabino Mullane [Fri, 11 Apr 2014 11:38:45 +0000 (07:38 -0400)]
Cleanup the temporary service files
Greg Sabino Mullane [Thu, 10 Apr 2014 22:36:46 +0000 (18:36 -0400)]
Move kid_pid_stop to global listening, not db-based.
Do a ping check before trying to cleanup databases on the kid exception handler.
Greg Sabino Mullane [Thu, 10 Apr 2014 18:25:42 +0000 (14:25 -0400)]
Better debugging for notifys
Greg Sabino Mullane [Thu, 10 Apr 2014 18:00:08 +0000 (14:00 -0400)]
Rewrite the delete_rows function.
Loop through and make sure we have the maximum number of async targets running at all times.
Moved much of the logic to per-target attribs.
Greg Sabino Mullane [Thu, 10 Apr 2014 13:43:28 +0000 (09:43 -0400)]
Debug the initdb call during testing
Greg Sabino Mullane [Thu, 10 Apr 2014 13:42:46 +0000 (09:42 -0400)]
Better tweaking for pgservice items
Greg Sabino Mullane [Wed, 9 Apr 2014 23:20:08 +0000 (19:20 -0400)]
Remove debugging.
Greg Sabino Mullane [Wed, 9 Apr 2014 23:18:20 +0000 (19:18 -0400)]
Better munging if we are using a service
Greg Sabino Mullane [Sun, 6 Apr 2014 20:07:03 +0000 (16:07 -0400)]
Ignore any local .bucardorc files.
Greg Sabino Mullane [Tue, 1 Apr 2014 11:05:29 +0000 (07:05 -0400)]
Remove useless comment
Greg Sabino Mullane [Mon, 31 Mar 2014 13:27:17 +0000 (09:27 -0400)]
Bump copyright to 2014
Joshua Tolley [Thu, 27 Mar 2014 21:01:09 +0000 (15:01 -0600)]
Add test for 'bucardo upgrade'
Joshua Tolley [Thu, 27 Mar 2014 20:51:47 +0000 (14:51 -0600)]
More bucardo.upgrade fixes
Now bucardo upgrade run against a database that doesn't need any changes
correctly reports that no changes are necessary
Greg Sabino Mullane [Thu, 20 Mar 2014 21:08:51 +0000 (17:08 -0400)]
If we are using a service file, make sure we don't default slurp in host and port env variables
Greg Sabino Mullane [Thu, 20 Mar 2014 18:24:35 +0000 (14:24 -0400)]
git should ignore service testing files
Joshua Tolley [Thu, 20 Mar 2014 17:36:27 +0000 (11:36 -0600)]
Modify syntax so the upgrade process can understand it
Greg Sabino Mullane [Mon, 17 Mar 2014 19:38:49 +0000 (15:38 -0400)]
Make the dbservice column have an empty default, so we don't have to start checking "defined or length..." everywhere
Joshua Tolley [Wed, 12 Mar 2014 18:29:44 +0000 (12:29 -0600)]
Revert comment, so it stays in sync with reverted code
Joshua Tolley [Wed, 12 Mar 2014 16:30:47 +0000 (10:30 -0600)]
Finish supporting non-ASCII table and column names
Joshua Tolley [Tue, 18 Feb 2014 20:06:37 +0000 (13:06 -0700)]
Bucardo now accepts pg_service for databases
Joshua Tolley [Fri, 14 Feb 2014 17:59:34 +0000 (10:59 -0700)]
service strings are working; still needs tests
Greg Sabino Mullane [Tue, 18 Feb 2014 14:54:46 +0000 (09:54 -0500)]
Always use async when doing deletes. The previous way had a thinko - this should ensure large deletes run concurrently on different hosts.
Joshua Tolley [Thu, 13 Feb 2014 23:25:17 +0000 (16:25 -0700)]
Make install quit ignoring dbname and dbuser options
... again.
Joshua Tolley [Fri, 14 Feb 2014 19:59:48 +0000 (12:59 -0700)]
Don't assume the bucardo database exists on install
Joshua Tolley [Fri, 14 Feb 2014 18:14:18 +0000 (11:14 -0700)]
Revert "Make install quit ignoring dbname and dbuser options"
I should probably have run the test suite before committing...
This reverts commit
39dcde241abcdfcc45415d2eb210d8b2c694f479.
Joshua Tolley [Thu, 13 Feb 2014 23:25:17 +0000 (16:25 -0700)]
Make install quit ignoring dbname and dbuser options
Joshua Tolley [Thu, 13 Feb 2014 20:55:15 +0000 (13:55 -0700)]
s/vaccuum/vacuum/
Greg Sabino Mullane [Fri, 17 Jan 2014 15:49:09 +0000 (10:49 -0500)]
Make our statement_chunk_size default match up.
Greg Sabino Mullane [Thu, 16 Jan 2014 15:58:40 +0000 (10:58 -0500)]
Debugging code
Greg Sabino Mullane [Thu, 16 Jan 2014 15:42:02 +0000 (10:42 -0500)]
Password should default to undef, not an empty string
Greg Sabino Mullane [Sun, 22 Dec 2013 16:12:52 +0000 (11:12 -0500)]
Gotta have log levels on all glog() calls.
Greg Sabino Mullane [Sun, 22 Dec 2013 16:12:15 +0000 (11:12 -0500)]
Version bump to 4.99.11
David E. Wheeler [Tue, 17 Dec 2013 19:46:21 +0000 (11:46 -0800)]
Eval everything.
David E. Wheeler [Tue, 17 Dec 2013 01:11:53 +0000 (17:11 -0800)]
Someone cannot count.
David E. Wheeler [Tue, 17 Dec 2013 01:00:14 +0000 (17:00 -0800)]
Fix typo in comment.
Greg Sabino Mullane [Tue, 3 Dec 2013 17:10:55 +0000 (12:10 -0500)]
Check @tableoids, not tablelist, as the latter may be empty (inside validate_sync)
Greg Sabino Mullane [Sat, 30 Nov 2013 16:43:45 +0000 (11:43 -0500)]
Meta things for the new dev/ directory
Greg Sabino Mullane [Sat, 30 Nov 2013 16:42:15 +0000 (11:42 -0500)]
Various tests for developers
Greg Sabino Mullane [Sat, 30 Nov 2013 16:41:06 +0000 (11:41 -0500)]
Better makedelta test. Need to move these somewhere else.
Greg Sabino Mullane [Sat, 30 Nov 2013 16:39:34 +0000 (11:39 -0500)]
Fix up some of the makedelta logic.
Greg Sabino Mullane [Fri, 22 Nov 2013 04:24:03 +0000 (23:24 -0500)]
Better handling of inactive databases.
Greg Sabino Mullane [Fri, 22 Nov 2013 03:20:52 +0000 (22:20 -0500)]
Add a fifth inactive database to ensure we handle inactive targets gracefully.
Michelle Sullivan [Sun, 17 Nov 2013 01:57:57 +0000 (20:57 -0500)]
Better cleanup of syncrun entries.
Wrap some of the kid cleanup in evals.
Greg Sabino Mullane [Sun, 17 Nov 2013 01:52:00 +0000 (20:52 -0500)]
Don't package up that MYMETA junk
Greg Sabino Mullane [Fri, 15 Nov 2013 23:06:59 +0000 (18:06 -0500)]
Version 4.99.10
Greg Sabino Mullane [Fri, 15 Nov 2013 23:04:14 +0000 (18:04 -0500)]
Add quick test of sequence-only sync. Needs to do some actual diff testing though.
Greg Sabino Mullane [Fri, 15 Nov 2013 22:45:24 +0000 (17:45 -0500)]
Reduce logging for VAC subprocess to warning - we don't want purging details flooding the logs.
Greg Sabino Mullane [Fri, 15 Nov 2013 22:19:33 +0000 (17:19 -0500)]
User-supplied messages should always be shown.
Greg Sabino Mullane [Thu, 14 Nov 2013 15:01:08 +0000 (10:01 -0500)]
Don't gather column information if there are no columns! Fix for validate_sync and a table-less sync.
Greg Sabino Mullane [Thu, 14 Nov 2013 03:13:47 +0000 (22:13 -0500)]
Don't populate bucardo_delta_names if no tables (e.g. sequence only sync)
Greg Sabino Mullane [Tue, 12 Nov 2013 18:14:36 +0000 (13:14 -0500)]
Signature for 4.99.9
Greg Sabino Mullane [Tue, 12 Nov 2013 18:13:01 +0000 (13:13 -0500)]
Version 4.99.9
Greg Sabino Mullane [Tue, 12 Nov 2013 17:59:27 +0000 (12:59 -0500)]
Fix quoting for inserts to the bucardo_deltanames table in validate_sync.
Greg Sabino Mullane [Tue, 12 Nov 2013 05:16:54 +0000 (00:16 -0500)]
Initialize var
Greg Sabino Mullane [Tue, 12 Nov 2013 05:15:23 +0000 (00:15 -0500)]
Fixes for the new bucardo_delta_names table.
Greg Sabino Mullane [Tue, 12 Nov 2013 05:02:33 +0000 (00:02 -0500)]
Make sure we specify schema
Greg Sabino Mullane [Tue, 12 Nov 2013 04:39:38 +0000 (23:39 -0500)]
Always check pg_async_status before calling pg_cancel
Greg Sabino Mullane [Tue, 12 Nov 2013 04:20:57 +0000 (23:20 -0500)]
Stupid tabs
Greg Sabino Mullane [Tue, 12 Nov 2013 04:19:59 +0000 (23:19 -0500)]
Make sure we only try to populate bucardo_delta_names for source databases.
Greg Sabino Mullane [Sat, 9 Nov 2013 19:31:24 +0000 (14:31 -0500)]
On FK warnings, show the contraint name, otherwise we often have duplicated we cannot distinguish.
Greg Sabino Mullane [Sat, 9 Nov 2013 18:18:46 +0000 (13:18 -0500)]
Show the host in our connection logging if we have one.
Greg Sabino Mullane [Sat, 9 Nov 2013 18:10:40 +0000 (13:10 -0500)]
Testing putting a pg_cancel at the end of the kid before the rollback.
Greg Sabino Mullane [Sat, 9 Nov 2013 18:06:57 +0000 (13:06 -0500)]
Don't show the initial DSN to the Bucardo database
Greg Sabino Mullane [Fri, 8 Nov 2013 19:41:35 +0000 (14:41 -0500)]
We don't need to check database wide when we already check database/table combo anyway.
Greg Sabino Mullane [Fri, 8 Nov 2013 04:53:09 +0000 (23:53 -0500)]
Introduce the concept of a "quick delta" check.
By default, this is turned on, but is left as a config variable in case this turns out to be a very bad idea. :)
This is a huge win on slow networks. Rather than getting a distinct pk count for every table in the sync, we ask the remote
database for a list of all tables and whether or not they have any changes. If they don't, we do not have to bother
with the normal "select distinct" query. This reduces the number of database calls drastically for syncs with a
large number of tables. In addition, the delta quick queries inside the function are extraorfinarily fast compared
to their distinct counterparts, as they do a select 1 ... limit 1.
All of this is accomplished by a new table on each source database called bucardo_delta_names, which stores a
list of all tables for a sync, along with their delta and track table names. This table's information is rewritten
on Bucardo startup each time. The new function uses this table to dynamically generate the quick delta queries
for each table, and then returns it in a simple text format.
This is especially noticeable on slow networks with a large number of tables in a sync. Testing in the case
that drove all this improved the sync run speed from 42 seconds to less than a second (for no rows - the
number of rows is a constant cost limited by how fast COPY goes, in most cases).
Greg Sabino Mullane [Wed, 6 Nov 2013 16:56:55 +0000 (11:56 -0500)]
Let's try this again...
Greg Sabino Mullane [Wed, 6 Nov 2013 16:55:15 +0000 (11:55 -0500)]
Typo.
Greg Sabino Mullane [Tue, 5 Nov 2013 23:20:31 +0000 (18:20 -0500)]
Show chunk when copying
Greg Sabino Mullane [Tue, 5 Nov 2013 20:31:13 +0000 (15:31 -0500)]
Change default statement_chunk_size from 10000 to 8000, based on real-world experience.
Greg Sabino Mullane [Tue, 5 Nov 2013 16:47:29 +0000 (11:47 -0500)]
Use prepare_cached, not prepare, inside of fetch1_sql
Greg Sabino Mullane [Tue, 5 Nov 2013 16:18:30 +0000 (11:18 -0500)]
Typo: compress_delta is a function, not a table
Greg Sabino Mullane [Tue, 29 Oct 2013 04:37:13 +0000 (00:37 -0400)]
Cache the column lookups inside of validate_sync.
Greg Sabino Mullane [Fri, 25 Oct 2013 03:12:12 +0000 (23:12 -0400)]
Make sure we do something sensible if we call for a log_level before we have loaded the config.
Greg Sabino Mullane [Thu, 24 Oct 2013 03:36:59 +0000 (23:36 -0400)]
Tabs cleanup
Greg Sabino Mullane [Thu, 24 Oct 2013 03:35:29 +0000 (23:35 -0400)]
Replace 'haztrig' with cached version inside of validate_sync.
Greg Sabino Mullane [Thu, 24 Oct 2013 02:38:56 +0000 (22:38 -0400)]
Show the DSN we are connecting with
Greg Sabino Mullane [Wed, 23 Oct 2013 14:58:49 +0000 (10:58 -0400)]
Report on total sync time in the logs, even if no rows were changed.
Greg Sabino Mullane [Wed, 23 Oct 2013 03:38:26 +0000 (23:38 -0400)]
Only try and create that index once.
Greg Sabino Mullane [Wed, 23 Oct 2013 03:29:22 +0000 (23:29 -0400)]
Speed up the initial validate sync quite a bit by pre-loading all the table, index, and function existence information.
Greg Sabino Mullane [Wed, 23 Oct 2013 02:16:06 +0000 (22:16 -0400)]
No longer use the whole prefix when building the listen/notify strings, as it may contain a syncname now.
Greg Sabino Mullane [Sun, 20 Oct 2013 02:10:32 +0000 (22:10 -0400)]
When doing `bucardo list databases`, don't show the psql port connection param unless it is set to something.
Greg Sabino Mullane [Sat, 19 Oct 2013 01:41:56 +0000 (21:41 -0400)]
Fix wrong search before inserting to bucardo_delta_targets.
This would explain the large number of rows seen in that table in the wild!
Greg Sabino Mullane [Fri, 18 Oct 2013 04:12:02 +0000 (00:12 -0400)]
Expand the arrays in the opening config dump in the logs.
Greg Sabino Mullane [Fri, 18 Oct 2013 03:41:20 +0000 (23:41 -0400)]
Enforce sane isolation levels
Greg Sabino Mullane [Fri, 18 Oct 2013 02:42:28 +0000 (22:42 -0400)]
Add new config 'log_showsyncname', defaulting to 1 (on), which will show the name of the current sync in parens after the 'KID' in the log output.
So rather than this:
(29985) [Thu Oct 17 22:40:12.885 2013] KID Database "A" backend PID: 29987
You will see this:
(29985) [Thu Oct 17 22:40:12.885 2013] KID (fctest) Database "A" backend PID: 29987
As suggested by Michelle Sullivan on the mailing list.
Greg Sabino Mullane [Fri, 18 Oct 2013 02:30:52 +0000 (22:30 -0400)]
Typo.
Greg Sabino Mullane [Thu, 17 Oct 2013 03:12:42 +0000 (23:12 -0400)]
Update versions in html docs
Greg Sabino Mullane [Thu, 17 Oct 2013 03:10:44 +0000 (23:10 -0400)]
Fix some pod
Greg Sabino Mullane [Thu, 17 Oct 2013 02:59:57 +0000 (22:59 -0400)]
Don't remember why this was here, but mark it as LOG_DEBUG
Greg Sabino Mullane [Thu, 17 Oct 2013 02:58:51 +0000 (22:58 -0400)]
Away with ye, tab