Starting down the ActiveSalesforce path, my first goal was to do a simple dump of a class of objects to yaml using the API.

$ gem install activesalesforce
$ cat - <<EOF > dump_accounts.rb
#!/usr/bin/ruby

require 'rubygems'
require_gem 'activerecord'
require_gem 'activesalesforce'

ActiveRecord::Base.logger = Logger.new(STDERR)

ActiveRecord::Base.establish_connection(
  :adapter => "activesalesforce",
  :url => "https://www.salesforce.com/services/Soap/u/7.0",
  :username => "yourlogin@yourdomain.com",
  :password => "yourpassword"
)

class Account < ActiveRecord::Base
end

puts Account.find(:all).to_yaml
EOF
$ chmod u+rx dump_accounts.rb
$ ./dump_accounts > accounts.yml

Next step: figure out how to handle user authentication for Account Contacts...

The following is a post I've just made to the pgcluster-general mailing list. As it is blog worthy, it seemed appropriate to post here.

I've been testing a pgcluster running 1.5.0rc7 with pgbench 8.0.2.

I have 6 servers in a cluster:

2 pglb servers (2.6 kernel debian, amd sempron 2800, 1G RAM, 2 IDE drives software RAID1) 2 pgreplicate servers (2.6 kernel debian, amd64x2, 4G RAM, 2 IDE drives software RAID1) 2 postgres database servers (2.6 kernel debian, amd64x2, 4G RAM, 4 IDE drives software RAID10)

The pgbench page is:

http://www.sitening.com/tools/postgresql-benchmark/

It's a simple build:

$ wget http://www.sitening.com/pgbench.8.0.2.c
$ gcc -I/usr/include/postgresql -o pgbench pgbench.8.0.2.c -lpq4 -lm

After a bit of postgresql tweaking, I'm finally getting some good numbers (see below).

Things to remember when installing pgcluster:

1. Your fully qualified hostnames must resolve and match the config.
   • add entries to /etc/hosts if you must, but make sure everything uses actual resolvable hostnames.
2. Watch your user process limit (ulimit -u unlimited).
   • on the pglb master: pglb will spawn a thread for each pooled connection.
   • on the pgreplicate master: pgreplicate goes absolutely insane with threads
   • on db nodes: postgres spawns a thread for each incoming connection
3. Your fully qualified hostnames must resolve and match the config.
   • add entries to /etc/hosts if you must, but make sure everything uses actual resolvable hostnames.
4. Don't forget about the cluster.conf buried in the postgres server configuration directory on the db nodes.
5. When you run things with "-v", expect a huge slowdown.
   • pglb drops from 12k tps (using "pgbench -S" for select() only) to only 6 tps. (5 orders of magnitude)
   • pgreplicate -v drops to below 1 tps. (2 orders of magnitude)
6. Setup ssh key trust between servers using the userid that postgres runs as (usually "postgres")
7. Remember to start slaves with pg_ctl -o "-i -R" the first time to pull down the rsync of the master.
   • this killed most of my "weird" deadlocks with select() only pgbench right away.

Back to the pgbench numbers.

The fastest mode of operation is select() only (pgbench -S):

$ ./pgbench -S -n -v -c 10 -t 1000 -m 10
  transaction type: SELECT only
  scaling factor: 1
  number of clients: 10
  number of transactions per client: 1000

  number of transactions actually processed: 10000/10000
  tps = 8394.268729 (including connections establishing)
  tps = 12815.846538 (excluding connections establishing)

  mean tps = 8399.209915 (including connections establishing)
  standard deviation = 202.800265

  mean tps = 12574.325714 (excluding connections establishing)
  standard deviation = 428.241079

Running this in parallel with an insert()/select() mix doesn't seem to impact it much. Meaning, a select() running in parallel with an insert()/select() run only seems to drop the numbers by 1k-2k tps or so.

To run an insert()/select() mix, run pgbench with the -N flag:

$ ./pgbench -N -n -v -c 10 -t 1000

  transaction type: Update only accounts
  scaling factor: 1
  number of clients: 10
  number of transactions per client: 1000

  number of transactions actually processed: 10000/10000
  tps = 115.539752 (including connections establishing)
  tps = 116.069260 (excluding connections establishing)

These numbers are to be expected with a synchronous replication system like pgcluster. As long as the select() to insert()/update() ratio is at least 9:1 things should be usable.

Trying to run the full "TPC-B (sort of)" mode, pgbench starts throwing update() into the mix.

This is where pgbench starts to deadlock for me.

You can add the "-d" flag to pgbench to debug things if it seems frozen.

The first pgcluster "bug":

It looks like I deadlock almost immediately after spawning pgbench with the following arguments:

$ ./pgbench -n -v -c 1 -t 1000 -m 1 -d
  pghost:  pgport: (null) nclients: 1 nxacts: 1000 dbName:
  message type 0x43 arrived from server while idle
  message type 0x5a arrived from server while idle
  client 0 sending begin
  client 0 receiving
  client 0 sending update accounts set abalance = abalance + 216 where aid = 52606

  client 0 receiving
  client 0 sending select abalance from accounts where aid = 52606
  client 0 receiving
  client 0 sending update tellers set tbalance = tbalance + 216 where tid = 7

  client 0 receiving
  client 0 sending update branches set bbalance = bbalance + 216 where bid = 1

  *deadlock*

The odd part is that only that pgbench seems hung. I can spawn any number of "pgbench -S" and "pgbench -N" sessions I want while that one is stuck, and things seem to continue running.

My second pgcluster "bug":

While doing this testing, I've found that pglb chokes if you request more client connections than it can handle.

In my testbed, I upped the max connections to 300 per server (each search tuned to allow 500 client connections), leaving me with 600 pooled connection threads running on my pglb server.

If I hit pglb with 600 available pooled connections with, say, 1000 pgbench connection attempts, pglb goes into a dead state refusing to accept more connections, even after pgbench is killed.

$ ./pgbench -S -n -v -c 10 -t 10000 -m 1
  transaction type: SELECT only
  scaling factor: 1
  number of clients: 10
  number of transactions per client: 10000

  number of transactions actually processed: 100000/100000
  tps = 11486.270576 (including connections establishing)
  tps = 11991.783709 (excluding connections establishing)

$ ./pgbench -S -n -v -c 1000 -t 10 -m 1
  Connection to database '' failed.
  pglb could not connect to server: no cluster available.
  $ ./pgbench -S -n -v -c 10 -t 10 -m 1
  Connection to database '' failed.
  pglb could not connect to server: no cluster available.
or
  Sorry, backend connection is full

After this state occurs, I need to kill off pglb and restart it, and sometimes this doesn't fix it (and I need to go through and restart the replication servers and the database servers).

In conclusion:

I actually have 3 distinct pgclusters going here, each containing 6 of the aforementioned servers, counting a total of 18 servers:

• one dev cluster
• one qa cluster
• one production cluster

My question to the pgcluster list is: what version of pgcluster is "stable" enough to be used in a production environment?

I'd rather not need to go the direction of Slony-I with something like pgpool or dbbalance (to shunt writes to the master), first due to the complexity of managing these layers, and secondly due to the data coherency lost between the master and slaves (I want atomic synchronous replication).

Then again, this is all for hosting a Ruby on Rails application. We can make application changes as needed.

I hope this helps.

Subversion is a wonderful revision control system, until it breaks.

Historically, the BDB store was often corrupted and a "svnadmin recover" is required to rebuild the berkeley databases.

With newer versions of Subversion, FSFS has taken over as the preferred repository store. Faster and much more reliable than BDB, it has made Subversion far more stable than it has been in the past.

Unfortunately, FSFS appears to continue to have rare random corruption issues. Subversion developers are actively working on tracking down the cause, but it remains elusive at the moment.

Though much rarer than its BDB counterpart, repairing a corrupt FSFS store isn't as simple as running "svnadmin recover" - recover only works with BDB.

Recently, we stumbled upon such a cause. The symptoms appear much like the FAQ apr0.9.6 solution, with a difference: local file:// checkouts errored out just the same. As this is an Apache bug fix for poll(), a local checkout really shouldn't be using poll() at all.

Sadly, my users were continuing to commit changes to their subtrees in the repository. As long as they didn't try checking out the corruption affected tree, they could continue doing their work.

Trying to do a "svnadmin verify" results in the same error as the checkout, and an "svnadmin dump" fails just the same (as dump and verify appear to be very similar in nature). Without the ability to dump, it is neigh impossible to backup the revision history to restore to a repository elsewhere. The only daunting alternative was to checkout the various subtrees and hand-commit each revision checkout to another repository... I was suprised to find that an automated script to do such a thing hadn't been written yet.

Step 1 was to visit #svn on irc.openprojects.net and voice my concern at potentially finding an unanswered bug.

The folks on #svn immediately replied "go to #svn-dev and be prepaired to back your claims".

Step 2 then was to visit #svn-dev and lay out the above facts once again.

In the end, the developers suggested I send an email with all of the facts to the subversion user mailing list.

So I put the IRC chat together with the facts and posted "Repository corruption? Problem similar to FAQ#tiger-apr-0.9.6" to the subversion user list.

John Szakmeister soon replied with a suggestion to try:

http://www.szakmeister.net/fsfsverify/

This python script verifies the transactions in a given fsfs revision, and potentially repairs some of the more common problems found.

When run, the following error presented itself:

$ fsfsverify.py db/revs/653
...
NodeRev Id: 1st.g.r653/17936924
 type: file
 pred: 1st.g.r611/34703561
 text: DELTA 653 1668558 16257066 24194048 c0bd2a8b7ee4db1ee816ea607392755d
 prop: UNKNOWN 405 9727810 53 0 113136892f2137aa0116093a524ade0b
 cpath: /cpp/Client/IE/Project/src/Observer.ncb
 copyroot: 178 /cpp/Client/IE
starting length: 16257062
offset: 1668583
Decoded too many bytes
total: 14384890
remaining: 1872172
Traceback (most recent call last):
  File "/root/fsfsverify.py", line 699, in ?
    process(noderev, rev_file, options.dump_instructions,
options.dump_windows)
  File "/root/fsfsverify.py", line 652, in verify
    dump_windows)
  File "/root/fsfsverify.py", line 289, in verify
    digest = parse_svndiff(f, self.length, dump_instructions, dump_windows)
  File "/root/fsfsverify.py", line 188, in parse_svndiff
    raise 'svndiff error'
svndiff error

John suggested that Andrew MacKenzie also reported a similar issue, but he would need a copy of the revision to verify it was the same problem.

Attempting to use fsfsrepair.py's "-f" or "--fix-read-length-line-error" option didn't seem to affect anything at all.

I posted the revision somewhere John could access it.

After looking at it, John thinks this may be a new kind of corruption, and he'll work something into fsfsverify.py to fix it when he can find the time.

In the interim, John suggested truncating the file in the node revision using the following command:

fsfsverify --truncate=1st.g.r653/17936924 653

"This command will basically truncate the file to 0 length in that revision." - John.

Hopefully others will find this blog post along their search for an FSFS corruption fix and consider this potential "fix".

I'm not entirely sure what the data loss will entail, but it may very well solve your immediate problems dumping and restoring the repository elsewhere.

Note: always do an "svnadmin hotcopy" to make a test repository to test on, and immediately take a backup if you can.