[GENERAL] pg_standby stuck on a wal file size <16MB

I've been having problem with pgsql-8.2.5 master/slave warm standby
replication setup where occasionally master node generates a wal file
smaller then expected 16MB. pg_standby on slave gets stuck on such short
files, and replication halts from that moment on. we have to do
pg_start_backup/ rsync of data to slave / pg_stop_backup and restart
slave in order to recover. database I'm replicating is write mostly.

this is process on slave that is waiting on log file which is smaller in
size:
postgres 14277 11857 0 03:34 ? 00:00:00 sh -c
/usr/local/pgsql/bin/pg_standby -d -k 255 -r 7 -s 10 -w 0 -t
/usr/local/recovery_control/quit_recovery /usr/local/wal_archive
000000010000034000000020 pg_xlog/RECOVERYXLOG 2>>
/usr/local/pgsql/data/standby.log
postgres 14278 14277 0 03:34 ? 00:00:00
/usr/local/pgsql/bin/pg_standby -d -k 255 -r 7 -s 10 -w 0 -t
/usr/local/recovery_control/quit_recovery /usr/local/wal_archive
000000010000034000000020 pg_xlog/RECOVERYXLOG

here is a sample list of wal_archive directory slave, note the size of
000000010000034000000020 expected wal file is less then 16MB:

...
-rw------- 1 postgres postgres 16777216 May 17 03:19
000000010000034000000017
-rw------- 1 postgres postgres 16777216 May 17 03:21
000000010000034000000018
-rw------- 1 postgres postgres 16777216 May 17 03:22
000000010000034000000019
-rw------- 1 postgres postgres 16777216 May 17 03:24
00000001000003400000001A
-rw------- 1 postgres postgres 16777216 May 17 03:26
00000001000003400000001B
-rw------- 1 postgres postgres 16777216 May 17 03:27
00000001000003400000001C
-rw------- 1 postgres postgres 16777216 May 17 03:29
00000001000003400000001D
-rw------- 1 postgres postgres 16777216 May 17 03:30
00000001000003400000001E
-rw------- 1 postgres postgres 16777216 May 17 03:32
00000001000003400000001F

-rw------- 1 postgres postgres 13746176 May 17 03:34
000000010000034000000020

-rw------- 1 postgres postgres 16777216 May 17 03:35
000000010000034000000021
-rw------- 1 postgres postgres 16777216 May 17 03:37
000000010000034000000022
-rw------- 1 postgres postgres 16777216 May 17 03:38
000000010000034000000023
-rw------- 1 postgres postgres 16777216 May 17 03:40
000000010000034000000024
-rw------- 1 postgres postgres 16777216 May 17 03:41
000000010000034000000025
-rw------- 1 postgres postgres 16777216 May 17 03:43
000000010000034000000026
-rw------- 1 postgres postgres 16777216 May 17 03:45
000000010000034000000027
...

skipping a bit further I see there is at least one other instance where
wal file is shorter then normal 16MB:

-rw------- 1 postgres postgres 16777216 May 17 05:42
00000001000003400000006F
-rw------- 1 postgres postgres 16777216 May 17 05:44
000000010000034000000070
-rw------- 1 postgres postgres 16777216 May 17 05:46
000000010000034000000071
-rw------- 1 postgres postgres 16777216 May 17 05:47
000000010000034000000072

-rw------- 1 postgres postgres 16384 May 17 05:50
000000010000034000000073

-rw------- 1 postgres postgres 16777216 May 17 05:51
000000010000034000000074
-rw------- 1 postgres postgres 16777216 May 17 05:52
000000010000034000000075
-rw------- 1 postgres postgres 16777216 May 17 05:54
000000010000034000000076

why would a master node create a wal file smaller then normal
checkpoint_segment size and how can this be avoided. I need a reliable
replication mechanism even at a cost of longer recovery on standby.

master's postgresql.conf:
..
fsync = on
wal_sync_method = open_sync
wal_buffers = 128
checkpoint_segments = 64
archive_command = 'test ! -f /usr/local/wal_archive_local/%f && cp %p
/usr/local/wal_archive_local/%f'

archive files are then moved on master to standby every other minute:

rsync -aq --remove-sent-files /usr/local/wal_archive_local/
slave::wal_archive/

slave's recovery.conf:
restore_command = '/usr/local/pgsql/bin/pg_standby -d -k 255 -r 7 -s 10
-w 0 -t /usr/local/recovery_control/quit_recovery /usr/local/wal_archive
%f %p 2>> /u
sr/local/pgsql/data/standby.log'

both servers are identical Dell PE1950 servers with 4 sas hd w/hardware
RAID 1+0 running:
2.6.18-8.1.8.el5 #1 SMP Tue Jul 10 06:39:17 EDT 2007 x86_64 x86_64
x86_64 GNU/Linux

Thanks much in advance
V.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general