[GENERAL] recovery do not finish

Hello,

I have a serious problem with a production database.
We had a no disk space left on device problem, and postgres did not stop, so it was killed ( kill -9 )
we made free space and reboot, but postgres not start properly.
We have waited for more than 2 hours, but psql still says that the system is starting up.
( Db size is about 20 gb )
the os is Ubuntu 4.0.3-1ubuntu5
postgres is 8.1.4


pg_controldata says the following :
pg_control version number:            812
Catalog version number:               200510211
Database system identifier:           5006307211022564835
Database cluster state:               in recovery
pg_control last modified:             Fri 11 Jul 2008 09:38:54 PM CEST
Current log file ID:                  48
Next log file segment:                205
Latest checkpoint location:           30/C6AECCAC
Prior checkpoint location:            30/C693404C
Latest checkpoint's REDO location:    30/C6AE62A8
Latest checkpoint's UNDO location:    0/0
Latest checkpoint's TimeLineID:       1
Latest checkpoint's NextXID:          1441774700
Latest checkpoint's NextOID:          25908
Latest checkpoint's NextMultiXactId:  1
Latest checkpoint's NextMultiOffset:  0
Time of latest checkpoint:            Fri 11 Jul 2008 05:22:49 PM CEST
Maximum data alignment:               4
Database block size:                  8192
Blocks per segment of large relation: 131072
Bytes per WAL segment:                16777216
Maximum length of identifiers:        64
Maximum columns in an index:          32
Date/time type storage:               64-bit integers
Maximum length of locale name:        128
LC_COLLATE:                           en_US.UTF-8
LC_CTYPE:                             en_US.UTF-8

In the log file we have the following lines :
DEBUG:  TZ "W-SU" scores 0: at 1074121200 2004-01-15 02:00:00 std versus 2004-01-15 00:00:00 std
DEBUG:  TZ "Zulu" scores 0: at 1074121200 2004-01-14 23:00:00 std versus 2004-01-15 00:00:00 std
LOG:  could not load root certificate file "root.crt": No SSL error reported
DETAIL:  Will not verify client certificates.
DEBUG:  invoking IpcMemoryCreate(size=10461184)
DEBUG:  max_safe_fds = 984, usable_fds = 1000, already_open = 6
LOG:  database system was interrupted while in recovery at 2008-07-11 19:08:09 CEST
HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
LOG:  checkpoint record is at 30/C6AECCAC
LOG:  redo record is at 30/C6AE62A8; undo record is at 0/0; shutdown FALSE
LOG:  next transaction ID: 1441774700; next OID: 25908
LOG:  next MultiXactId: 1; next MultiXactOffset: 0
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 30/C6AE62A8
LOG:  connection received: host=[local]
LOG:  incomplete startup packet
DEBUG:  proc_exit(0)
DEBUG:  shmem_exit(0)
DEBUG:  exit(0)
DEBUG:  forked new backend, pid=3760 socket=7
DEBUG:  reaping dead processes
DEBUG:  server process (PID 3760) exited with exit code 0
LOG:  connection received: host=[local]
DEBUG:  forked new backend, pid=3763 socket=7
FATAL:  the database system is starting up
DEBUG:  proc_exit(0)
DEBUG:  shmem_exit(0)
DEBUG:  exit(0)
DEBUG:  reaping dead processes
DEBUG:  server process (PID 3763) exited with exit code 0
LOG:  connection received: host=[local]
FATAL:  the database system is starting up
DEBUG:  proc_exit(0)
DEBUG:  shmem_exit(0)
DEBUG:  exit(0)
DEBUG:  forked new backend, pid=3766 socket=7
DEBUG:  reaping dead processes
DEBUG:  server process (PID 3766) exited with exit code 0
LOG:  connection received: host=[local]
FATAL:  the database system is starting up
DEBUG:  proc_exit(0)
DEBUG:  shmem_exit(0)
DEBUG:  exit(0)

Any help is appreciated

________________________________________________________
VÁSÁROLNA? SEGÍTÜNK! Igényeljen hitelkártyát online és élvezze Ön is a vásárlás szabadságát!