Wednesday, September 24, 2008

Re: [BUGS] BUG #4436: (E'\\' LIKE E'\\') => f

Bruce Momjian <bruce@momjian.us> writes:
> Mathieu Fenniak wrote:
>> I noticed that (SELECT E'\\' LIKE E'\\') returns false,

> I believe this is caused because backslash is the default escape
> character for LIKE, so you need:
> test=> SELECT E'\\' LIKE E'\\\\';

Yeah. The given case is actually an invalid LIKE pattern. I wonder
whether we should make LIKE throw error for an invalid pattern.
You get an error for the corresponding case in regex:

regression=# select E'\\' ~ E'\\';
ERROR: invalid regular expression: invalid escape \ sequence

but IIRC the LIKE code just silently ignores a trailing escape
character.

regards, tom lane

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Re: [GENERAL] Returning NEW in an on-delete trigger

Jeff Davis wrote:
> On Thu, 2008-09-18 at 15:04 -0400, Tom Lane wrote:
> > This does seem like a bit of a gotcha for someone who writes RETURN NEW
> > instead of RETURN OLD or vice versa, but I'm not sure how much we can do
> > about that. Lots of people like to write triggers that fire on multiple
> > event types, so we couldn't throw a syntax error for such a reference.
> > A runtime error for a use of the variable might be possible, but a quick
> > look at the code doesn't make it look easy.
> >
>
> Here's a doc patch that may clear up some of the confusion.

Thanks, patch applied.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[COMMITTERS] pgsql: Add documentation about when trigger values NEW/OLD return NULL.

Log Message:
-----------
Add documentation about when trigger values NEW/OLD return NULL.

Jeff Davis

Modified Files:
--------------
pgsql/doc/src/sgml:
plpgsql.sgml (r1.133 -> r1.134)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/doc/src/sgml/plpgsql.sgml?r1=1.133&r2=1.134)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [HACKERS] Interval literal rounding bug(?) and patch.

Ron Mayer wrote:
> I think it's a bug that these 3 different ways of writing 0.7 seconds
> produce different results from each other on HEAD.
>
> head=# select interval '0:0:0.7', interval '@ 0.70 secs', interval '0.7 seconds';
> interval | interval | interval
> -------------+-----------------+-----------------
> 00:00:00.70 | 00:00:00.699999 | 00:00:00.699999
> (1 row)
>
> The attached patch will make all of those output "00:00:00.70" which.
>
> Postgres 8.3 tended to output the "00:00:00.70" like this patch, I believe
> because it didn't default to HAVE_INT64_TIMESTAMP like HEAD is. The patch
> seems to pass the existing regression tests.
>
> Does this seem reasonable?

Yes, very nice; patch applied. Thanks.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal of SE-PostgreSQL patches (for CommitFest:Sep)

Josh Berkus <josh@agliodbs.com> writes:
> Peter,
>> Yeah, but do we even have the slightest bit of information about what
>> exactly would be required to achieve the required levels? And whether
>> this patch does it? And whether there would be alternative, more
>> desirable ways to achieve a similar level?

> Actually, I have some direct communication that SEPostgres will lead to
> PostgreSQL being widely adopted in at least one US government agency.

Does that mean that they have looked at this specific patch and
concluded that it meets their requirements? Or just that SELinux
is a checkbox item for them?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[COMMITTERS] pgsql: Fix integral timestamps so the output is consistent in all cases

Log Message:
-----------
Fix integral timestamps so the output is consistent in all cases to
round:

select interval '0:0:0.7', interval '@ 0.70 secs', interval '0.7
seconds';

Ron Mayer

Modified Files:
--------------
pgsql/src/backend/utils/adt:
datetime.c (r1.193 -> r1.194)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/utils/adt/datetime.c?r1=1.193&r2=1.194)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [ADMIN] Hex representation

I used this very simple little php script to make this

filename: mk55:
#!/usr/bin/php -q
<?php
for ($i=0;$i<262144;$i++){
print chr(85);
}
?>

and ran it:

./mk55 > 55

ls -l 55
-rw-r--r-- 1 smarlowe smarlowe 262144 2008-09-24 13:41 55

i.e. it's 256k.

And it's attached.

On Wed, Sep 24, 2008 at 1:20 PM, Carol Walter <walterc@indiana.edu> wrote:
> Hello,
>
> Does anyone know what the format of hex characters for postgres are? I'm
> trying to create files that contain a 0x55. It looks to me like it should
> require a delimiter of some sort between the characters. I don't know how
> postgres would know that the string was a hex representation and not just a
> character zero, followed by a character x, followed by a character 5,
> followed by a character 5.
>
> Carol
>
> --
> Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin
>

Re: [GENERAL] Oracle and Postgresql

On Wed, Sep 24, 2008 at 1:02 PM, Casey Allen Shobe <cshobe@bepress.com> wrote:

> A knowledgeable PostgreSQL DBA can make significantly more than an Oracle
> DBA as they're a scarcer resource and generally higher quality on average.
> But it may be harder for them to find work - they may end up having to
> move, telecommute, or commute a longer distance simply because there are
> less PostgreSQL shops. It also means a much higher probability of working
> for a small-medium company versus a corporation.
>
> An Oracle DBA can be a lot lazier, and lean on the vendor a lot more. There
> are open Oracle DBA positions everywhere, and it is very easy for them to
> find another job, so learning a lot and focusing on doing a good job are not
> important to them. In the corporate environment in which most of these jobs
> are, they are responsible for far less in their job role, whereas the
> PostgreSQL DBA tends to end up responsible for a lot more pieces of the
> puzzle.

These two paragraphs really ring true for me. I've yet to meet an
oracle dba who was the jack of all trades that being a postgresql DBA
requires.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] pgsql: Make LC_COLLATE and LC_CTYPE database-level settings.

Chris Browne wrote:
> Let me report a problem with this...
>
> Running pg_dump encounters the following:
>
> chris@dba2:Slony-I/CMD/slony1-HEAD/tests> pg_dump -s slonyregress1
> pg_dump: column number -1 is out of range 0..7
> pg_dump: column number -1 is out of range 0..7
> zsh: segmentation fault pg_dump -s slonyregress1

Thanks, fixed. I changed the column names from "collate" and "ctype" to
"datcollate" and "datctype" at the last minute, and clearly didn't test
pg_dump after that.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[COMMITTERS] pgsql: Fix pg_dump bug in the database-level collation patch.

Log Message:
-----------
Fix pg_dump bug in the database-level collation patch. "datcollate" and
"datctype" columns were misspelled. Per report from Chris Browne.

Modified Files:
--------------
pgsql/src/bin/pg_dump:
pg_dump.c (r1.501 -> r1.502)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/bin/pg_dump/pg_dump.c?r1=1.501&r2=1.502)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [ADMIN] Hex representation

Carol Walter написа:
> Hello,
>
> Does anyone know what the format of hex characters for postgres are?
> I'm trying to create files that contain a 0x55. It looks to me like it
> should require a delimiter of some sort between the characters. I don't
> know how postgres would know that the string was a hex representation
> and not just a character zero, followed by a character x, followed by a
> character 5, followed by a character 5.


http://www.postgresql.org/docs/current/static/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS

Look for "Also supported are \digits,..." in section "4.1.2.1. String Constants".


--
Milen A. Radev


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Obfuscated stored procedures (was Re: [GENERAL] Oracle and Postgresql)

> C is not magic obfuscation gear. Anybody with a debugger
> can expose
> what it's doing. There have been math papers showing
> that it's
> impossible to hide the functionality of a piece of software
> based only
> on the ability to run it, so the entire prospect of
> obscuring the
> software's functionality when people can send arbitrary
> inputs to it
> is one of those "known-impossible" problems like
> the halting problem.

And the first word in the title is "obfuscated", not encrypted, secured or anything else...


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches

KaiGai Kohei wrote:
> I updated the series of patches for SE-PostgreSQL 8.4devel.
>
> [1/5] http://sepgsql.googlecode.com/files/sepostgresql-sepgsql-8.4devel-3-r1043.patch
> [2/5] http://sepgsql.googlecode.com/files/sepostgresql-pg_dump-8.4devel-3-r1043.patch
> [3/5] http://sepgsql.googlecode.com/files/sepostgresql-policy-8.4devel-3-r1043.patch
> [4/5] http://sepgsql.googlecode.com/files/sepostgresql-docs-8.4devel-3-r1043.patch
> [5/5] http://sepgsql.googlecode.com/files/sepostgresql-tests-8.4devel-3-r1043.patch

I looked over the patches listed above. They have line counts listed
below:

10759 sepostgresql-sepgsql-8.4devel-3-r1043.patch
616 sepostgresql-pg_dump-8.4devel-3-r1043.patch
826 sepostgresql-policy-8.4devel-3-r1043.patch
1237 sepostgresql-docs-8.4devel-3-r1043.patch
836 sepostgresql-tests-8.4devel-3-r1043.patch
14274 total

Particularly interesting was the doc patch,
sepostgresql-docs-8.4devel-3-r1043.patch. It explains how SE-PostgreSQL
checks the permission level of the client process (getpeercon) and uses
that to determine what the user should see. Also interesting is how a
new row-level system permission column references a new table
'pg_security', which holds security credentials for the row.

The bulk of the patch is in sepostgresql-sepgsql-8.4devel-3-r1043.patch,
which modifies the backend. About 30% of it or 3k lines modify our
backend, and the rest are indepdendent support routines in their own C
files.

So, I am now reevaluating how we should proceed with this patch.

I think we know we want column-level permissions and that is being
worked on, so it should reduce the size of the 3k part of that patch
slightly.

As far as backend changes the largest part is the row-level permissions.
Do we want row-level permissions to be accessible at the SQL level,
perhaps optionally, by having a role be associated with a row, and only
role members can see it. If we do, and implement it, the 3k part is
reduced significantly.

FYI, SE-PostgreSQL sets _row_ permissions by assigning to the new
permissions system column:

INSERT INTO drink (security_context, id, name, price)
VALUES('system_u:object_r:sepgsql_table_t:SystemHigh', 7, 'tea', 130);
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

That string is what is placed in 'pg_security' and a reference to is
placed on the row.

The other conclusion I came to is that the other 11k of patch is really
independent SE-Linux interface code and not likely to change in size no
matter what we implement at the SQL level.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[ADMIN] Postgres Stats after Crash Recovery


Testing AutoVac on 8.3 , i came across the problem of loosing stats data, which was discussed in my last post

http://archives.postgresql.org/pgsql-general/2008-08/msg00198.php

that problem was recognized that doing hard stop, server will throw away the stats while going through  crash recovery.

Problem i see is after crash recovery , we have to manually analyze database in order for autovac to work but it not working as expected.

Here is test case:

foo=# SELECT version();
                                            version                                            
------------------------------------------------------------------------------------------------
 PostgreSQL 8.3.3 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 (Ubuntu 4.1.2-0ubuntu4)
(1 row)

foo=# SELECT * from pg_stat_user_tables where relname='accounts';
-[ RECORD 1 ]----+------------------------------
relid            | 57350
schemaname       | public
relname          | accounts
seq_scan         | 1
seq_tup_read     | 1000000
idx_scan         | 0
idx_tup_fetch    | 0
n_tup_ins        | 1000000
n_tup_upd        | 0
n_tup_del        | 0
n_tup_hot_upd    | 0
n_live_tup       | 1000000
n_dead_tup       | 0
last_vacuum      | 2008-09-24 15:04:35.384012-04
last_autovacuum  |
last_analyze     | 2008-09-24 15:04:35.384012-04
last_autoanalyze |


Next i will stop DB immediate and expect to loose stats as normal behavior.

pg_ctl -m immediate -D /opt/rg/foo/pgsql83 stop
waiting for server to shut down...LOG:  received immediate shutdown request
 done
server stopped

After stating the DB, as expected:
foo=# SELECT * from pg_stat_user_tables where relname='accounts';
-[ RECORD 1 ]----+---------
relid            | 57350
schemaname       | public
relname          | accounts
seq_scan         | 0
seq_tup_read     | 0
idx_scan         | 0
idx_tup_fetch    | 0
n_tup_ins        | 0
n_tup_upd        | 0
n_tup_del        | 0
n_tup_hot_upd    | 0
n_live_tup       | 0
n_dead_tup       | 0
last_vacuum      |
last_autovacuum  |
last_analyze     |
last_autoanalyze |

Next step is to manually analyse to collects the stats again:
foo=# ANALYZE ;
ANALYZE
foo=# SELECT * from pg_stat_user_tables where relname='accounts';
-[ RECORD 1 ]----+---------
relid            | 57350
schemaname       | public
relname          | accounts
seq_scan         | 0
seq_tup_read     | 0
idx_scan         | 0
idx_tup_fetch    | 0
n_tup_ins        | 0
n_tup_upd        | 0
n_tup_del        | 0
n_tup_hot_upd    | 0
n_live_tup       | 0
n_dead_tup       | 0
last_vacuum      |
last_autovacuum  |
last_analyze     |
last_autoanalyze |

note: After ANALYSE, ststs were not updated.

Running ANALYSE, second time seems to does the trick.

foo=# ANALYZE ;
ANALYZE
foo=# SELECT * from pg_stat_user_tables where relname='accounts';
-[ RECORD 1 ]----+------------------------------
relid            | 57350
schemaname       | public
relname          | accounts
seq_scan         | 0
seq_tup_read     | 0
idx_scan         | 0
idx_tup_fetch    | 0
n_tup_ins        | 0
n_tup_upd        | 0
n_tup_del        | 0
n_tup_hot_upd    | 0
n_live_tup       | 1000062
n_dead_tup       | 0
last_vacuum      |
last_autovacuum  |
last_analyze     | 2008-09-24 15:13:13.423424-04
last_autoanalyze |


So question i have is, is this normal operation,why we need to analyze twice to updates the stats ? if table/tables are candidate for vacuuming after crash recovery will never get auto-vac unless you do 'ANALYZE' twice.

Thanks in advance,

Chirag Dave
DBA
Afilias







Re: [pgsql-es-ayuda] Obtener nombre de funcion

2008/9/24 Conrado Blasetti <conrado@mapfre.com.ar>:
> Buenos días a todos.
> Dentro de una funciones plpgsql, como puedo saber el nombre de la función donde estoy parado?
>
> Ej.
>
>
> CREATE FUNCTION test() RETURNS TEXT AS
> $BODY$
> DECLARE
> BEGIN
> RETURN [acá necesitaría que se rescate el nombre la de function, en este caso 'test()' ]::TEXT;
> END;
> $BODY$
> LANGUAGE 'plpgsql' VOLATILE;
>
> Desde ya, gracias.
> Saludos,
> --
> TIP 8: explain analyze es tu amigo
>


Creo que no hay.
Lo que podés hacer es agregarle a cada función una línea en el declare ...

DECLARE
var_my_func varchar default 'test()';

Y luego usar var_my_func como nombre de función

Saludos,
Silvio

--
Silvio Quadri
--
TIP 6: ¿Has buscado en los archivos de nuestra lista de correo?
http://archives.postgresql.org/pgsql-es-ayuda

Re: [PERFORM] query planner and scanning methods

On Tue, Sep 23, 2008 at 3:57 PM, Richard Broersma
<richard.broersma@gmail.com> wrote:
> SELECT A."dimension_book"."call", SUM( B."dimension_book"."call" ) AS
> OrderedRowNbr
> FROM ( your_above_query_without_the_limits ) AS A
> INNER JOIN ( your_above_query_without_the_limits ) AS B
> ON A."dimension_book"."call" >= B."dimension_book"."call"
> ORDER BY A."dimension_book"."call"
> HAVING SUM( A."dimension_book"."call" ) % 10 = 0;

Oops I just noticed that I used sum() where count() should be used and
that I forgot to include the group by clause. Other than that, I hope
the suggestion was at least halfway helpful.

--
Regards,
Richard Broersma Jr.

Visit the Los Angeles PostgreSQL Users Group (LAPUG)
http://pugs.postgresql.org/lapug

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

[ADMIN] Hex representation

Hello,

Does anyone know what the format of hex characters for postgres are?
I'm trying to create files that contain a 0x55. It looks to me like
it should require a delimiter of some sort between the characters. I
don't know how postgres would know that the string was a hex
representation and not just a character zero, followed by a character
x, followed by a character 5, followed by a character 5.

Carol

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [ADMIN] missing chunk number 0 for toast value

I ran into this issue awhile ago. Here's my long internal tech note to my
dev guys on what I did. A bit modified for more genericism:

I'm in the process of migrating our internal db server, and I decided to use
the helpdesk as
my test database. It backed up fine last night. Something went horribly
wrong today, since pg_dump tells me:

> pg_dump: ERROR: missing chunk number 0 for toast value 110439697
> pg_dump: SQL command to dump the contents of table "attachments" failed:
> PQendcopy() failed.
> pg_dump: Error message from server: ERROR: missing chunk number 0 for
toast
> value 110439697
> pg_dump: The command was: COPY public.attachments (id, transactionid,
> parent, messageid, subject, filename, contenttype, contentencoding,
content,
> headers, creator, created) TO stdout;


I reindexed attachments. I reindexed the db. I retoasted the index. Or
reindexed the toast. Or toasted the index. Or something:

rt3=# select reltoastrelid::regclass from pg_class where relname =
'attachments';
reltoastrelid
---------------------------
pg_toast.pg_toast_8507627
(1 row)

rt3=# reindex table pg_toast.pg_toast_8507627;
REINDEX
rt3=# \q
-bash-2.05b$ pg_dump rt3 > /tmp/rt3

pg_dump: ERROR: missing chunk number 0 for toast value 110439697
pg_dump: SQL command to dump the contents of table "attachments" failed:
PQendcopy() failed.
pg_dump: Error message from server: ERROR: missing chunk number 0 for toast
value 110439697
pg_dump: The command was: COPY public.attachments (id, transactionid,
parent, messageid, subject, filename, contenttype, contentencoding, content,
headers, creator, created) TO stdout;


That didn't work. So...I figured I could find out what the bad rec was.
Doing this:

Select * from attachments limit 5000 offset 0
Select * from attachments limit 5000 offset 5000
Select * from attachments limit 5000 offset 10000
Select * from attachments limit 5000 offset 15000

quickly showed me that record 16179 was causing this issue.


So, in order to resolve, this I did this:

pg_dump -s rt3 > /tmp/rt3schema

followed by:

for each in `psql rt3 -c "\d" | awk {'print $3'} | grep -vw attachments`;do
pg_dump rt3 -t $each >> /tmp/rt3data; done

(This second one removes only the table named attachments)

Then I used pg.py to do this:

import pg
olddb=pg.connect('rt3','myolddbserver')
new=pg.connect('rt3','localhost')
first=olddb.query("""select * from attachments limit 16169""").getresult()
last=olddb.query("""select * from attachments limit 100000 offset 16170""")
for eachRec in first + last:
new.query("""insert into attachments values %r""" % (eachRec,))

*** I did have to deal with some quotification issues, but you get the
point.


-----Original Message-----
From: pgsql-admin-owner@postgresql.org
[mailto:pgsql-admin-owner@postgresql.org] On Behalf Of BJ Taylor
Sent: Sep 24, 2008 1:32 PM
To: pgsql-admin@postgresql.org
Subject: [ADMIN] missing chunk number 0 for toast value

Our database seems to have been corrupted. It is a heavily used database,
and went several months without any type of vacuuming. When we finally
realized that it wasn't being vacuumed, we started the process, but the
process never successfully completed, and our database has never been the
same since.

The exact error that we receive now is as follows:


postgres@server:~> pg_dumpall -p 5433 > dbmail_dumpall_23092008.sql
pg_dump: SQL command failed
pg_dump: Error message from server: ERROR: missing chunk number 0 for toast
value 554339
pg_dump: The command was: COPY public.dbmail_messageblks (messageblk_idnr,
physmessage_id, blocksize, is_header, messageblk) TO stdout;
pg_dumpall: pg_dump failed on database "dbmail", exiting


We have tried using the -d option of the pg_dumpall so we could get a full
dump of the database, and just start over from that, but that fails as well.
We have also tried reindexing the table, but although the reindex didn't
fail, it didn't solve our problem either. Our next option is to do a full
vacuum, but we are reluctant to take our mail server down for that long,
especially when we do not know for sure that it will either succeed or fix
our problem. I have searched the forums, but was unable to find a solution
that we have not already tried. The solutions didn't appear to help others
who had this problem either.

Any suggestions?

Thanks,
BJ

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [pgsql-es-ayuda] Grupo local en españa: Parte II

Jordi Molina escribió:

> Tomo en cuenta el consejo, pero si al final hacemos lo que recomendó
> Álvaro estará todo integrado en la web de postgre.

Ojo, el nombre es PostgreSQL, se pronuncia "postgres q l" (no
"postgre s q l") y la abreviación usual es "Postgres". Por favor no
uses "Postgre".

--
Alvaro Herrera Valdivia, Chile Geotag: -39,815 -73,257
"I think my standards have lowered enough that now I think 'good design'
is when the page doesn't irritate the living f*ck out of me." (JWZ)
--
TIP 10: no uses HTML en tu pregunta, seguro que quien responda no podrá leerlo

Re: [GENERAL] Slony vs Longiste

On Wednesday 24 September 2008 12:34:17 Jason Long wrote:
> Richard Huxton wrote:
> > Jason Long wrote:
> >> I need to set up master vs slave replication.
> >>
> >> My use case is quite simple. I need to back up a small but fairly
> >> complex(30 MB data, 175 tables) DB remotely over T1 and be able to
> >> switch to that if the main server fails. The switch can even be a
> >> script run manually.
> >>
> >> Can someone either comment in as much detail as possible or point me to
> >> a comparison of Slony vs Longiste. Or some other option I have not
> >> heard of?
> >
> > Three questions you need to ask yourself.
> > 1. How heavily updated is the database?
> > 2. How often do you change the database's schema?
> > 3. Are there other databases in the installation?
> >
> > If #1 is "very heavy" then you'll want to do some testing with any
> > solution you use.
> >
> > If #2 is "a lot" then you'll want to consider WAL shipping as mentioned
> > below. Slony can handle schema changes, but you'll need to process them
> > through its own script. I'm afraid I can't comment on Londiste.
> >
> > If you just want a backup and the answer to #3 is no, look at WAL
> > shipping (see the various archive_xxx config settings in the manual and
> > google a bit).
> >
> >> From what I read Longiste is easy to set up while I got a quote for
> >> Slony setup for 5-10k.
> >
> > Unless your requirements are strange, that seems a little high, even
> > assuming USD as a currency. Of course, if you want support and
> > maintenance that will tend to make things mount.
>
> The database has 10-20 concurrent users so updates are not very heavy.
>
> The schema changes very frequently.
>
> There are not other databases in the installation.
>
> This quote included initial setup, failure testing, and scripts that
> were to automate setup and manage the installation. It did not include
> support and maintenance.

Are you planning on hiring someone to do it, or are you going to do it
yourself, because the prices of the solution is completely orthogonal to
which is the better fit technically.

In your case, since you do a lot of DDL changes, I'd go with londiste over
slony if I had to pick from those two. However, given the requirements you
laid out, PITR is probably your best option (this is what Richard alluded
too), and certainly the one I would recommend you try first.

--
Robert Treat
http://www.omniti.com/
Database: Scalability: Consulting

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] parallel pg_restore

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 24 sept. 08 à 18:56, Andrew Dunstan a écrit :
> The is purely a patch to pg_restore. No backend changes at all (and
> if I did it would not use anything that isn't in core anyway).

Ok, good.
I'm eager to see what -core hackers will want to do with Postgres-R
patches, but that shouldn't be a reason to distract them, sorry...

> Also, you ignored the point about clustered data. Maybe that doesn't
> matter to some people, but it does to others. This is designed to
> provide the same result as a single threaded pg_restore. Splitting
> data will break that.

I'm not sure I understand what you mean by "clustered data" here, in
fact...

> Having pg_dump do the split would mean you get it for free, pretty
> much. Rejecting that for a solution that could well be a bottleneck
> at restore time would require lots more than just a feeling. I don't
> see how it would give you any less reason to trust your backups.

Well, when pg_restore's COPY fail, the table is not loaded and you get
an ERROR, and if you're running with the -1 option, the restore stops
here and you get a nice ROLLBACK.
With this later option, even if pg_dump did split your tables, the
ROLLBACK still happens.

Now, what happens when only one part of the data cannot be restored
but you didn't pg_restore -1. I guess you're simply left with a
partially restored table. How will you know which part contains the
error? How will you replay the restoring of this part only?

It the answer is to play with the restore catalogue, ok, if that's not
it, I'm feeling the dumps are now less trustworthy with the split
option than they were before.

Of course all this remains hypothetical as your work is not including
such a feature, which as we see is yet to be designed.

> I still think the multiple data members of the archive approach
> would be best here. One that allowed you to tell pg_dump to split
> every nn rows, or every nn megabytes. Quite apart from any
> parallelism issues, that could help enormously when there is a data
> problem as happens from time to time, and can get quite annoying if
> it's in the middle of a humungous data load.

Agreed, but it depends a lot on the ways to control the part that
failed, IMVHO. And I think we'd prefer to have a version of COPY FROM
with the capability to continue loading on failure...

Regards,
- --
dim

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkjakLQACgkQlBXRlnbh1bm4jgCg0WenIOsaHwD9GDpI6C2mhVYB
pdwAoJYesvDYByQbSxqMjIEZOR9KiVXu
=AVy3
-----END PGP SIGNATURE-----

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Oracle and Postgresql

On Aug 31, 2008, at 8:44 PM, David Fetter wrote:
> What they want to have is a huge entity they can blame when everything
> goes wrong. They're not interested in the actual response times or
> even in the much more important time-to-fix because once they've
> blamed Oracle, they know the responsibility is no longer on their
> shoulders.

The usual individual developer, open source community, and small
company attitude is one that prefers to employ intelligent staff and
give them a lot of responsibility (with varying degrees of success).
They would rather spend a week refactoring code for performance, or
experimenting with another language or database, sending an employee
to training or conferences, contribute patches to open source
projects, etc. They will try to make a well thought-out decision up
front when possible, and often this results in an early choice for
PostgreSQL, especially because it fits within any budget. When these
people end up working in larger companies with different mindsets,
they sometimes are successful at getting PostgreSQL utilized through
resources like Command Prompt, EnterpriseDB, Greenplum, and so on, to
replace the vendor support that Oracle comes with. They might start
off with MySQL as a first database, but once learning about
PostgreSQL, will invest lots of time into porting if they understand
the advantages (this can be evidenced all the time by people
communication in the #postgresql IRC channel), and will put a lot of
effort into doing things the architecturally best way over time rather
than just slopping together bandaided bits and poor lazy table design.

Others are lazy, go with some popular name of something free they
hear, and end up as Red Hat/PHP/MySQL shops, with a huge pile of
random crappy free apps bandaided together - hoping to make some quick
cash. If they end up with PostgreSQL it's not a thought-out decision
(well or otherwise), and they use it very irresponsibly and then
everyone will blame "PostgreSQL" for all their problems simply because
it's not the most common name. There's a perception here that MySQL
is better for them because it's more popular, has more random free
crappy apps available for it, and they don't care much about the added
reliability of PostgreSQL (often they'll run with fsync=off), as
they're a rickety shop anyways. They'll also have a perception that
Oracle is some magical holy grail simply because it is so out of their
reach during early development, but as they become profitable enough,
the idea of buying Oracle becomes very exciting.

The corporate attitude is one that prefers things to be as
encompassing, bundled, automatic, and self-maintaining as possible.
They prefer Oracle because they provide a wide array of inbuilt
replication, backup, and administrative functionality - things like
raw device management (Oracle has implemented a couple different
filesystems as well), early integration with Java when it was all the
rage, tons of complicated shinies for permission control that managers
don't really comprehend but think they like and need and are using
correctly. These are typically Java shops, with no diversity at all -
you won't find a single perl or bash script lying around - they'll
load up a slow common Java framework for even the simplest task. Code
quality tends to be pretty decent, but there is heavy denial of any
problems and fixes are slow and obstructed by managerial complexity
and too much influence from Sales. :P

Another similar example can be found with ZFS or VxFS versus
traditional simple filesystems. ZFS has few tuning options and their
use is discouraged. It does a lot of things automatically and there
is a favoring of defaults over workload-specific tuning and
administrative overhead. It builds in every filesystem-related thing
(the filesystem itself, a logical volume manager, clustering tools,
and even it's own ZFS-specific filesystem cache daemon) into a single
manager-friendly bundle. You can't get the same levels of performance
out of it as you can by carefully tuning other systems, but that
tuning requires hiring intelligent staff and this seems to be
amazingly challenging for large corporations, and they'd rather pay
some middle-level manager a salary worth 5 developers, and have him
buy and assemble big packaged solutions instead.

PostgreSQL can't really take over the corporate market - Oracle and
DB2 will always be around too. :) To do that we'd need to do a lot of
unwanted things to the code, build in much more unecessary complexity
that most will never use, reduce flexibility and options in the
process, spange up incredible amounts of well-placed marketing dollars
and slowly get more acceptance by proving years of experience at an
increasing number of corporate PostgreSQL users.

I worked with PostgreSQL at Cingular - and the simple fact is that it
was not a good choice for the requirements of the task. The task as
defined could easily be called stupid and easy to change to do a
better way, but in that environment you can't stop a ball once it's
rolling and getting that ball or another one for change rolling takes
months and many meetings. Their minds were steeped in the ways of
Oracle, and Oracle has better optimizations for some stupid common
design/architectural/PHB mistakes. So then they try to bring their
broken methods to PostgreSQL verbatim, and it fails terribly. This is
not PostgreSQL's fault, but "PostgreSQL" gets the blame and label as a
substandard ("open source" in disdaining scoffing voice) database that
people only use because they can't afford Oracle, and even if the
effort is put in to do things right and it *is* faster, they still
won't like it and will feel that it's chosen out of resignation. So
really, it's better that we don't get the corporate market. This is
good, because it enables intelligent small businesses to have more of
a competitive edge and saner budget. Things like RAC and multimaster
replication are not good designs architecturally, but they are
encompassing ways to scale pretty much any existing design with
minimal effort. Corporations would much rather throw money/hardware
at a problem than refactor.

Right now where I work, our biggest performance problem could easily
be solved by moving a couple tables into an Oracle database. It
would be an easy port, and Oracle would handle the bad design much
better. Fixing it correctly is going to be a tedious process that
takes months to reach completion. As a small company, we plan to do
the latter, seek out quick fixes to buy us a little breathing room
until then, and avoid spending money (we'd much rather spend the
Oracle dollars on hardware improvements, etc.). A corporation will
usually throw money at Oracle and avoid spending developer time.

Currently PostgreSQL has a very good, strong, intelligent user
community. This has been the case consistently for all the years that
I've been using it, and it grows steadily over time without
significant changes in the overall community vibe. That said, when we
ported to Windows, there was an influx of a lot of former MySQL and
Microsoft SQL folks with very little general database knowledge
compared to the average PostgreSQL user. Fortunately this has not
been overpowering, and has lead to more people learning and doing
things more intelligently. But if we somehow magically took over the
majority of the corporate market, we'd have a lot more people involved
who frankly don't care to learn and just bicker and blame, which would
deplete the community resources for everyone else.

A knowledgeable PostgreSQL DBA can make significantly more than an
Oracle DBA as they're a scarcer resource and generally higher quality
on average. But it may be harder for them to find work - they may end
up having to move, telecommute, or commute a longer distance simply
because there are less PostgreSQL shops. It also means a much higher
probability of working for a small-medium company versus a corporation.

An Oracle DBA can be a lot lazier, and lean on the vendor a lot more.
There are open Oracle DBA positions everywhere, and it is very easy
for them to find another job, so learning a lot and focusing on doing
a good job are not important to them. In the corporate environment in
which most of these jobs are, they are responsible for far less in
their job role, whereas the PostgreSQL DBA tends to end up responsible
for a lot more pieces of the puzzle.

Cheers,
--
Casey Allen Shobe
Database Architect, The Berkeley Electronic Press

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] Proposal of SE-PostgreSQL patches (for CommitFest:Sep)

Josh Berkus wrote:
> Peter,
>
> > Yeah, but do we even have the slightest bit of information about what
> > exactly would be required to achieve the required levels? And whether
> > this patch does it? And whether there would be alternative, more
> > desirable ways to achieve a similar level?
>
> Actually, I have some direct communication that SEPostgres will lead to
> PostgreSQL being widely adopted in at least one US government agency.
> Can't say more, of course. ;-)

We can't make decisions based on just one adoption agency, of course.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal of SE-PostgreSQL patches (for CommitFest:Sep)

On Sep 24, 2008, at 2:38 PM, Josh Berkus wrote:

> Peter,
>
>> Yeah, but do we even have the slightest bit of information about
>> what exactly would be required to achieve the required levels? And
>> whether this patch does it? And whether there would be
>> alternative, more desirable ways to achieve a similar level?
>
> Actually, I have some direct communication that SEPostgres will lead
> to PostgreSQL being widely adopted in at least one US government
> agency. Can't say more, of course. ;-)

And the consideration is predicated on the PostgreSQL community
accepting the patch? This sounds like an opportunity for one of the
numerous "enterprise" spin-offs.

Cheers,
M

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pgsql: Make LC_COLLATE and LC_CTYPE database-level settings.

heikki@postgresql.org (Heikki Linnakangas) writes:
> Log Message:
> -----------
> Make LC_COLLATE and LC_CTYPE database-level settings. Collation and
> ctype are now more like encoding, stored in new datcollate and datctype
> columns in pg_database.
>
> This is a stripped-down version of Radek Strnad's patch, with further
> changes by me.
> pgsql/src/bin/pg_dump:
> pg_dump.c (r1.500 -> r1.501)
> (http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/bin/pg_dump/pg_dump.c?r1=1.500&r2=1.501)
> pg_dumpall.c (r1.106 -> r1.107)
> (http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/bin/pg_dump/pg_dumpall.c?r1=1.106&r2=1.107)

Let me report a problem with this...

Running pg_dump encounters the following:

chris@dba2:Slony-I/CMD/slony1-HEAD/tests> pg_dump -s slonyregress1
pg_dump: column number -1 is out of range 0..7
pg_dump: column number -1 is out of range 0..7
zsh: segmentation fault pg_dump -s slonyregress1

It's not a Slony-I-related issue; I get identical behaviour with other
databases that do not have Slony-I in place.

Things "seem to work" if I roll back this patch to pg_dump.c.

So something's up...
--
(reverse (concatenate 'string "moc.enworbbc" "@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/emacs.html
"Being really good at C++ is like being really good at using rocks to
sharpen sticks." -- Thant Tessman

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal of SE-PostgreSQL patches (for CommitFest:Sep)

On Wed, 24 Sep 2008 11:38:36 -0700
Josh Berkus <josh@agliodbs.com> wrote:

> Peter,
>
> > Yeah, but do we even have the slightest bit of information about
> > what exactly would be required to achieve the required levels? And
> > whether this patch does it? And whether there would be
> > alternative, more desirable ways to achieve a similar level?
>
> Actually, I have some direct communication that SEPostgres will lead
> to PostgreSQL being widely adopted in at least one US government
> agency. Can't say more, of course. ;-)

/me notes his tin foil hat argument.

Joshua D. Drake

>
> --Josh
>


--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal of SE-PostgreSQL patches (for CommitFest:Sep)

Peter,

> Yeah, but do we even have the slightest bit of information about what
> exactly would be required to achieve the required levels? And whether
> this patch does it? And whether there would be alternative, more
> desirable ways to achieve a similar level?

Actually, I have some direct communication that SEPostgres will lead to
PostgreSQL being widely adopted in at least one US government agency.
Can't say more, of course. ;-)

--Josh

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[ADMIN] missing chunk number 0 for toast value

Our database seems to have been corrupted.  It is a heavily used database, and went several months without any type of vacuuming.  When we finally realized that it wasn't being vacuumed, we started the process, but the process never successfully completed, and our database has never been the same since. 

The exact error that we receive now is as follows:

postgres@server:~> pg_dumpall -p 5433 > dbmail_dumpall_23092008.sql
pg_dump: SQL command failed
pg_dump: Error message from server: ERROR: missing chunk number 0 for toast value 554339
pg_dump: The command was: COPY public.dbmail_messageblks (messageblk_idnr, physmessage_id, blocksize, is_header, messageblk) TO stdout;
pg_dumpall: pg_dump failed on database "dbmail", exiting


We have tried using the -d option of the pg_dumpall so we could get a full dump of the database, and just start over from that, but that fails as well.  We have also tried reindexing the table, but although the reindex didn't fail, it didn't solve our problem either.  Our next option is to do a full vacuum, but we are reluctant to take our mail server down for that long, especially when we do not know for sure that it will either succeed or fix our problem.  I have searched the forums, but was unable to find a solution that we have not already tried.  The solutions didn't appear to help others who had this problem either. 

Any suggestions?

Thanks,
BJ

Re: [JDBC] COPY support in JDBC driver?

On Wed, 24 Sep 2008, Daniel Migowski wrote:

> AFAIK is UTF-8 the only encoding which the driver supports, anyway. And
> the native Java encoding, too. In my opinion the API should either
> support Writers and Readers (instead of Output- and InputStream), so the
> application has to take care for the encoding itself, or the API should
> encapsulate setting an arbitrary encoding on the server side before the
> copy command, and return to the default encoding directly afterwards.
>

Yes, the current copy patches only support *Stream which does leave the
user exposed to encoding issues. Providing a Reader/Writer API doesn't
support COPY ... BINARY, but I don't know how many people would actually
use such a thing. Parallel interfaces are a possibility, but I'd guess
people would end up using the Stream versions for non-binary data anyway.

Does anyone have the need to do COPY BINARY?

I also wonder what the encoding conversion hit is if no conversion needs
to be done. Perhaps we should measure that before abandonding the Stream
API?

Kris Jurka

--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

Re: [ADMIN] Postgres client Configuration

>>> Napolean Periathambi <Napolean.Periathambi@VSOFTCORP.COM> wrote:

> Is there any ways to include wildcard in pg_hba.conf file to accept
all
> postgres clients on the network rather than specifying specific IP
addresses
> on this file?

# CIDR-ADDRESS specifies the set of hosts the record matches.
# It is made up of an IP address and a CIDR mask that is an integer
# (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that specifies
# the number of significant bits in the mask. Alternatively, you can
write
# an IP address and netmask in separate columns to specify the set of
hosts.

In other words, it says how many bits of the IP address need to match.

For example:

# TYPE DATABASE USER CIDR-ADDRESS METHOD

# "local" is for Unix domain socket connections only
local all postgres ident sameuser
local cir all md5

# IPv4 local connections:
host cir all 127.0.0.1/32 md5
# IPv6 local connections:
host cir all ::1/128 md5

# Programmer/analyst network segment
host cir all 165.219.88.0/22 md5
# STEP network segment
host cir all 165.219.80.0/24 md5
# TECH network segment
host cir all 165.219.95.0/24 md5

-Kevin

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [JDBC] COPY support in JDBC driver?

On Wed, 24 Sep 2008, Michael Nacos wrote:

> if you want COPY support from the JDBC driver just for performance reasons,
> you should probably also consider using the driver's batch mode.

Copy is quite a bit faster than batch execution. See:

http://archives.postgresql.org/pgsql-jdbc/2008-02/msg00018.php

> Encoding is a serious consideration. While testing pgBee, I ran into
> problems inserting codepage 1252 files into a SQL_ASCII database,
> because of character mismatches.

You should not use a SQL_ASCII database if you care about encoding your
data properly. Once you put data in, there is no way of telling what
encoding it was inserted with, so there is no way of retrieving it
correctly.

Kris Jurka


--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

Re: [ADMIN] Missing pg_clog files

To use the hex value 0x55, do I need to enclose it in single quotes?

Carol

On Sep 24, 2008, at 11:44 AM, Tom Lane wrote:

> Carol Walter <walterc@indiana.edu> writes:
>> Are the files that contain the hex characters supposed to contain a
>> single string and no control characters?
>
> Yes, you want 256K occurrences of the byte value 0x55 and nothing
> else.
>
>> I'm also wondering if, after I create the dummy files, and pg_dump
>> works, I could restore an old pg_dumpall file and then insert any
>> data that aren't there from the pg_dumps.
>
> It'd be a good idea to do as much cross-checking as you can, since
> it's highly probable that the dumped data will be at least partly
> wrong.
>
> No, I'm afraid updating to 8.2.latest won't get you out of this.
> It might possibly prevent a recurrence.
>
> regards, tom lane
>
> --
> Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [BUGS] Postgres won't start and doesn't generate any messages.

"Chris Barnes" <cbarnes@recognia.com> writes:
> [postgres@pgprd01 ~]$ /opt/PostgreSQL/8.3/bin/pg_ctl -w start -D
> \"/data/pgsql/data/\" -l \"/data/pgsql/data//pg_log/startup.log\"

Those backslashes don't look like a good idea: the quotes are ending
up as part of the file names.

regards, tom lane

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Re: [HACKERS] FSM, now without WAL-logging

Zdenek Kotala wrote:
> Heikki Linnakangas napsal(a):
>
>> It would be
>> simple to update the FSM at every heap insert and update record, but
>> that then might be an unacceptable amount of overhead at recovery. Also,
>> the index FSM is not yet updated during recovery.
>
> I expect locking problems specially on small tables where FSM has only one level
> instead slower recovery. Maybe background writer could update FSM info, but it
> breaks modularity and It could bring problems during checkpoints.

Recovery is single-threaded (for the lack of a better word) anyway, so
there can't be other backends competing for the locks. Background writer
is one option.

One option is to update the FSM if there's less than X% of free space on
the page left after the insert/update. That would be similar to the rule
we use during normal operation, which is to update the FSM whenever the
target page fills up and we have to get a new page from the FSM.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] FSM, now without WAL-logging

Zdenek Kotala wrote:
> Heikki Linnakangas napsal(a):
>> There's one known bug left. If we crash after truncating a relation, and
>> the truncation of the FSM doesn't hit the disk but the truncation of the
>> main fork does, we can end up with an FSM that shows that there's free
>> space on pages that don't exist anymore. The straightforward way to fix
>> that is to put back the WAL-logging of FSM truncations, but given that I
>> just ripped off all the rest of the WAL-logging, does anyone have other
>> ideas?
>
> What's about truncate FSM during WAL replay of main fork truncation record?

That would work, except that there's no WAL record for truncating the
main fork.

I considered adding that, WAL-logging the truncation of the main fork,
in vacuum.c. But it would have to be done for all indexams that use the
FSM as well.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal of SE-PostgreSQL patches (for CommitFest:Sep)

Joshua Drake wrote:
> I know of no one that really uses SELinux because it is a nightmare. On
> the other hand, this type of security is required to get into certain
> scary tin foil hat producing institutions.

Yeah, but do we even have the slightest bit of information about what
exactly would be required to achieve the required levels? And whether
this patch does it? And whether there would be alternative, more
desirable ways to achieve a similar level?

I am not arguing for or against this patch now, but I would like to know
whether someone has an agenda for it. Without an agenda, future
maintenance will be difficult. Reference to standards or other public
documents would work best to define that agenda.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal of SE-PostgreSQL patches (for CommitFest:Sep)

Bruce Momjian wrote:
> Peter, I am confused how the above statement relates to a posting you
> made a week ago:
>
> http://archives.postgresql.org/pgsql-hackers/2008-09/msg01067.php
>
> Now these items are arguably useful and welcome features in their own
> right. Unfortunately, this patch has chosen to provide these features in
> a way that makes them accessible to the least amount of users. And
> moreover, it bunches them all in one feature, while they should really
> be available independently.

I just want to distinguish the causalities in the various arguments that
are being made. There are many ways to approach this, two of which
could be:

We want MAC => SELinux is the only proven way to implement MAC => let's
take the patch

or

SELinux is way too complex => We don't take the patch => Figure out the
MAC issue some other way


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Debian packages for Postgres 8.2

Joris Dobbelsteen wrote:
> The good question would be for what reason they have removed the
> backports package? Maybe shortage on maintainers?

As a matter of policy, backports are made from Debian testing.
Continued maintenance of PG 8.2 packages is not really backporting,
since there is nothing to backport from.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [pgadmin-support] Problen al conectar to server

On 24/09/2008 16:52, rene wrote:

> Error connecting to the server: FATAL: la autentificación «password»
> falló para el usuario «rene»

Well, the problem is just what the message says - you supplied an
incorrect user/password combination when trying to connect.

Ray.

------------------------------------------------------------------
Raymond O'Donnell, Director of Music, Galway Cathedral, Ireland
rod@iol.ie
Galway Cathedral Recitals: http://www.galwaycathedral.org/recitals
------------------------------------------------------------------

--
Sent via pgadmin-support mailing list (pgadmin-support@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-support

Re: [HACKERS] parallel pg_restore

Dimitri Fontaine wrote:
> Le mercredi 24 septembre 2008, Andrew Dunstan a écrit :
>
>> No. The proposal will perform exactly the same set of steps as
>> single-threaded pg_restore, but in parallel. The individual steps won't
>> be broken up.
>>
>
> Ok, good for a solid trustworthy parallelism restore. Which is exactly what we
> want. Just out of curiosity, do you plan to use Postgres-R helper backends
> infrastructure?
>

The is purely a patch to pg_restore. No backend changes at all (and if I
did it would not use anything that isn't in core anyway).
>
>> Quite apart from anything else, parallel data loading of individual
>> tables will defeat clustering, as well as making it impossible to avoid
>> WAL logging of the load (which I have made provision for).
>>
>
> Depends whether the different workers are able to work from the same
> transaction or not, I imagine. Some work has been done to allow multiple
> backends to be working in the exact same transaction (Simon's snapclone and
> Postgres-R helper backend infrastructure), so one of them could TRUNCATE the
> table and give a go signal to workers to fill the table. In the same
> transaction.
> Ok, I need to wake up now... :)
>
>

Again, I am not doing anything on the backend. I am following Tom's
original suggestion of simply having pg_restore run steps in parallel,
with no backend changes.

Also, you ignored the point about clustered data. Maybe that doesn't
matter to some people, but it does to others. This is designed to
provide the same result as a single threaded pg_restore. Splitting data
will break that.

>> The fact that custom archives are compressed by default would in fact
>> make parallel loading of individual tables' data difficult with the
>> present format. We'd have to do something like expanding it on the
>> client (which might not even have enough disk space) and then split it
>> before loading it to the server. That's pretty yucky. Alternatively,
>> each loader thread would need to start decompressing the data from the
>> start and thow away data until it got to the point it wanted to start
>> restoring from. Also pretty yucky.
>>
>
> Another alternative is the round-robin reader implemented in pgloader, where
> all the archive reading is done by a single worker, which then split what it
> read to any number of coworkers, filling next queue(s) while previous one(s)
> are busy COPYing to the server.
>
>
>> Far better would be to provide for multiple data members in the archive
>> and teach pg_dump to split large tables as it writes the archive. Then
>> pg_restore would need comparatively little adjustment.
>>
>
> Well, that's another possibility, but I tend to prefer having the parallelism
> mecanics into the restore side of things. It may be only an illusion, but
> this way I have far more trust into my backups.
>

Having pg_dump do the split would mean you get it for free, pretty much.
Rejecting that for a solution that could well be a bottleneck at restore
time would require lots more than just a feeling. I don't see how it
would give you any less reason to trust your backups.
>
>> Also, of course, you can split tables yourself by partitioning them.
>> That would buy you parallel data load with what I am doing now, with no
>> extra work.
>>
>
> And that's excellent :)
>
>
>> In any case, data loading is very far from being the only problem. One
>> of my clients has long running restores where the data load takes about
>> 20% or so of the time - the rest is in index creation and the like. No
>> amount of table splitting will make a huge difference to them, but
>> parallel processing will.
>>
>
> Oh yes, I'm running into this too (not on the same level but still).
> Parallel seqscan should help creating indexes in parallel without having the
> disks going crazy for read - write - read - write etc sequences, and posix
> advices should help a lot here too.
> Does the dependancy tracker in pg_restore allows to consider FK creation are
> dependant on matching PK being already there?
>

I believe so. If not, that's a bug and we should fix it IMNSHO.

>
>> As against that, if your problem is in loading
>> one huge table, this won't help you much. However, this is not a pattern
>> I see much - most of my clients seem to have several large tables plus a
>> boatload of indexes. They will benefit a lot.
>>
>
> The use case given by Greg Smith at the time was loading a multi terabyte
> table on a raid array with a lot of spindles. It then become impossible for a
> single CPU to take full profit of the available write bandwith. No idea how
> common this situation is in the field, though.
>
>
>

I still think the multiple data members of the archive approach would be
best here. One that allowed you to tell pg_dump to split every nn rows,
or every nn megabytes. Quite apart from any parallelism issues, that
could help enormously when there is a data problem as happens from time
to time, and can get quite annoying if it's in the middle of a humungous
data load.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[COMMITTERS] pgsql: Fix more problems with rewriter failing to set Query.hasSubLinks

Log Message:
-----------
Fix more problems with rewriter failing to set Query.hasSubLinks when inserting
a SubLink expression into a rule query. We missed cases where the original
query contained a sub-SELECT in a function in FROM, a multi-row VALUES list,
or a RETURNING list. Per bug #4434 from Dean Rasheed and subsequent
investigation.

Back-patch to 8.1; older releases don't have the issue because they didn't
try to be smart about setting hasSubLinks only when needed.

Tags:
----
REL8_1_STABLE

Modified Files:
--------------
pgsql/src/backend/rewrite:
rewriteHandler.c (r1.158.2.3 -> r1.158.2.4)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/rewrite/rewriteHandler.c?r1=1.158.2.3&r2=1.158.2.4)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix more problems with rewriter failing to set Query.hasSubLinks

Log Message:
-----------
Fix more problems with rewriter failing to set Query.hasSubLinks when inserting
a SubLink expression into a rule query. We missed cases where the original
query contained a sub-SELECT in a function in FROM, a multi-row VALUES list,
or a RETURNING list. Per bug #4434 from Dean Rasheed and subsequent
investigation.

Back-patch to 8.1; older releases don't have the issue because they didn't
try to be smart about setting hasSubLinks only when needed.

Tags:
----
REL8_2_STABLE

Modified Files:
--------------
pgsql/src/backend/rewrite:
rewriteHandler.c (r1.168.2.1 -> r1.168.2.2)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/rewrite/rewriteHandler.c?r1=1.168.2.1&r2=1.168.2.2)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix more problems with rewriter failing to set Query.hasSubLinks

Log Message:
-----------
Fix more problems with rewriter failing to set Query.hasSubLinks when inserting
a SubLink expression into a rule query. We missed cases where the original
query contained a sub-SELECT in a function in FROM, a multi-row VALUES list,
or a RETURNING list. Per bug #4434 from Dean Rasheed and subsequent
investigation.

Back-patch to 8.1; older releases don't have the issue because they didn't
try to be smart about setting hasSubLinks only when needed.

Modified Files:
--------------
pgsql/src/backend/rewrite:
rewriteHandler.c (r1.179 -> r1.180)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/rewrite/rewriteHandler.c?r1=1.179&r2=1.180)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix more problems with rewriter failing to set Query.hasSubLinks

Log Message:
-----------
Fix more problems with rewriter failing to set Query.hasSubLinks when inserting
a SubLink expression into a rule query. We missed cases where the original
query contained a sub-SELECT in a function in FROM, a multi-row VALUES list,
or a RETURNING list. Per bug #4434 from Dean Rasheed and subsequent
investigation.

Back-patch to 8.1; older releases don't have the issue because they didn't
try to be smart about setting hasSubLinks only when needed.

Tags:
----
REL8_3_STABLE

Modified Files:
--------------
pgsql/src/backend/rewrite:
rewriteHandler.c (r1.177 -> r1.177.2.1)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/rewrite/rewriteHandler.c?r1=1.177&r2=1.177.2.1)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [HACKERS] Hot Standby Design

On Wed, 2008-09-24 at 12:35 -0400, Robert Treat wrote:
> On Wednesday 24 September 2008 03:27:44 Simon Riggs wrote:
> > On Wed, 2008-09-24 at 00:30 -0400, Robert Treat wrote:
> > > * "However, some WAL redo actions will be for DDL actions. These DDL
> > > actions are repeating actions that have already committed on the primary
> > > node, so they must not fail on the standby node. These DDL locks take
> > > priority and will automatically cancel any read-only transactions that
> > > get in their way."
> > >
> > > Some people will want the option to stack-up ddl transactions behind
> > > long-running queries (one of the main use cases of a hot slave is
> > > reporting stye and other long running queries)
> >
> > Scheduling tools can help here. Run set of queries on Standby, then when
> > complete run DDL on Primary and have its changes filter through.
> >
>
> true... i am just reminded of Oracle 8's log replay, where you had to stop
> replay to run any queries... given that was a usefull feature, I suspect
> we'll hear complaints about it not going that way. I think one could argue
> that we might be able to provide such an option in the future, if not in the
> first release.

It probably sounds worse than it is. If you drop a table on the Primary,
then somebody running a query against it on the Standby is in for a
shock. But on the other hand, why are you dropping a table that people
still consider worth reading? DROP TABLE should be carefully researched
before use, so I don't think its a big problem.

We could make it wait for a while before cancelling, as an option, if
you think its important?

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Debian packages for Postgres 8.2

Markus Wanner wrote:
> Hi,
>
> I'm running several productive servers on Debian etch (stable) with
> Postgres 8.2 which has been in lenny (testing) and made available for
> etch through the backports project [1]. Unfortunately, they
> discontinued maintaining 8.2 and switched to 8.3 in testing and thus
> also for the backports.
>
> As I don't currently want to switch to 8.3 due to the involved
> downtime and upgrading troubles involved. So I've compiled up to date
> Debian packages for Postgres 8.2.10. You can get them (maybe just
> temporarily) from here: http://www.bluegap.ch/debian, I'm providing
> packages as etch-backports for amd64 and i386. Upgrading from earlier
> 8.2 backports should work just fine.
>
> I'm trying to convince the backports people to re-add Postgres 8.2. As
> soon as that happens my own repository will probably disappear again.
>
> Please drop me a note if you are interested in 8.2 for etch. (Postgres
> 8.3 should become available via the backports within a few days, I
> guess).
I still have interest and I'm actually actively using it. Its a shame,
as the postgresql community still support 8.2 and probably more rely on it.
Besides this, Debian's tools are well polished and support many versions
side-by-side.

The good question would be for what reason they have removed the
backports package? Maybe shortage on maintainers?

- Joris

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] PDF Documentation for 8.3?

At 4:12am -0400 on Wed, 24 Sep 2008, Michelle Konzack wrote:
>> http://www.postgresql.org/files/documentation/pdf/8.3/postgresql-8.3-A4.pdf
>
> I was not able to download ANY PDFs
>
> I am sitting here @home behind my TP570 and I am connected over GSM
> (Bouygues Telecom) to the Internet and if I klick the PDF links, the
> conection timed out.

Hmm, it works splendidly here (Chapel Hill, NC). I wonder if it's too
large for your particular network setup? It's more than 16 MB. Do you
have another network connection you could try?

You also might try getting it in chunks. I've found the 'wget -c <url>'
(or curl --continue) paradigm invaluable for downloading large files
over slow, inconsistent, or otherwise flaky networks.

Kevin

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] Hot Standby Design

On Wednesday 24 September 2008 03:27:44 Simon Riggs wrote:
> On Wed, 2008-09-24 at 00:30 -0400, Robert Treat wrote:
> > here are some scattered thoughts i had while reading it :
>
> Thanks for your comments.
>
> It is very detailed, so further feedback is going to be very beneficial
> in making this all work in the way we hope and expect.
>
> > * "However, some WAL redo actions will be for DDL actions. These DDL
> > actions are repeating actions that have already committed on the primary
> > node, so they must not fail on the standby node. These DDL locks take
> > priority and will automatically cancel any read-only transactions that
> > get in their way."
> >
> > Some people will want the option to stack-up ddl transactions behind
> > long-running queries (one of the main use cases of a hot slave is
> > reporting stye and other long running queries)
>
> Scheduling tools can help here. Run set of queries on Standby, then when
> complete run DDL on Primary and have its changes filter through.
>

true... i am just reminded of Oracle 8's log replay, where you had to stop
replay to run any queries... given that was a usefull feature, I suspect
we'll hear complaints about it not going that way. I think one could argue
that we might be able to provide such an option in the future, if not in the
first release.

> > * Actions not allowed on Standby are:
> > DML - Insert, Update, Delete, COPY FROM, Truncate
> >
> > copy from suprised me a bit, since it is something i could see people
> > wanting to do... did you mean COPY TO in this case?
>
> I checked...
>
> COPY TO is allowed, since it extracts data. So pg_dump will run.
> COPY FROM will not be allowed since it loads data. So pg_restore will
> not run.
>

ah, of course... the late hour had me thinking backwards. pg_dump on the
clone will be a big plus.

<snip>
> > * Looking for suggestions about what monitoring capability will be
> > required.
> >
> > one place to start might be to think about which checks in check_nagios
> > might still apply. Possibly also looking to systems like slony for some
> > guidence... for example
>
> check_pgsql will work, but its very simple.
>
> check_postgres will also work, though many some actions could give
> different or confusing results. e.g. last vacuum time will not be
> maintained for example, since no vacuum occurs on the standby.
>

yep.

> slony won't run on the standby either, so those checks won't work
> either.
>
> > everyone will want some way to check how far the lag
> > is on a stnadby machine.
>
> Agreed
>

right... my thought with slony was, what do people monitor on thier slony
slaves? (there is actually a nagios script for that iirc)

<snip>
> The Standby is a Clone and not a Slave. A Slave is a separate database
> that is forced to duplicate the actions of the Master. The Standby is an
> exact copy, in every detail that matters.
>

This is an interesting clarification.

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Slony vs Longiste

Richard Huxton wrote:
Jason Long wrote:   
I need to set up master vs slave replication.  My use case is quite simple.  I need to back up a small but fairly complex(30 MB data, 175 tables) DB remotely over T1 and be able to switch to that if the main server fails.  The switch can even be a script run manually.  Can someone either comment in as much detail as possible or point me to a comparison of Slony vs Longiste.  Or some other option I have not heard of?     
 Three questions you need to ask yourself. 1. How heavily updated is the database? 2. How often do you change the database's schema? 3. Are there other databases in the installation?  If #1 is "very heavy" then you'll want to do some testing with any solution you use.  If #2 is "a lot" then you'll want to consider WAL shipping as mentioned below. Slony can handle schema changes, but you'll need to process them through its own script. I'm afraid I can't comment on Londiste.  If you just want a backup and the answer to #3 is no, look at WAL shipping (see the various archive_xxx config settings in the manual and google a bit).    
From what I read Longiste is easy to set up while I got a quote for Slony setup for 5-10k.     
 Unless your requirements are strange, that seems a little high, even assuming USD as a currency. Of course, if you want support and maintenance that will tend to make things mount.    
The database has 10-20 concurrent users so updates are not very heavy.

The schema changes very frequently.

There are not other databases in the installation.

This quote included initial setup, failure testing, and scripts that were to automate setup and manage the installation.  It did not include support and maintenance.

[ADMIN] Postgres client Configuration

Is there any ways to include wildcard in pg_hba.conf file to accept all postgres clients on the network rather than specifying specific IP addresses on this file?

Any assistance is appreciated

Thanks
Napolean

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin