Sunday, July 27, 2008

Re: [PERFORM] how does pg handle concurrent queries and same queries

> I have taken over the maintenance of a server farm , recently. 2 webserver
> on db server. They are quite powerful 2 processor xeon w/ 6Gig of ram .
>
> Couple of days ago we had a serious performance hit and the db server (pg.
> v7.4) was overloaded w/ something in a way that operating system was almost
> not able to respond or in cases it did not.
>
> After some analysis i suspect that there is a query that takes up to 1
> second and that is the cause. Upon each page loading this query fires and
> takes the one second and blocks the page to load completly . The load was
> roughly ~300 connections in one minute .
>
> So here are my questions :
>
> · Why does the second and the later queries take the whole on second
> if the dataset is the same . Shouldn't PG realise that the query is the same
> so i give the user the same resultset ?
>
> · How do I know if one query blocks the other ?
>
> · Is there a way to log the long running queries in 7.4 ? If not is
> it available in any newer version ?

Can you post the queries? Can you provide an 'analyze explain'? Do you
perform a 'vacuum analyze' on a regular basis?

--
regards
Claus

When lenity and cruelty play for a kingdom,
the gentlest gamester is the soonest winner.

Shakespeare

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: [HACKERS] WITH RECUSIVE patches 0723

Hello

I played with CTE and I have to say, it's great feature - great work.

One questions - can I enforce materialisation of query?

It would be usefull for some analytical queries like:

with tmp as (select a, sum(b) as b from test) select * from tmp union
all select 'all', sum(b) from tmp;

regards
Pavel Stehule

2008/7/27 Tatsuo Ishii <ishii@postgresql.org>:
>> At David's request I've been looking through this patch.
>>
>> Regarding documentation: if it would help, I can write some; I have
>> already made a start on writing down what is going on internally in
>> order to understand it myself.
>
> Thanks. There was some docs written in Japanese by Yoshiyuki. Recently
> he updagted it. I will translate into English and post here.
>
>> I've found three more bugs so far:
>>
>> 1)
>>
>> create view v2(id) as values (1);
>> with recursive t(id) as (select id from v2
>> union all select id+1 from t where id < 5)
>> select * from t;
>> ERROR: could not open relation 1663/16384/24588: No such file or directory
>>
>> Here it seems that rewriting is simply not being applied to CTEs where
>> a recursive clause is present; the reference to "v2" remains in the
>> query up until execution time, at which point it errors out (in
>> ExecInitSeqScan called from InitPlan).
>
> Yes, we need to make the rewrite system to understand CTEs. Probably
> fireRIRrules() needs to have lines something like:
>
> if (rte->rtekind == RTE_RECURSIVE)
> {
> rte->non_recursive_query = fireRIRrules(rte->non_recursive_query, activeRIRs);
> continue;
> }
>
> But I still see the error message. Will look into more.
>
> For below, I will ask Yoshiyuki.
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
>
>> 2)
>>
>> with recursive t(id) as (values (1)
>> union all select id+1 from t where id < 5
>> union all values (2))
>> select * from t;
>> ERROR: table "t" has 0 columns available but 1 columns specified
>>
>> This seems to be caused by incorrect assumptions in checkWellFormedCte
>> and checkCteSelectStmt (which should have been rejecting the query).
>> The query tree as seen by checkWellFormedCte here is (values(1) union
>> all select ...) union all (values (2)), and when the left subtree is
>> passed to checkCteSelectStmt, it believes it to be non-recursive due
>> to the lack of any From clause. The unexpected error is produced
>> later.
>>
>> 3)
>>
>> with recursive t(id)
>> as (values (1)
>> union all select t.id+1
>> from t left join (values (1)) as s(x) on (false)
>> where t.id < 5)
>> select * from t;
>> id
>> ----
>> 1
>> 2
>> (2 rows)
>>
>> This behaviour is clearly intentional, since the entire mechanism of
>> estate->es_disallow_tuplestore exists for no other reason, but it
>> seems to me to be clearly wrong. What is the justification for it?
>>
>> --
>> Andrew (irc:RhodiumToad)
>>
>> --
>> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-hackers
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] copy ... from stdin csv; and bytea

Klint Gore <kgore4@une.edu.au> writes:
> David Wilson wrote:
>> I'm not certain how to check the actual byte width of a column within a
>> row,

> select length(bytea_field) from table

If you want the actual on-disk footprint, use pg_column_size()

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [SQL] PERSISTANT PREPARE (another point of view)

Hello

2008/7/28 Milan Oparnica <milan.opa@gmail.com>:
> Pavel Stehule wrote:
>>
>> Hello
>>
>> this is near my implemenation of stored procedures - it's not far from
>> your view on prepared statements. There result of any unbound select
>> is directly forwarded to client - there is little bit different
>> syntax, but idea is same
>>
>>
>> create procedure foo(params)
>> as $$
>> begin
>> select a, b, c from tab1 -- unbound select
>> select ....
>> end;
>>
>> and you can call with CALL statement. Curent implementation of SRF
>> function in plpgsql sn't eficient for big results - it needs
>> materialised table in memory. Pipeline Oracle's functions are better,
>> but we are limited one thread architecture.
>>
>> regards
>> Pavel Stehule
>>
>
> Yeah, this would be pretty the same.
>
> I just didn't understand if this is already implemented in PG ?
>

no - I have workin prototype now - and I am searching sponsors for
finishing this work.

http://okbob.blogspot.com/2007/11/stacked-recordset-multirecordset.html
http://okbob.blogspot.com/2007/11/first-real-procedures-on-postgresql.html

But it's really similar to what do you want and it's can be
interesting together with plpgpsm for you, because plpgpsm (SQL/PSM)
allows one statement procedures, like:

create procedure foo(...) as $$
select * from tab;
$$ language plpgpsm;

so this is standard syntax for persistent prepared statement :)

regards
Pavel Stehule

> I've tried to create a PROCEDURE in QueryTool of pgAdminIII and I get
> following result:
>
> ERROR: syntax error at or near "PROCEDURE"
> LINE 1: CREATE PROCEDURE aatestubnd(refcursor, insklid integer, inda...
> ^
>
> ********** Error **********
>
> ERROR: syntax error at or near "PROCEDURE"
> SQL state: 42601
> Character: 8
>
>
> If I try with CREATE FUNCTION I get this result (as expected):
>
> ERROR: function result type must be specified
>
> ********** Error **********
>
> ERROR: function result type must be specified
> SQL state: 42P13
>
>
> Is this because of QueryTool's limitations or is this feature not yet
> implemented in Postgre ?
>
> Though, CREATE PROCEDURE should be a valid SQL 92 standard procedure.
>
> Best regards,
>
> Milan Oparnica
>
> --
> Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-sql
>

--
Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-sql

[PERFORM] how does pg handle concurrent queries and same queries

Hi All,

 

I have taken over the maintenance of a server farm , recently. 2 webserver on  db server. They are quite powerful 2 processor xeon w/ 6Gig of ram .

 

Couple of days ago we had a serious performance hit and the db server (pg. v7.4) was overloaded w/ something in a way that operating system was almost not able to respond or in cases it did not.

 

 

After some analysis i suspect that there is a query that takes up to 1 second and that is the cause. Upon  each page loading this query fires and takes the one second and blocks the page to load completly . The load was roughly ~300 connections in one minute .

 

So here are my questions :

 

·         Why does the second and the later queries take the whole on second if the dataset is the same . Shouldn’t PG realise that the query is the same so i give the user the same resultset ?

·         How do I know if one query blocks the other ?

·         Is there a way to log the long running queries in 7.4 ? If not is it available in any newer version ?

 

thanks for your help !

 

Üdvözlettel/kind regards,

Faludi, Gábor

FITS Magyarország Kft.

http://www.FITS.hu

Tel.:+36 30 4945862

Email: falu@fits.hu

Ingyenes videó tanfolyamok(Excel,Access,Word) : http://www.fits.hu/trainings

 

[BUGS] BUG #4328: help in creating database encoded with LATIN1

The following bug has been logged online:

Bug reference: 4328
Logged by: andrew victoria
Email address: rtz_andy@yahoo.com
PostgreSQL version: 8.3.1
Operating system: fedora core 9
Description: help in creating database encoded with LATIN1
Details:

i got an error message
createdb: database creation failed: ERROR: encoding LATIN1 does not match
server's locale en_US.utf8
DETAIL: The server's LC_CTYPE setting requires encoding UTF8.

when im trying to create a database encoded with LATIN1 which is needed in
our work. what will i do?

many thanks en more power..

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Re: [GENERAL] copy ... from stdin csv; and bytea

David Wilson wrote:
> On Mon, Jul 28, 2008 at 1:24 AM, Klint Gore <kgore4@une.edu.au> wrote:
> > Try just a single \
> >
> > e.g.
> > "ge.xls","application/vnd.ms-excel",71168,"\320\317\021\340\241[snip]
>
> Thanks- I did try that, and it at least gave the expected output from
> select, but is there a way to verify that it's actually handling it
> correctly rather than simply storing the sequence of characters? I'm
> not certain how to check the actual byte width of a column within a
> row, and I'd *really* rather not be storing 4 bytes for every 1 in the
> binary if I can avoid it- this column is already going to be doubling
> field width; quadrupling it would give me space headaches I really
> don't want to deal with. :)
>
>
select length(bytea_field) from table

You could use ||pg_relation_size|(|text|)| or
||pg_total_relation_size|(|text|) |to see how much disk space it takes up.

You can play with the storage settings for the column if you want to try
and handle the space better. see alter table set storage.

klint.

--
Klint Gore
Database Manager
Sheep CRC
A.G.B.U.
University of New England
Armidale NSW 2350

Ph: 02 6773 3789
Fax: 02 6773 3266
EMail: kgore4@une.edu.au


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[COMMITTERS] npgsql - Npgsql2: [#1010231] Reading result set with UUID is very slow

Log Message:
-----------

[#1010231] Reading result set with UUID is very slow

Forgot to add the patch for the NpgsqlTypeConverters.cs file. Sorry for that. Now cvs head compiles ok. Thanks Matthew Gabeler-Lee and Josh Cooley for heads up.

Modified Files:
--------------
Npgsql2/src/NpgsqlTypes:
NpgsqlTypeConverters.cs (r1.7 -> r1.8)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/npgsql/Npgsql2/src/NpgsqlTypes/NpgsqlTypeConverters.cs.diff?r1=1.7&r2=1.8)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [GENERAL] copy ... from stdin csv; and bytea

On Mon, Jul 28, 2008 at 1:24 AM, Klint Gore <kgore4@une.edu.au> wrote:
> Try just a single \
>
> e.g.
> "ge.xls","application/vnd.ms-excel",71168,"\320\317\021\340\241[snip]

Thanks- I did try that, and it at least gave the expected output from
select, but is there a way to verify that it's actually handling it
correctly rather than simply storing the sequence of characters? I'm
not certain how to check the actual byte width of a column within a
row, and I'd *really* rather not be storing 4 bytes for every 1 in the
binary if I can avoid it- this column is already going to be doubling
field width; quadrupling it would give me space headaches I really
don't want to deal with. :)

--
- David T. Wilson
david.t.wilson@gmail.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] copy ... from stdin csv; and bytea

David Wilson wrote:
> My application is adding a bytea column to a table into which data is
> dumped in approximately 4k row batches, one batch approximately every
> 10 seconds. To this point, those dumps have used copy from stdin;
> however, I'm having some difficulty getting bytea encodings to work
> with it. Specifically, I can't seem to get the parser to recognize
> that what I'm handing it is an escaped string that needs to be parsed
> back into individual bytes rather than stored as-is. The encoding is
> very straightforward for INSERT, of course, but the COPY ... FROM
> STDIN CSV doesn't seem to want to work no matter what I've tried:
>
> \\000
> "\\000"
> "E'\\000'"
>
> etc.
>
> Is there a trick to this that I just didn't see in the documentation,
> or is this some limitation of CSV copy-in? If the latter, are there
> suggestions for workarounds other than to fallback on the inserts?
>
Try just a single \

e.g.
"ge.xls","application/vnd.ms-excel",71168,"\320\317\021\340\241[snip]

klint.

--
Klint Gore
Database Manager
Sheep CRC
A.G.B.U.
University of New England
Armidale NSW 2350

Ph: 02 6773 3789
Fax: 02 6773 3266
EMail: kgore4@une.edu.au


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Getting data from Xml to Postgresql database

aravind chandu wrote:
> Hi,
>
> I have some data in XML format and i need to upload in postgresql database using stored procedure. Can someone tell
> me the step by step procedure of the same as i will be doing it for the
> first time.

As Stephanie Bortzmeyer already explained to you, there is no simple
step-by-step process.

What you need to do depends on the nature of the XML data, and on how
you want to store it in PostgreSQL. Do you wish to load the XML and
transform it into interrelated sets of records in one or more tables? Do
you want to insert the XML document into a single `xml' data field in
the database? etc.

Remember, XML is just a very general mark-up language. It can describe
practically anything.

I can't give you a step-by-step process to load it, but I can give you a
list of things to think about that will help you solve the problem. I'm
assuming that you want to load the XML and transform it into records in
related tables in the database.

- Look at the XML document. Is it a standard XML dialect of some sort?
If so, what does the dialect describe? How would you model that in a
database?

- If the dialect is not standard, what is the XML document describing?
What data does it contain and how is it arranged?

- What are the relationships between data items in the XML document?
Look for repeating sets of records. Look for 1:N relationships where a
"master" entry has zero, one, or more subsiduary entries. At this point,
you should be getting an idea of the "entities" described by the
document, which should translate to tables in a simple database design.

- Determine what the data types of the various items are, which data
items are required and which are optional, etc.

- Look for "key" fields. These will be unique, and might be either ID
numbers or alphanumeric identifiers/codes. If there are keys they might
be used to reference one entry from another, so they're important for:

You've now had a decent look at the data you need to import. You should
now be able to follow an entity relationship modelling process (well
documented on the 'net - use Google) to design your database.

With your database designed, you should now know enough to write and
load the DDL code for it, then write a program that loads and inserts
the XML data. The program should parse the XML into a DOM (or use SAX to
progressively read it, if more appropriate) then, using your knowledge
of its structure and meaning, transform it into data tuples that can be
inserted into the database.

Here's a very simple XML document I just made up.

<?xml encoding=utf-8>
<contact>
<firstname>Fred</lastname>
<lastname>Jones</lastname>
<phone type="mobile">000 000 000</phone>
<phone type="home">0000 0000</phone>
<address>
<street>55 Nowhere Rd</street>
<town>Sub Urbia</town>
<state>Western Australia</state>
<country>Australia</country>
<postcode>6999</postcode>
<ismailingaddress/>
<isresidentialaddress/>
</address>
<address>
<street>8 Void Street</street>
<town>Some Tiny Village</town>
<state>Western Australia</state>
<country>Australia</country>
<postcode>6888</postcode>
<isworkaddress/>
</address>
</contact>

It's pretty obvious that this document describes contacts, each of which
has a first and last name, a list of phone numbers, and a list of addresses.

Without a DTD to tell us the rules the document follows (must there be
at least one phone number? etc) we have to make some guesses about how
to store the data. We'll generate a synthetic primary key since no
really appropriate key is present (the only alternative would be to use
a composite (firstname,lastname) primary key, which has plenty of problems):

Something like this should be usable to store the data:

CREATE TABLE contact (
contact_id SERIAL PRIMARY KEY,
firstname text NOT NULL,
lastname text NOT NULL
);

CREATE TABLE phone (
phone_id SERIAL PRIMARY KEY,
contact_id integer REFERENCES contact(contact_id),
phonetype text,
phonenumber text NOT NULL
);

CREATE TABLE address (
address_id SERIAL PRIMARY KEY,
contact_id integer REFERENCES contact(contact_id),
street text,
town text,
state text,
country text,
postcode text,
ismailingaddress bool,
isstreetaddress bool,
isworkaddress bool
);

If you knew more about the rules the XML document structure and data
must follow you could produce a better design with NOT NULL constraints
where appropriate, etc. You might fold the "phone" table into the
"contact" table if the document only permitted "type" values of "home"
or "mobile" for the "phone" field. That sort of thing.

--
Craig Ringer

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [pgsql-www] Insecure DNS servers on PG infrastructure

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Friday, July 25, 2008 11:02:03 -0400 Tom Lane <tgl@sss.pgh.pa.us> wrote:

> I just noted that cvs.postgresql.org and svr1.postgresql.org are not

Was out camping, just saw this ... svr1.postgresql.org's name server is
shutdown/disabled now, since its not needed ... and all of the primary name
servers have had recursion disabled since Andrew pointed out to me awhile back
how to do so ...


- --
Marc G. Fournier Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (FreeBSD)

iEYEARECAAYFAkiNSnwACgkQ4QvfyHIvDvOs2QCfTJPaWg2si8wbdmGSVy4p8Xdx
GIsAoIl4TDOy4axlpAKDfmgwgOoxPb4s
=Ro1U
-----END PGP SIGNATURE-----


--
Sent via pgsql-www mailing list (pgsql-www@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

[GENERAL] copy ... from stdin csv; and bytea

My application is adding a bytea column to a table into which data is
dumped in approximately 4k row batches, one batch approximately every
10 seconds. To this point, those dumps have used copy from stdin;
however, I'm having some difficulty getting bytea encodings to work
with it. Specifically, I can't seem to get the parser to recognize
that what I'm handing it is an escaped string that needs to be parsed
back into individual bytes rather than stored as-is. The encoding is
very straightforward for INSERT, of course, but the COPY ... FROM
STDIN CSV doesn't seem to want to work no matter what I've tried:

\\000
"\\000"
"E'\\000'"

etc.

Is there a trick to this that I just didn't see in the documentation,
or is this some limitation of CSV copy-in? If the latter, are there
suggestions for workarounds other than to fallback on the inserts?

Using 8.3.3, and this is specifically via libpq, if that makes a difference.

Thanks much.
--
- David T. Wilson
david.t.wilson@gmail.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[ANNOUNCE] == PostgreSQL Weekly News - July 27 2008 ==

== PostgreSQL Weekly News - July 27 2008 ==

July's commitfest is nearly over, with lots of new features being
added.

== PostgreSQL Product News ==

Benetl 2.2 released.
http://www.benetl.net

check_postgres 2.1.0 released.
http://bucardo.org/check_postgres/

pgpool-II 2.1 released.
http://pgfoundry.org/frs/shownotes.php?group_id=1000055&release_id=1187

== PostgreSQL Jobs for July ==

http://archives.postgresql.org/pgsql-jobs/2008-07/threads.php

== PostgreSQL Local ==

Sponsor the European PGDay!
http://www.pgday.org/en/sponsors/campaign

The Call for Papers for European PGDay has begun.
http://www.pgday.org/en/call4papers

pgDay San Francisco will be August 5. Schedule:
http://pugs.postgresql.org/node/447
Register here:
http://www.linuxworldexpo.com/live/12/ehall//SN460564

PGCon Brazil 2008 will be on September 26-27 at Unicamp in Campinas.
http://pgcon.postgresql.org.br/index.en.html

PGDay.(IT|EU) 2008 will be October 17 and 18 in Prato.
http://www.pgday.org/it/

== PostgreSQL in the News ==

Planet PostgreSQL: http://www.planetpostgresql.org/

General Bits, Archives and occasional new articles:
http://www.varlena.com/GeneralBits/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david@fetter.org, German language
to pwn@pgug.de, Italian language to pwn@itpug.org.

== Applied Patches ==

Tom Lane committed:

- In pgsql/src/backend/commands/prepare.c, add comment about the two
different query strings that ExecuteQuery() has to deal with.

- In pgsql/doc/src/sgml/gin.sgml, update link to Oleg and Teodor's GIN
page. (Extracted from fast-insert patch, since it ought to be
back-patched)

- Use guc.c's parse_int() instead of pg_atoi() to parse fillfactor in
default_reloptions(). The previous coding was really a bug because
pg_atoi() will always throw elog on bad input data, whereas
default_reloptions is not supposed to complain about bad input
unless its validate parameter is true. Right now you could only
expose the problem by hand-modifying pg_class.reloptions into an
invalid state, so it doesn't seem worth back-patching; but we should
get it right in HEAD because there might be other situations in
future. Noted while studying GIN fast-update patch.

- In pgsql/src/backend/libpq/hba.c, remove some redundant tests and
improve comments in next_token(). Cosmetic, but it might make this
a bit less confusing to the next reader.

- In pgsql/src/backend/libpq/auth.c, fix parsing of LDAP URLs so it
doesn't reject spaces in the "suffix" part. Per report from Cesar
Miguel Oliveira Alves.

- As noted by Andrew Gierth, there's really no need any more to force
a junk filter to be used when INSERT or SELECT INTO has a plan that
returns raw disk tuples. The virtual-tuple-slot optimizations that
were put in place awhile ago mean that ExecInsert has to do
ExecMaterializeSlot, and that already copies the tuple if it's raw
(and does so more efficiently than a junk filter, too). So get rid
of that logic. This in turn means that we can throw away
ExecMayReturnRawTuples, which wasn't used for any other purpose, and
was always a kluge anyway. In passing, move a couple of
SELECT-INTO-specific fields out of EState and into the private state
of the SELECT INTO DestReceiver, as was foreseen in an old comment
there. Also make intorel_receive use ExecMaterializeSlot not
ExecCopySlotTuple, for consistency with ExecInsert and to possibly
save a tuple copy step in some cases.

Alvaro Herrera committed:

- In pgsql/src/backend/postmaster/autovacuum.c, publish more openly
the fact that autovacuum is working for wraparound protection.
Simon Riggs.

- In pgsql/src/backend/postmaster/autovacuum.c, ratchet up patch to
improve autovacuum wraparound messages. Simon Riggs.

Peter Eisentraut committed:

- Abort if Tcl support was configured and no tcl shell was found.
This is required because the value is substituted into the
pltcl_*mod scripts.

== Rejected Patches (for now) ==

No one was disappointed this week :-)

== Pending Patches ==

Markus Wanner sent in a patch to rename query_string to prepared_qs.

Robert Lor sent in a revised patch for dtrace probes.

David Gould sent in a patch for pg_dumpall which adds a
--lock-wait-timeout parameter similar to the one he added for pg_dump.

Simon Riggs sent in revision of his patch to make autovacuum report
when it is preventing a transaction ID wraparound for CVS HEAD and
8.3.

Simon Riggs sent in two more revisions of a patch to add
--schema-before-data and --schema-after-data options to pg_dump.

Xiao Meng sent in another version of his patch to improve performance
on hash indexes.

Robert Haas sent in a patch to add a separate permission for TRUNCATE.

Tatsuo Ishii sent in three more revisions of Yoshiyuki Asaba's patch
to implement CTEs.

Simon Riggs sent a patch to add a contrib module which adds a
Transportable Optimizer Mode to PostgreSQL.


---------------------------(end of broadcast)---------------------------
-To unsubscribe from this list, send an email to:

pgsql-announce-unsubscribe@postgresql.org

Re: [SQL] PERSISTANT PREPARE (another point of view)

> > 2. perhaps "global" could mean simply that
> the definition is global - if called for session and not
> exist in session, then session prepares it first from the
> global def. there would need to be a back reference in
> case the global def was changed or dropped.
> >
>
> Yes, this seems to be a good idea. Something like this
> would be easier
> to implement then having the whole process run in some
> shared memory
> space. The implementation could by like:
>
> 1. App cals SQL like "EXEC
> <statement_name>"
> 2. PG Engine looks first in local prepared statements as it
> does now
> 3. If it doesn't find it there it looks in public space
> 4. If it's found there copy both the SQL and the
> execution plan stored
> in global space to local process
> 5. Execute the statement as if it was prepared in local
> process.
>
> Simplified greatly, new implementation could
> "only" add steps 3 and 4 to
> current implementation of PREPARED statements.

I think it only makes sense if the prepared statement is in the session.

For example, say the Globally submitted statement is
stmt: "select * from tab1"
assume
Session #1 has search_path=schema1
Session #2 has search_path=schema2
if session #1 attempts to exec stmt, it prepare and use schema1
if session #2 attempts to use stmt, if prepared globally, disaster

in submitted globally, it is not prepared at all until used, but, if ever used, prepared in the current session's environment. really more like a rule who's preparation is remembered in the session.


--
Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-sql

Re: [PATCHES] pg_dump additional options for performance

* Joshua D. Drake (jd@commandprompt.com) wrote:
> Custom format rocks for partial set restores from a whole dump. See the
> TOC option :)

I imagine it does, but that's very rarely what I need. Most of the time
we're dumping out a schema to load it into a seperate schema (usually on
another host). Sometimes that can be done by simply vi'ing the file to
change the search_path and whatnot, though more often we end up pipe'ing
the whole thing through sed. Since we don't allow regular users to do
much, and you have to 'set role postgres;' to do anything as superuser,
we also often end up adding 'set role postgres;' to the top of the .sql
files.

Thanks,

Stephen

[JDBC] New to pgsql-jdbc

Hi, I am new to PostgreSQL. We are using XBow wireless motes with PostgreSQL database(8.0 version). Since XBow already provides DB's structure and we don't have to develop pgsql DB. For our needs, we want to bind pgsql with our database called "Cache" which also support with standard JDBC and Java proxy classes mapping. I wonder which driver I should download and how to develop Java bindings for XBow's pgsql?  How could we generate proxy classes for XBow's pgsql? Can somebody post some links for starter? I am using windows XP sp2/ windows 2003 server. Thanks

--
Best Regards

Kelvin Feng

Re: [PATCHES] pg_dump additional options for performance

Stephen Frost wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> Stephen Frost <sfrost@snowman.net> writes:
>>> I dislike, and doubt that I'd use, this approach. At the end of the
>>> day, it ends up processing the same (very large amount of data) multiple
>>> times.

> This would depend on the dump being in the custom format, though I
> suppose that ends up being true for any usage of these options. I've
> never really been a fan of the custom format, in large part because it
> doesn't really buy you all that much and makes changing things more
> difficult (by having to extract out what you want to change, and then
> omit it from the restore).

Custom format rocks for partial set restores from a whole dump. See the
TOC option :)

Joshua D. Drake

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [GENERAL] Profiling postgres

configure postgres with the following options

1. ./configure --enable-profiling
2. make and make install
3. start psql and issue the query
4. gmon.out will be in the data directory once you exit from psql

-Sushant.

On Sun, 2008-07-27 at 05:55 -0700, Suresh wrote:
>
> Hello,
>
> I want to profile some functions in postgres using gprof. I am
> following this procedure listed in nabble forum.
>
> # cd postgres/bin
> # cd .../src/backend
> # make clean
> # make PROFILE="-pg" all
>
> How to proceed after this ? I am not getting the gmon.out file. How do
> I run ./postgres test under the profiler ?
>
> thanks and regards,
> Suresh
>
>
>
>


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] [PATCHES] pg_dump additional options for performance

On Sun, Jul 27, 2008 at 10:37:34AM +0100, Simon Riggs wrote:
>
> On Sat, 2008-07-26 at 11:03 -0700, Joshua D. Drake wrote:
>
> > 2. We have no concurrency which means, anyone with any database over 50G
> > has unacceptable restore times.
>
> Agreed.
>
> Also the core reason for wanting -w
>
> > 3. We have to continue develop hacks to define custom utilization. Why
> > am I passing pre-data anything? It should be automatic. For example:

[adding hackers for discussion]

On Sat, Jul 26, 2008 at 01:56:14PM -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > I want to dump tables separately for performance reasons. There are
> > documented tests showing 100% gains using this method. There is no gain
> > adding this to pg_restore. There is a gain to be had - parallelising
> > index creation, but this patch doesn't provide parallelisation.
>
> Right, but the parallelization is going to happen sometime, and it is
> going to happen in the context of pg_restore. So I think it's pretty
> silly to argue that no one will ever want this feature to work in
> pg_restore.
>
> To extend the example I just gave to Stephen, I think a fairly probable
> scenario is where you only need to tweak some "before" object
> definitions, and then you could do
>
> pg_restore --schema-before-data whole.dump >before.sql
> edit before.sql
> psql -f before.sql target_db
> pg_restore --data-only --schema-after-data -d target_db whole.dump
>
> which (given a parallelizing pg_restore) would do all the time-consuming
> steps in a fully parallelized fashion.

A few thoughts about pg_restore performance:

To take advantage of non-logged copy, the table create and data load should
be in the same transaction.

To take advantage of file and buffer cache, it would be better to create
indexes immediately after loading table data. Many tables will be small
enough to fit in cache on and this will avoid re-reading them for index
builds. This is more advantagious with more indexes on one table. There
may also be some filesytem placement benefits to building the indexes for
a table immediately after loading the data.

Creating constraints immediately after loading data also would benefit from
warm buffer and file caches. Doing this this is complicated by the need
for indexes and data in the referenced tables to exist first.

It seems that a high performance restore will want to procede in a different
order than the current sort order or that proposed by the before/data/after
patch.

- The simplest unit of work for parallelism may be the table and its
"decorations", eg indexes and relational constraints.

- Sort tables by foreign key dependency so that referenced tables are
loaded before referencing tables.

- Do table creation and data load together in one transaction to use
non-logged copy. Index builds, and constraint creation should follow
immediately, either as part of the same transaction, or possibly
parallelized themselves.

Table creation, data load, index builds, and constraint creation could
be packaged up as the unit of work to be done in a subprocess which either
completes or fails as a unit. The worker process would be called with
connection info, a file pointer to the data, and the DDL for the table.
pg_restore would keep a work queue of tables to be restored in FK dependency
order and also do the other schema operations such as functions and types.

-dg

--
David Gould daveg@sonic.net 510 536 1443 510 282 0869
If simplicity worked, the world would be overrun with insects.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [PATCHES] pg_dump additional options for performance

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Right, but the parallelization is going to happen sometime, and it is
> going to happen in the context of pg_restore. So I think it's pretty
> silly to argue that no one will ever want this feature to work in
> pg_restore.

I think you've about convinced me on this, and it annoys me. ;) Worse
is that it sounds like this might cause the options to not make it in
for 8.4, which would be quite frustrating.

> To extend the example I just gave to Stephen, I think a fairly probable
> scenario is where you only need to tweak some "before" object
> definitions, and then you could do
>
> pg_restore --schema-before-data whole.dump >before.sql
> edit before.sql
> psql -f before.sql target_db
> pg_restore --data-only --schema-after-data -d target_db whole.dump
>
> which (given a parallelizing pg_restore) would do all the time-consuming
> steps in a fully parallelized fashion.

Alright, this has been mulling around in the back of my head a bit and
has now finally surfaced- I like having the whole dump contained in a
single file, but I hate having what ends up being "out-dated" or "wrong"
or "not what was loaded" in the dump file. Doesn't seem likely to be
possible, but it'd be neat to be able to modify objects in the dump
file.

Also, something which often happens to me is that I need to change the
search_path or the role at the top of a .sql from pg_dump before
restoring it. Seems like using the custom format would make that
difficult without some pipe/cat/sed magic. Parallelization would make
using that kind of magic more difficult too, I would guess. Might be
something to think about.

Thanks,

Stephen

Re: [PATCHES] pg_dump additional options for performance

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > I dislike, and doubt that I'd use, this approach. At the end of the
> > day, it ends up processing the same (very large amount of data) multiple
> > times.
>
> Well, that's easily avoided: just replace the third step by restoring
> directly to the target database.
>
> pg_restore --schema-before-data whole.dump >before.sql
> edit before.sql
> pg_restore --schema-after-data whole.dump >after.sql
> edit after.sql
> psql -f before.sql target_db
> pg_restore --data-only -d target_db whole.dump
> psql -f after.sql target_db

This would depend on the dump being in the custom format, though I
suppose that ends up being true for any usage of these options. I've
never really been a fan of the custom format, in large part because it
doesn't really buy you all that much and makes changing things more
difficult (by having to extract out what you want to change, and then
omit it from the restore).

I can see some advantage to having the entire dump contained in a single
file and still being able to pull out pieces based on before/after.
Should we get a binary format which is much faster, I could see myself
being more likely to use pg_restore. Same for parallelization or, in my
fantasies, the ability to copy schema, tables, indexes, etc, in 'raw' PG
format between servers. Worse than having to vi an insanely large file,
or split it up to be able to modify the pieces you want, is having to
rebuild indexes, especially GIST ones. That's another topic though.

Thanks,

Stephen

Re: [pgsql-www] Insecure DNS servers on PG infrastructure

Alvaro Herrera wrote:
> Tom Lane wrote:
>> "Joshua D. Drake" <jd@commandprompt.com> writes:
>>> Tom can you check if this has been resolved? If not I am going to start
>>> paging people.
>> The machines I have access to (cvs and svr1) are still unpatched
>> according to the "dig porttest" test. Personally, though, I'd be
>> more interested in getting someone to work on fixing the mailing
>> lists. I've received nothing since about midnight EDT, and I know
>> that there has been traffic.
>
> Maybe the mail servers have been DDoS'ed again?

I did have a bunch of alerts throw recovery around 2:00am but then where
is all the traffic?

>


--
Sent via pgsql-www mailing list (pgsql-www@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

Re: [PATCHES] pg_dump additional options for performance

Andrew Dunstan wrote:
>
>
> Joshua D. Drake wrote:
>>
>> Agreed but that is a problem I understand with a solution I don't. I
>> am all eyes on a way to fix that. One thought I had and please, be
>> gentle in response was some sort of async transaction capability. I
>> know that libpq has the ability to send async queries. Is it possible
>> to do this:
>>
>> send async(copy table to foo)
>> send async(copy table to bar)
>> send async(copy table to baz)
>>
>> Where all three copies are happening in the background?
>>
>>
>
> IIRC, libpq doesn't let you have more than one async query active at one
> time.

Now that I think on it harder, this isn't even a libpq problem (although
its involved), we need the postmaster do be able to do a background
async query. Which is (I am guessing) why libpq can only do one at a time.

Sincerely,

Joshua D. Drake

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [SQL] Re: Efficiently determining the number of bits set in the contents of, a VARBIT field

TJ O'Donnell wrote:
> I use a c function, nbits_set that will do what you need.
> I've posted the code in this email.
>
> TJ O'Donnell
> http://www.gnova.com
>
> #include "postgres.h"
> #include "utils/varbit.h"
>
> Datum nbits_set(PG_FUNCTION_ARGS);
> PG_FUNCTION_INFO_V1(nbits_set);
> Datum
> nbits_set(PG_FUNCTION_ARGS)
> {
> /* how many bits are set in a bitstring? */
>
> VarBit *a = PG_GETARG_VARBIT_P(0);
> int n=0;
> int i;
> unsigned char *ap = VARBITS(a);
> unsigned char aval;
> for (i=0; i < VARBITBYTES(a); ++i) {
> aval = *ap; ++ap;
> if (aval == 0) continue;
> if (aval & 1) ++n;
> if (aval & 2) ++n;
> if (aval & 4) ++n;
> if (aval & 8) ++n;
> if (aval & 16) ++n;
> if (aval & 32) ++n;
> if (aval & 64) ++n;
> if (aval & 128) ++n;
> }
> PG_RETURN_INT32(n);
> }
>
>
>
>> Hi all,
>> Am looking for a fast and efficient way to count the number of bits set
>> (to 1) in a VARBIT field. I am currently using
>> "LENGTH(REGEXP_REPLACE(CAST(a.somefield_bit_code AS TEXT),'0','','g'))".
>>
>> Allan.
>
>
When I had to do that, in days with smaller amounts of RAM, but very long
bit-vectors, I used a faster function sort-of like this:

static char table[256] = {
0,1,1,2,1,2,2,3,1,.....
};

Then like above, but instead of the loop,

n+= table[aval];


You get the idea.

--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key: 9A2FC99A Registered Machine 241939.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 20:20:01 up 7 days, 1:08, 4 users, load average: 4.16, 4.15, 4.10

--
Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-sql

Re: [pgsql-www] Insecure DNS servers on PG infrastructure

Tom Lane wrote:
> "Joshua D. Drake" <jd@commandprompt.com> writes:
> > Tom can you check if this has been resolved? If not I am going to start
> > paging people.
>
> The machines I have access to (cvs and svr1) are still unpatched
> according to the "dig porttest" test. Personally, though, I'd be
> more interested in getting someone to work on fixing the mailing
> lists. I've received nothing since about midnight EDT, and I know
> that there has been traffic.

Maybe the mail servers have been DDoS'ed again?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

--
Sent via pgsql-www mailing list (pgsql-www@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

Re: [PATCHES] pg_dump additional options for performance

Joshua D. Drake wrote:
>
> Agreed but that is a problem I understand with a solution I don't. I
> am all eyes on a way to fix that. One thought I had and please, be
> gentle in response was some sort of async transaction capability. I
> know that libpq has the ability to send async queries. Is it possible
> to do this:
>
> send async(copy table to foo)
> send async(copy table to bar)
> send async(copy table to baz)
>
> Where all three copies are happening in the background?
>
>

IIRC, libpq doesn't let you have more than one async query active at one
time.

cheers

andrew

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [GENERAL] Substitute a variable in PL/PGSQL.

artacus@comcast.net wrote:

>You can do it in straight sql like so.
>
>SELECT (array[col1, col2, col3, col4, col5, col6, col7, col8, col9, col10])[i]
>FROM test t, generate_series(1,10) i
>
>Art
>
>
>
Hi Art,
Thanks for the advice, in my case using arrays was not a option as the
data could be null.
Steve M.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Substitute a variable in PL/PGSQL.

Klint Gore wrote:

> Steve Martin wrote:
>
>> I am trying to create a PL/PGSQL function to return the values of the
>> fields in a record, e.g. 1 value per row in the output of the function.
>>
>> How do you substitute a variable?
>>
>>
>> CREATE OR REPLACE FUNCTION testfunc() RETURNS SETOF text AS $$
>> DECLARE ted varchar;
>> bob RECORD;
>> BEGIN
>> FOR bob IN SELECT * FROM test LOOP
>> FOR i IN 1..10 LOOP
>> ted := 'bob.col' || i;
>> RETURN NEXT ted;
>> END LOOP;
>> END LOOP;
>> RETURN;
>> END
>> $$ LANGUAGE plpgsql;
>>
>>
>> Or is there another way other than using another procedural language.
>>
>> Thanks - Steve M.
>>
>
>
> There's no direct way to reference a particular field in a record
> variable where the field name is held in a variable in pl/pgsql.
> I.E. if ted = 'col1' there's no way to reference bob.ted to give you
> the value of bob.col1.
>
> If you want it easy to code but have to create something for every
> table and modify it ever time the table changes
>
> create view test_vertical_table as
> select col1::text from test
> union all
> select col2::text from test
> union all
> select col3::text from test
> union all
> select col4::text from test
> union all
> select col5::text from test
> ...
>
>
> If you want to go the generic function route
>
> CREATE OR REPLACE FUNCTION testfunc(text) RETURNS SETOF text AS $$
> DECLARE vertTableName alias for $1;
> ted text;
> bob RECORD;
> bill record;
> BEGIN
> for bill in select table_name, column_name from
> information_schema.columns where table_schema = public
> and table_name = vertTableName
> loop
> FOR bob IN execute 'SELECT '||bill.column_name||' as
> thiscol FROM '||bill.table_name LOOP
> ted := bob.thiscol;
> RETURN NEXT ted;
> END LOOP;
> end loop;
> RETURN;
> END
> $$ LANGUAGE plpgsql;
>
>
>
> klint.
>
Hi Klint,
Thanks for the advice, I found the sql to get the column names useful.
Steve M.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Substitute a variable in PL/PGSQL.

Roberts, Jon wrote:

>>What I am trying to do is find the difference between two tables, one
>>that stores the
>>information in a single column, and the other which stores the same
>>
>>
>data
>
>
>>in multiple
>>columns.
>>
>>E.g.
>>CREATE TABLE test(col1 text, col2 text, col3 text, col4 text, col5
>>
>>
>text,
>
>
>>col6 text, col7 text, col8 text, col9 text, col10 text);
>>CREATE TABLE test2(col_data text NOT NULL, some_data text NOT NULL,
>>other_data text,
>> CONSTRAINT test2_index PRIMARY
>>
>>
>KEY(
>
>
>> col_data,
>> some_data ));
>>
>>Trying to find data set in test2.col_data that is not in test.col1 to
>>test.col10.
>>
>>
>>
>
>FINALLY you get to the requirements. Next time, just ask a question
>like the above. You were asking how to solve a technical problem that
>didn't relate to the actual business need.
>
>Here are three ways to skin this cat.
>
>--version 1
>select col_data from test2
>except
>select coalesce(col1, '') || coalesce(col2, '') || coalesce(col3, '') ||
>
> coalesce(col4, '') || coalesce(col5, '') || coalesce(col6, '') ||
>
> coalesce(col7, '') || coalesce(col8, '') || coalesce(col9, '') ||
>
> coalesce(col10, '')
> from test
>
>--version 2
>select col_data
> from test2 t2
> where not exists (select null
> from test t
> where t2.col_data = coalesce(t.col1, '') ||
> coalesce(t.col2, '') ||
> coalesce(t.col3, '') ||
> coalesce(t.col4, '') ||
> coalesce(t.col5, '') ||
> coalesce(t.col6, '') ||
> coalesce(t.col7, '') ||
> coalesce(t.col8, '') ||
> coalesce(t.col9, '') ||
> coalesce(t.col10, ''))
>--version 3
>select t2.col_data
> from test2 t2
> left join (select coalesce(col1, '') || coalesce(col2, '') ||
> coalesce(col3, '') || coalesce(col4, '') ||
> coalesce(col5, '') || coalesce(col6, '') ||
> coalesce(col7, '') || coalesce(col8, '') ||
> coalesce(col9, '') || coalesce(col10, '') as
>col_data
> from test) t
> on t2.col_data = t.col_data
> where t.col_data is null
>
>
>Jon
>
>
Thanks Jon for the hints.
Steve

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [SQL] PERSISTANT PREPARE (another point of view)

Pavel Stehule wrote:
> Hello
>
> this is near my implemenation of stored procedures - it's not far from
> your view on prepared statements. There result of any unbound select
> is directly forwarded to client - there is little bit different
> syntax, but idea is same
>
>
> create procedure foo(params)
> as $$
> begin
> select a, b, c from tab1 -- unbound select
> select ....
> end;
>
> and you can call with CALL statement. Curent implementation of SRF
> function in plpgsql sn't eficient for big results - it needs
> materialised table in memory. Pipeline Oracle's functions are better,
> but we are limited one thread architecture.
>
> regards
> Pavel Stehule
>

Yeah, this would be pretty the same.

I just didn't understand if this is already implemented in PG ?

I've tried to create a PROCEDURE in QueryTool of pgAdminIII and I get
following result:

ERROR: syntax error at or near "PROCEDURE"
LINE 1: CREATE PROCEDURE aatestubnd(refcursor, insklid integer, inda...
^

********** Error **********

ERROR: syntax error at or near "PROCEDURE"
SQL state: 42601
Character: 8


If I try with CREATE FUNCTION I get this result (as expected):

ERROR: function result type must be specified

********** Error **********

ERROR: function result type must be specified
SQL state: 42P13


Is this because of QueryTool's limitations or is this feature not yet
implemented in Postgre ?

Though, CREATE PROCEDURE should be a valid SQL 92 standard procedure.

Best regards,

Milan Oparnica

--
Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-sql

Re: [GENERAL] Problems Restarting PostgreSQL Daemon

On Mon, 28 Jul 2008, Andrej Ricnik-Bay wrote:

> Can you do a 'locate pg_ctl|xargs ls -l' and see whether you have more
> than one installed, and if so, which one comes first in the PATH?

Andrej,

There are two:

-rwxr-xr-x 1 root root 24320 2008-06-17 16:18 /usr/bin/pg_ctl
-rw-r--r-- 1 root root 17286 2007-11-10 13:48
/usr/doc/postgresql-8.3.3/src/sgml/ref/pg_ctl-ref.sgml
-rw-r--r-- 1 root root 3536 2008-06-17 16:18 /usr/man/man1/pg_ctl.1.gz
-rwxr-xr-x 1 root root 32316 2008-06-18 09:50 /usr3/pg813/bin/pg_ctl
-rw-r--r-- 1 root root 8888 2008-06-18 09:50 /usr3/pg813/man/man1/pg_ctl.1

and /usr3/pg813 is not in anyone's path. It's a data storage filesystem and
holds the earlier version during the upgrade. And, now that 8.3.3 is
running, I'll remove that directory.

Postgres' path is: /bin:/usr/bin:/home/postgres/bin:/sbin:/usr/sbin.

Rich

--
Richard B. Shepard, Ph.D. | Integrity Credibility
Applied Ecosystem Services, Inc. | Innovation
<http://www.appl-ecosys.com> Voice: 503-667-4517 Fax: 503-667-8863

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [SQL] PERSISTANT PREPARE (another point of view)

chester c young wrote:
> 1. like the idea because right now I'm having to track which prepared statement (sets) are in which session. using xcache to track this, but little bit dangerous. could query the database first but the whole idea is to take a load off the db.
>

Thnx for support. The whole idea is to have DBMS support as much SQL
related functionality as possible. This way you wouldn't have to wonder
if the client has prepared the statement already or not. If you stored
it in the public schema then it MUST be there.

> 2. perhaps "global" could mean simply that the definition is global - if called for session and not exist in session, then session prepares it first from the global def. there would need to be a back reference in case the global def was changed or dropped.
>

Yes, this seems to be a good idea. Something like this would be easier
to implement then having the whole process run in some shared memory
space. The implementation could by like:

1. App cals SQL like "EXEC <statement_name>"
2. PG Engine looks first in local prepared statements as it does now
3. If it doesn't find it there it looks in public space
4. If it's found there copy both the SQL and the execution plan stored
in global space to local process
5. Execute the statement as if it was prepared in local process.

Simplified greatly, new implementation could "only" add steps 3 and 4 to
current implementation of PREPARED statements.

> 3. don't understand your statement about how prepared statements can be used as subqueries - you mean something like
>
> select .. from tab1 t1 where t1.col1 in( exec prep1(..) )
>
> or exactly what?
>

Well, its best explained on the example. Note that such construction
would require extra coding from PG developers.

##1 Lets pretend we have a prepared statement:

PREPARE PS_Sector (InSector INTEGER) AS
SELECT SectorID,SectorName,SectorType
FROM CompanySectors
WHERE SectorID = InSector;


##2 Now we can build another statement over the firs one like this:

PREPARE PS_SectorExpences (InDateFrom DATETIME, InDateTo DATETIME,
InSector INTEGER) AS
SELECT S.SectorID,S.SectorName,S.SectorType,E.ExpType,E.Ammount
FROM PS_Sector AS S INNER JOIN CompanyExpences AS E
ON S.SectorID=E.SectorID
WHERE E.Date>=InDateFrom AND E.Date<=InDateTo;


Let me explain. Statement ##2 uses PS_Sector in direct join with a table
with expences. By automatically passing "by name" the parameter InSector
to the underlaying prepared statement it first returns results from it.
Then it joins it to the table CompanyExpences by field SectorID.

This may look complicated to implement but it's just a simplified
version of implementing SUB-QUERIES. The same effect is gained if you
simple replace the PS_Sector reference in ##2 by actual SQL statement it
holds.

PREPARE PS_SectorExpences (InDateFrom DATETIME, InDateTo DATETIME,
InSector INTEGER) AS
SELECT S.SectorID,S.SectorName,S.SectorType,E.ExpType,E.Ammount
FROM (SELECT SectorID,SectorName,SectorType
FROM CompanySectors
WHERE SectorID = InSector) AS S INNER JOIN CompanyExpences AS E
ON S.SectorID=E.SectorID
WHERE E.Date>=InDateFrom AND E.Date<=InDateTo;


Only, this way, subquery can come with execution plan from ##1 prepared
statemnt.


Where could this be used ?

Since it's obvious performance downgrade, you must have a good reason
for using such approach. Mostly it's reporting !!!

Good report relies on data it gets. Sometimes statements needed for
reporting purposes include very complex data selection. Usually it takes
a long period of time until "everything is covered" by your SELECT
query. At that point you can implement the core logic into a "base
perpared statement" and then use prepared statements over it to get
various aggregations, or different point of views of data.

But, the best thing is: WHEN YOU ADD SOMETHING TO BASIC LOGIC OF YOUR
SYSTEM, YOU ONLY CHANGE THE BASE PREPARED STATEMENT. If you wrote it
cleverly, than all statements built upon it WILL IMMEDIATELY SEE THE NEW
IMPLEMENTATION. This is very powerful and efficient way of introducing
improvements in the system not having to worry that majority of your
reports will show false data until you find time to recompile them.

NOTE. You don't want to manipulate huge database systems this way but
its very usable for 99% of small and medium business apps (databases up
to few gigabytes). It greatly depends on your internal database
representation though.

Best regards,

Milan Oparnica

--
Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-sql

Re: [GENERAL] Problems Restarting PostgreSQL Daemon

2008/7/28 Rich Shepard <rshepard@appl-ecosys.com>:
> Thank you. I think that for some reason using pg_ctl to start the
> postmaster is no longer working here. As I have time, I'll look into why.
Can you do a 'locate pg_ctl|xargs ls -l' and see whether you have more than one
installed, and if so, which one comes first in the PATH?

> Rich
Cheers,
Andrej


--
Please don't top post, and don't use HTML e-Mail :} Make your quotes concise.

http://www.american.edu/econ/notes/htmlmail.htm

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [pgadmin-hackers] SQL tab on Server's properties dialog?

Dave Page a écrit :
> On Sat, Jul 26, 2008 at 10:44 PM, Guillaume Lelarge
> <guillaume@lelarge.info> wrote:
>> [...]
>> While working on the dialogs' review, I was wondering why we show the SQL
>> tab for the Server's properties dialog. I think we shouldn't display this
>> tab on this dialog. The patch enclosed does this.
>>
>> I'm not sure this qualifies as a bug. But it shouldn't be hard to apply the
>> patch on the 1.8 branch.
>>
>> Comments?
>
> Bung it on -head, but don't bother with the 1.8 branch.
>

Done. I don't know why the commit message didn't arrive on the list.


--
Guillaume.
http://www.postgresqlfr.org
http://dalibo.com

--
Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers

Re: [pgsql-www] Insecure DNS servers on PG infrastructure

Andrew Sullivan wrote:
> On Fri, Jul 25, 2008 at 11:02:03AM -0400, Tom Lane wrote:
>> I just noted that cvs.postgresql.org and svr1.postgresql.org are not
>> running the latest bind release, which means that they are vulnerable to
>> the DNS cache poisoning attack recently discovered by Dan Kaminsky.
>> Vixie and co think this is a pretty big deal, so folks might want to
>> update sooner rather than later.
>
> This is an extremely big deal. The numbers I've seen suggest windows
> somewhere around 10 minutes. If the systems above are doing
> recursion, then they need to be patched right away. (If they're
> running both authority and recursive services in the same BIND
> instance, I suggest that the practice be abandoned immediately.)

cvs.postgresql.org is not running bind at all - what it is using are two
(purely) recursive resolvers upstream. One of them is only going to get
upgraded tomorrow(some changes need to be rolled out in a staged
fashion) the other one was done a while ago - I have simply removed that
one from the resolv.conf for the time being.

Stefan

--
Sent via pgsql-www mailing list (pgsql-www@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

Re: [GENERAL] Problems Restarting PostgreSQL Daemon

On Sun, 27 Jul 2008, Andrej Ricnik-Bay wrote:

> I thought we had established that this issue was caused by the current
> instance pointing at the old installs data directory?

No, that wasn't the problem.

If I use 'postgres -D /var/lib/pgsql/data &' the postmaster starts
correctly and everything runs as intended. If I use '/etc/rc.d/rc.postgresql
start' I get error messages about the postmaster already running and an
invalid .pid.

> That should be quite easy to tweak, really ... my current script (slightly
> modified from the one in contrib/startup-scripts) is attached... You may
> need to change the dirs in the script yet a bit.

Thank you. I think that for some reason using pg_ctl to start the
postmaster is no longer working here. As I have time, I'll look into why.

Rich

--
Richard B. Shepard, Ph.D. | Integrity Credibility
Applied Ecosystem Services, Inc. | Innovation
<http://www.appl-ecosys.com> Voice: 503-667-4517 Fax: 503-667-8863

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[GENERAL] Getting data from Xml to Postgresql database

Hi,

    I have some data in XML format and i need to upload in postgresql database using stored procedure. Can someone tell me the step by step procedure of the same as i will be doing it for the first time.
 

Thank You,
Avin.

Re: [HACKERS] Protocol 3, Execute, maxrows to return, impact?

Stephen R. van den Berg wrote:
>My Pike drivers now support multiple simultaneous portals and
>automatic streaming by presending overlapping Execute statements with
>a dynamically adapted fetchlimit calculated per select as the query
>progresses.

They also support COPY now.

The driver beats libpq in speed by about 62%.
The memory consumption is on demand, by row, and not the whole result set.
Transport to and from the query is in binary and dynamically determined
per datatype, no quoting necessary.

Anyone interested in taking a peek at the (GPL copyright) driver, I
temporarily put up a small package which contains the working driver
in Pike at:

http://admin.cuci.nl/psgsql.pike.tar.gz

Pike is a C/C++/Java like interpreted language.
The production driver uses a PGsql assist class which is written in C to
accelerate (amazingly) few core functions (not included, but the driver
works fully without the PGsql assist class).
--
Sincerely,
Stephen R. van den Berg.
"There are 10 types of people in the world.
Those who understand binary and those who do not."

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [pgsql-www] Insecure DNS servers on PG infrastructure

"Joshua D. Drake" <jd@commandprompt.com> writes:
> Tom can you check if this has been resolved? If not I am going to start
> paging people.

The machines I have access to (cvs and svr1) are still unpatched
according to the "dig porttest" test. Personally, though, I'd be
more interested in getting someone to work on fixing the mailing
lists. I've received nothing since about midnight EDT, and I know
that there has been traffic.

regards, tom lane

--
Sent via pgsql-www mailing list (pgsql-www@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

Re: [pgsql-www] Insecure DNS servers on PG infrastructure

Tom Lane wrote:
> Andrew Sullivan <ajs@commandprompt.com> writes:
>> On Fri, Jul 25, 2008 at 11:02:03AM -0400, Tom Lane wrote:
>>> If it says FAIR or POOR then you have an unpatched server or there
>>> is something interfering with the port randomization. If the server
>>> is behind a NAT firewall then the latter is entirely likely.
>
>> There's no reason that a NAT should do that, if the device is
>> competently built: if you randomise source ports on the inside, the
>> NAT device could just use the same port on the outside.


Tom can you check if this has been resolved? If not I am going to start
paging people.

Joshua D. Drake

--
Sent via pgsql-www mailing list (pgsql-www@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

[COMMITTERS] pgloader - pgloader: some more facilities in the TODO list

Log Message:
-----------
some more facilities in the TODO list

Modified Files:
--------------
pgloader:
TODO.txt (r1.7 -> r1.8)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgloader/pgloader/TODO.txt.diff?r1=1.7&r2=1.8)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[GENERAL] How to get the real postgreql error from visual basic

Dear Sirs,

when I execute a function that returns an error, visual basic shows always
the same error code ( -214767259) but I would like to know the real postgres
code of the error.

The visual basic code that I use is the following:

Dim Cmd As new ADODB.Command
Cmd.CommandText = "delete from <tablename> where id=<some_number>;"
Cmd.ActiveConnection = mvarConnection
Cmd.Execute

Whichever error appens the visual basic Err object returns the same number.


What I have to do?

Any suggestion will be appreciated.

Domenico


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [PATCHES] pg_dump additional options for performance

Simon Riggs wrote:
> On Sat, 2008-07-26 at 11:03 -0700, Joshua D. Drake wrote:
>
>> 2. We have no concurrency which means, anyone with any database over 50G
>> has unacceptable restore times.
>
> Agreed.

> Sounds good.
>
> Doesn't help with the main element of dump time: one table at a time to
> one output file. We need a way to dump multiple tables concurrently,
> ending in multiple files/filesystems.

Agreed but that is a problem I understand with a solution I don't. I am
all eyes on a way to fix that. One thought I had and please, be gentle
in response was some sort of async transaction capability. I know that
libpq has the ability to send async queries. Is it possible to do this:

send async(copy table to foo)
send async(copy table to bar)
send async(copy table to baz)

Where all three copies are happening in the background?

Sincerely,

Joshua D. Drake


--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

[COMMITTERS] pgscript - pgScript: Global script updates

Log Message:
-----------
Global script updates

Modified Files:
--------------
pgScript/lib:
parser.sh (r1.2 -> r1.3)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/lib/parser.sh.diff?r1=1.2&r2=1.3)
pgadmin.sh (r1.5 -> r1.6)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/lib/pgadmin.sh.diff?r1=1.5&r2=1.6)
pgScript/src:
pgsMain.cpp (r1.4 -> r1.5)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/src/pgsMain.cpp.diff?r1=1.4&r2=1.5)
pgScript:
autogen.sh (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/autogen.sh.diff?r1=1.3&r2=1.4)
autoclean.sh (r1.6 -> r1.7)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/autoclean.sh.diff?r1=1.6&r2=1.7)
pgScript/files/tests:
execute.sh (r1.1 -> r1.2)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/files/tests/execute.sh.diff?r1=1.1&r2=1.2)
pgScript/doc:
generate.sh (r1.4 -> r1.5)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/doc/generate.sh.diff?r1=1.4&r2=1.5)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [lapug] August's LAPUG meeting location

There is a Dennys in El Monte with a nice back room. No internet though.
I don't know a location in SGV works for everyone. :)
There is also a Dennys in Van Nuys that has a back room, and one in
Northridge.

So if people would like to meet in the SGV or SFV then I am happy to
provide further details and can even make the reservation.

Nice thing about Dennys is that it has food. Granted it may not be the
best food, but it's something of a variety and everyone can
buy there own.

Charles


Richard Broersma wrote:
> I wanted to start a discussion regarding where we will meet in August.
>
> Would anyone like to volunteer a location?
>
>


--
Charles Wyble (818) 280 - 7059
http://charlesnw.blogspot.com
CTO Known Element Enterprises / SoCal WiFI project


--
Sent via lapug mailing list (lapug@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/lapug

[pgadmin-hackers] pgScript patch

Hi pgadmin hackers,

pgScript can now be integrated into pgAdmin3. I have made a patch on
revision 7394 of pgAdmin. This patch is big therefore I do not post it
in this email, it is instead available on the following server:
http://pgscript.projects.postgresql.org/pgadmin

This patch works on a fresh copy of revision 7394. I tested it on:
* Linux Slackware / g++ 4.2.3
* Windows XP / VC++ Express 2005

It adds a pgScript button and a menu entry in the SQL Query Tool. When
one clicks on this button, the content of the text box is sent to the
pgScript engine instead of PostgreSQL. The pgScript syntax is detailed
on this page:
http://pgscript.projects.postgresql.org/SCRIPT.html

That's it. I hope to have feedbacks.

Best regards,
Mickael

--
Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers

[SQL] Re: Efficiently determining the number of bits set in the contents of, a VARBIT field

I use a c function, nbits_set that will do what you need.
I've posted the code in this email.

TJ O'Donnell
http://www.gnova.com

#include "postgres.h"
#include "utils/varbit.h"

Datum nbits_set(PG_FUNCTION_ARGS);
PG_FUNCTION_INFO_V1(nbits_set);
Datum
nbits_set(PG_FUNCTION_ARGS)
{
/* how many bits are set in a bitstring? */

VarBit *a = PG_GETARG_VARBIT_P(0);
int n=0;
int i;
unsigned char *ap = VARBITS(a);
unsigned char aval;
for (i=0; i < VARBITBYTES(a); ++i) {
aval = *ap; ++ap;
if (aval == 0) continue;
if (aval & 1) ++n;
if (aval & 2) ++n;
if (aval & 4) ++n;
if (aval & 8) ++n;
if (aval & 16) ++n;
if (aval & 32) ++n;
if (aval & 64) ++n;
if (aval & 128) ++n;
}
PG_RETURN_INT32(n);
}

> Hi all,
> Am looking for a fast and efficient way to count the number of bits set
> (to 1) in a VARBIT field. I am currently using
> "LENGTH(REGEXP_REPLACE(CAST(a.somefield_bit_code AS TEXT),'0','','g'))".
>
> Allan.


--
Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-sql

[GENERAL] Profiling postgres


Hello,

I want to profile some functions in postgres using gprof. I am following this procedure listed in nabble forum.

# cd postgres/bin
#  cd .../src/backend
#  make clean
#  make PROFILE="-pg" all

How to proceed after this ? I am not getting the gmon.out file. How do I run ./postgres test under the profiler ?

thanks and regards,
Suresh



[COMMITTERS] pgscript - pgScript: autofind deletion and documentation update

Log Message:
-----------
autofind deletion and documentation update

Modified Files:
--------------
pgScript/doc:
INSTALL.xml (r1.2 -> r1.3)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/doc/INSTALL.xml.diff?r1=1.2&r2=1.3)
DEVELOPERS.xml (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/doc/DEVELOPERS.xml.diff?r1=1.3&r2=1.4)

Removed Files:
-------------
pgScript/etc:
autofind.sh
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgscript/pgScript/etc/autofind.sh)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[pgsql-fr-generale] Petit conseil

Bonjour à tous,

Je suis la liste depuis quelques mois maintenant mais ceci est mon
premier message alors je vais vite fait me présenter:
Je me nomme Thibaut, je suis étudiant en informatique à supinfo (oué je
sais moyen) sur le récent site de Limoges et sinon je vis sur Sarlat en
Dordogne.
Je suis intéresse par tout ce qui est base de donnée et j'essaye
actuellement d'avoir des notions de bases sur les différents système
utilisées (MySQL, Oracle, Postgresql biensur, mais aussi SQL server, DB2
et autres...)

J'aurai bien aimé être présent lors des RMLL mais le programme ne me
plaisait que moyennement.
Et je ne pourrai malheureusement pas être présent lors de la PGDay car
je serai en stage en Eure-et-Loire à coder des script SQL et PL/SQL sur
les serveurs Oracle de chez MMA.

Je me suis donc intéressé à Postgresql car après quelques lectures sur
le net, j'ai vu que les performances de celui si pouvais rivaliser avec
Oracle et sans doute donc dépasser les limitations de MySQL que je
rencontre régulièrement en codant quelques sites relativement volumineux
en donnés.

J'en viens donc à me pauser quelques questions auxquelles je ne peux pas
encore répondre seul:

Pourquoi PostgreSQL plutôt que MySQL? (je suppose que ça reviens presque
au même pourquoi Oracle, hormis la question du prix et de la licence)

De la même manière quelle volume peut supporter Postgresql? Est-il
gourmand en ressources?

Enfin, pour les sauvegarde, comment ça se passe avec Postgresql?

Merci pour vos conseil.
Thibaut


--
Sent via pgsql-fr-generale mailing list (pgsql-fr-generale@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-fr-generale

Re: [GENERAL] procedure to load xml file data in postgesql

On Sat, Jul 26, 2008 at 02:32:05PM -0700,
aravind chandu <avin_friends@yahoo.com> wrote
a message of 149 lines which said:

> I have to load xml file data into postgresql database
> table using a stored procedure,but I didn't have any
> idea how to start it.

Well, the problem is much too open to provide any ready-to-use
solution. The way you describe it, it looks like a school
assignment. Is it so?

First, you need to decide what the XML data will look like in
PostgreSQL:

* you can slurp the entire file in a TEXT field (the simplest
solution),

* you can convert it to relational data (the schema conversion is not
obvious because XML data model is hierarchical, not relational, but
this solution will give you "nice" SQL data; your actual data look
quite tabular and therefore will fit well in a relational schema)

* you can use PostgreSQL native XML facilities (I cannot help, I've
never used them).
<http://developer.postgresql.org/pgdocs/postgres/datatype-xml.html>
and <http://developer.postgresql.org/pgdocs/postgres/functions-xml.html>

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [PATCHES] pg_dump additional options for performance

On Sat, 2008-07-26 at 11:03 -0700, Joshua D. Drake wrote:

> 2. We have no concurrency which means, anyone with any database over 50G
> has unacceptable restore times.

Agreed.

Also the core reason for wanting -w

> 3. We have to continue develop hacks to define custom utilization. Why
> am I passing pre-data anything? It should be automatic. For example:
>
> pg_backup (not dump, we aren't dumping. Dumping is usually associated
> with some sort of crash or fould human behavoir. We are backing up).
> pg_backup -U <user> -D database -F -f mybackup.sqlc
>
> If I were to extract <mybackup.sqlc> I would get:
>
> mybackup.datatypes
> mybackup.tables
> mybackup.data
> mybackup.primary_keys
> mybackup.indexes
> mybackup.constraints
> mybackup.grants

Sounds good.

Doesn't help with the main element of dump time: one table at a time to
one output file. We need a way to dump multiple tables concurrently,
ending in multiple files/filesystems.

> Oh and pg_dumpall? It should have been removed right around the release
> of 7.2, pg_dump -A please.

Good idea

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Re: [PATCHES] pg_dump additional options for performance

On Sat, 2008-07-26 at 13:56 -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > I want to dump tables separately for performance reasons. There are
> > documented tests showing 100% gains using this method. There is no gain
> > adding this to pg_restore. There is a gain to be had - parallelising
> > index creation, but this patch doesn't provide parallelisation.
>
> Right, but the parallelization is going to happen sometime, and it is
> going to happen in the context of pg_restore.

I honestly think there is less benefit that way than if we consider
things more as a whole:

To do data dump quickly we need to dump different tables to different
disks simultaneously. By its very nature, that cannot end with just a
single file. So the starting point for any restore must be potentially
more than one file.

There are two ways of dumping: either multi-thread pg_dump, or allow
multiple pg_dumps to work together. Second option much less work, same
result. (Either way we also need a way for multiple concurrent sessions
to share a snapshot.)

When restoring, we can then just use multiple pg_restore sessions to
restore the individual data files. Or again we can write a
multi-threaded pg_restore to do the same thing - why would I bother
doing that when I already can? It gains us nothing.

Parallelising the index creation seems best done using concurrent psql.
We've agreed some mods to psql to put multi-sessions in there. If we do
that right, then we can make pg_restore generate a psql script with
multi-session commands scattered appropriately throughout.

Parallel pg_restore is a lot of work for a narrow use case. Concurrent
psql provides a much wider set of use cases.

So fully parallelising dump/restore can be achieved by

* splitting dump into pieces (this patch)
* allowing sessions to share a common snapshot
* concurrent psql
* changes to pg_restore/psql/pg_dump to allow commands to be inserted
which will use concurrent psql features

If we do things this way then we have some useful tools that can be used
in a range of use cases, not just restore.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches