Saturday, July 19, 2008

Re: [pgsql-fr-generale] Probleme Traffic réseau PostgreSQL

Antony Resbeut wrote:

> Lorsque je fais un "select toto from table where titi='x'" alors le
nombre
> d'octet de la réponse est identique entre Oracle et PostgreSQL
(parfois
> meilleur sous PostgreSQL).
>
> Si j'enlève la condition "where" alors PostgreSQL est beaucoup plus
bavard
> qu'Oracle aussi bien en nombre de packet que sur la taille des
packets. Et
> c'est pire si l'on fais un vidage "select * from table" brutal.
> Est-ce normal ?

La différence de taille peut s'expliquer par le fait que les nombres et
les
dates transitent par défaut en ASCII avec Postgres et probablement en
binaire
avec Oracle.
Réduire le trafic réseau avec Postgres doit être possible en passant
par des
curseurs binaires au lieu d'un select simple. Encore qu'au final ça
dépendra du
contenu effectif des données. puisque par exemple les petits nombres
prennent
moins de place en ASCII et les grands moins de place en binaire.

--
Daniel
PostgreSQL-powered mail user agent and storage:
http://www.manitou-mail.org

--
Sent via pgsql-fr-generale mailing list (pgsql-fr-generale@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-fr-generale

Re: [PERFORM] Performance on Sun Fire X4150 x64 (dd, bonnie++, pgbench)

pgbench is unrelated to the workload you are concerned with if ETL/ELT and decision support / data warehousing queries are your target.

Also - placing the xlog on dedicated disks is mostly irrelevant to data warehouse / decision support work or ELT.  If you need to maximize loading speed while concurrent queries are running, it may be necessary, but I think you'll be limited in load speed by CPU related to data formatting anyway.

The primary performance driver for ELT / DW is sequential transfer rate, thus the dd test at 2X memory.  With six data disks of this type, you should expect a maximum of around 6 x 80 = 480 MB/s.  With RAID10, depending on the raid adapter, you may need to have two or more IO streams to use all platters, otherwise your max speed for one query would be 1/2 that, or 240 MB/s.

I'd suggest RAID5, or even better, configure all eight disks as a JBOD in the RAID adapter and run ZFS RAIDZ.  You would then expect to get about 7 x 80 = 560 MB/s on your single query.

That said, your single cpu on one query will only be able to scan that data at about 300 MB/s (try running a SELECT COUNT(*) against a table that is 2X memory size).

- Luke

----- Original Message -----
From: pgsql-performance-owner@postgresql.org <pgsql-performance-owner@postgresql.org>
To: pgsql-performance@postgresql.org <pgsql-performance@postgresql.org>
Sent: Sat Jul 19 09:19:43 2008
Subject: [PERFORM] Performance on Sun Fire X4150 x64 (dd, bonnie++, pgbench)


I'm trying to run a few basic tests to see what a current machine can
deliver (typical workload ETL like, long running aggregate queries,
medium size db ~100 to 200GB).

I'm currently checking the system (dd, bonnie++) to see if performances
are within the normal range but I'm having trouble relating it to
anything known. Scouting the archives there are more than a few people
familiar with it, so if someone can have a look at those numbers and
raise a flag where some numbers look very out of range for such system,
that would be appreciated. I also added some raw pgbench numbers at the end.

(Many thanks to Greg Smith, his pages was extremely helpful to get
started. Any mistake is mine)

Hardware:

Sun Fire X4150 x64

2 Quad-Core Intel(R) Xeon(R) X5460 processor (2x6MB L2, 3.16 GHz, 1333
MHz FSB)
16GB of memory (4x2GB PC2-5300 667 MHz ECC fully buffered DDR2 DIMMs)

6x 146GB 10K RPM SAS  in RAID10 - for os + data
2x 146GB 10K RPM SAS  in RAID1 - for xlog
Sun StorageTek SAS HBA Internal (Adaptec AAC-RAID)


OS is Ubuntu 7.10 x86_64 running  2.6.22-14
os in on ext3
data is on xfs noatime
xlog is on ext2 noatime


data
$ time sh -c "dd if=/dev/zero of=bigfile bs=8k count=4000000 && sync"
4000000+0 records in
4000000+0 records out
32768000000 bytes (33 GB) copied, 152.359 seconds, 215 MB/s

real    2m36.895s
user    0m0.570s
sys     0m36.520s

$ time dd if=bigfile of=/dev/null bs=8k
4000000+0 records in
4000000+0 records out
32768000000 bytes (33 GB) copied, 114.723 seconds, 286 MB/s

real    1m54.725s
user    0m0.450s
sys     0m22.060s


xlog
$ time sh -c "dd if=/dev/zero of=bigfile bs=8k count=4000000 && sync"
4000000+0 records in
4000000+0 records out
32768000000 bytes (33 GB) copied, 389.216 seconds, 84.2 MB/s

real    6m50.155s
user    0m0.420s
sys     0m26.490s

$ time dd if=bigfile of=/dev/null bs=8k
4000000+0 records in
4000000+0 records out
32768000000 bytes (33 GB) copied, 294.556 seconds, 111 MB/s

real    4m54.558s
user    0m0.430s
sys     0m23.480s



bonnie++ -s 32g -n 256

data:
Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
lid-statsdb-1   32G 101188  98 202523  20 107642  13 88931  88 271576 
19 980.7   2
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                256 11429  93 +++++ +++ 17492  71 11097  91 +++++ +++ 
2473  11



xlog
Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
lid-statsdb-1   32G 62973  59 69981   5 35433   4 87977  85 119749   9
496.2   1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                256   551  99 +++++ +++ 300935  99   573  99 +++++ +++ 
1384  99

pgbench

postgresql 8.2.9 with data and xlog as mentioned above

postgresql.conf:
shared_buffers = 4GB
checkpoint_segments = 8
effective_cache_size = 8GB

Script running over scaling factor 1 to 1000 and running 3 times pgbench
with "pgbench -t 2000 -c 8 -S pgbench"

It's a bit limited and will try to do a much much longer run and
increase the # of tests and calculate mean and stddev as I have a pretty
large variation for the 3 runs sometimes (typically for the scaling
factor at 1000, the runs are respectively 1952, 940, 3162)  so the graph
is pretty ugly.

I get (scaling factor, size of db in MB, middle tps)

1 20 22150
5 82 22998
10 160 22301
20 316 22857
30 472 23012
40 629 17434
50 785 22179
100 1565 20193
200 3127 23788
300 4688 15494
400 6249 23513
500 7810 18868
600 9372 22146
700 11000 14555
800 12000 10742
900 14000 13696
1000 15000 940

cheers,

-- stephane

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: [pgsql-fr-generale] Re: Réf. : [pgsql-fr-generale] Probleme Traffic réseau PostgreSQL

Jonathan Ballet a écrit :
> Bonjour,
>
> Le Sat, 19 Jul 2008 10:27:10 +0200, Guillaume Lelarge <guillaume@lelarge.info> a écrit :
>
>>> WireShark est vraiment génial, puisqu'il sait formater en clair les
>>> messages du protocole PostgreSQL :-)
>> Tout comme le fait Etherreal à ma connaissance. Mais j'avoue encore une
>> fois n'avoir jamais eu la curiosité de regarder ce que cela donnait.
>
> C'est normal, WireShark est le nouveau nom d'Ethereal :)
> (cf. http://www.wireshark.org/faq.html#q1.2 )
>

Oups... je ne suis pas trop l'actualité de ce soft. Quoi, ça s'est vu ? :)

Merci pour l'info.


--
Guillaume.
http://www.postgresqlfr.org
http://dalibo.com

--
Sent via pgsql-fr-generale mailing list (pgsql-fr-generale@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-fr-generale

Re: [GENERAL] Initdb problem on debian mips cobalt: Bus error

> From: Stefan Kaltenbrunner <stefan@kaltenbrunner.cc>

> Tom Lane wrote:
> > Glyn Astill writes:
> >> No. Will recompile with debug info and post back when done.
> >
> > FWIW, the most likely issue here is the MIPS-specific assembly code in
> > src/include/storage/s_lock.h --- I'm not sure how many MIPS platforms
> > that's really been exercised on, but it may not work on yours. While
> > you're waiting for the rebuild you might try to find a MIPS guru to
> > show that code to.
>
> hmm well - lionfish (which is now offline due to a broken power supply)
> is actually a cobalt cube too (and is running debian). So if we really
> managed to break mipsel it must have happened in the last few months:
>
>
> http://www.pgbuildfarm.org/cgi-bin/show_history.pl?nm=lionfish&br=HEAD
>
>
> Stefandumb here

Hmm, well I've still not ruled out the possibility that I've done something stupid yet. Also I see that lionfish is running sarge, and I'm running etch.

Once I get time to recompile I'll try and find out a little more.

Would the mips specific code behave differently on different oses? My other qube is running netbsd4, I'd be interested to see what goes off on that, however I'ts my live email/web/everything server and I didn't want to upset it.


__________________________________________________________
Not happy with your email address?.
Get the one you really want - millions of new email addresses available now at Yahoo! http://uk.docs.yahoo.com/ymail/new.html

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [ADMIN] Query

On Sat, Jul 19, 2008 at 04:29:00PM +0530, Kartik wrote:
> hello there,i am new to postgresql
> i am using postgresql 8.3.3 and i am writing one whole transaction. i want
> to know how to set auto commit off
> because if any error occours i want the whole transaction to be rolled back.
> when i tried alter database dbname set autocommit = off
> then it gives me error saying autocommit not available
> so can anyone suggest how to set autocommit as off so that i can rollback
> the transaction..

Simply issue the command "BEGIN" at the start of your transaction.
Then nothing will be commited until you issue "COMMIT" and you may abort
the transaction by sending "ROLLBACK" (or "ABORT").

HTH,

Tino.

--
"What we nourish flourishes." - "Was wir nähren erblüht."

www.craniosacralzentrum.de
www.forteego.de

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

[PERFORM] Performance on Sun Fire X4150 x64 (dd, bonnie++, pgbench)

I'm trying to run a few basic tests to see what a current machine can
deliver (typical workload ETL like, long running aggregate queries,
medium size db ~100 to 200GB).

I'm currently checking the system (dd, bonnie++) to see if performances
are within the normal range but I'm having trouble relating it to
anything known. Scouting the archives there are more than a few people
familiar with it, so if someone can have a look at those numbers and
raise a flag where some numbers look very out of range for such system,
that would be appreciated. I also added some raw pgbench numbers at the end.

(Many thanks to Greg Smith, his pages was extremely helpful to get
started. Any mistake is mine)

Hardware:

Sun Fire X4150 x64

2 Quad-Core Intel(R) Xeon(R) X5460 processor (2x6MB L2, 3.16 GHz, 1333
MHz FSB)
16GB of memory (4x2GB PC2-5300 667 MHz ECC fully buffered DDR2 DIMMs)

6x 146GB 10K RPM SAS in RAID10 - for os + data
2x 146GB 10K RPM SAS in RAID1 - for xlog
Sun StorageTek SAS HBA Internal (Adaptec AAC-RAID)


OS is Ubuntu 7.10 x86_64 running 2.6.22-14
os in on ext3
data is on xfs noatime
xlog is on ext2 noatime


data
$ time sh -c "dd if=/dev/zero of=bigfile bs=8k count=4000000 && sync"
4000000+0 records in
4000000+0 records out
32768000000 bytes (33 GB) copied, 152.359 seconds, 215 MB/s

real 2m36.895s
user 0m0.570s
sys 0m36.520s

$ time dd if=bigfile of=/dev/null bs=8k
4000000+0 records in
4000000+0 records out
32768000000 bytes (33 GB) copied, 114.723 seconds, 286 MB/s

real 1m54.725s
user 0m0.450s
sys 0m22.060s


xlog
$ time sh -c "dd if=/dev/zero of=bigfile bs=8k count=4000000 && sync"
4000000+0 records in
4000000+0 records out
32768000000 bytes (33 GB) copied, 389.216 seconds, 84.2 MB/s

real 6m50.155s
user 0m0.420s
sys 0m26.490s

$ time dd if=bigfile of=/dev/null bs=8k
4000000+0 records in
4000000+0 records out
32768000000 bytes (33 GB) copied, 294.556 seconds, 111 MB/s

real 4m54.558s
user 0m0.430s
sys 0m23.480s

bonnie++ -s 32g -n 256

data:
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
lid-statsdb-1 32G 101188 98 202523 20 107642 13 88931 88 271576
19 980.7 2
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
256 11429 93 +++++ +++ 17492 71 11097 91 +++++ +++
2473 11

xlog
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
lid-statsdb-1 32G 62973 59 69981 5 35433 4 87977 85 119749 9
496.2 1
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
256 551 99 +++++ +++ 300935 99 573 99 +++++ +++
1384 99

pgbench

postgresql 8.2.9 with data and xlog as mentioned above

postgresql.conf:
shared_buffers = 4GB
checkpoint_segments = 8
effective_cache_size = 8GB

Script running over scaling factor 1 to 1000 and running 3 times pgbench
with "pgbench -t 2000 -c 8 -S pgbench"

It's a bit limited and will try to do a much much longer run and
increase the # of tests and calculate mean and stddev as I have a pretty
large variation for the 3 runs sometimes (typically for the scaling
factor at 1000, the runs are respectively 1952, 940, 3162) so the graph
is pretty ugly.

I get (scaling factor, size of db in MB, middle tps)

1 20 22150
5 82 22998
10 160 22301
20 316 22857
30 472 23012
40 629 17434
50 785 22179
100 1565 20193
200 3127 23788
300 4688 15494
400 6249 23513
500 7810 18868
600 9372 22146
700 11000 14555
800 12000 10742
900 14000 13696
1000 15000 940

cheers,

-- stephane

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: [GENERAL] Reducing memory usage of insert into select operations? [Solved]

Martijn van Oosterhout wrote:
> Can you make them not deferred?
How?


I found the issue.
I had the foreign key in the master table instead of the children.
Deleted RI from master table and put into the inherited partitions.
My whole 230 million rows merged in about an hour!
And I even had two of those running at the same time. (one setup with 14
partitions per month and another with 5 partitions per month to test
difference in performance).

It was so fast I even had to do a count(*) to make sure both actually
merged.
That is 117K rows per second for rows that were about 33 bytes long.
That only comes down to about 3 MB/sec+overhead, but still 117K rows/sec
is not too shabby.

In case it is of interest to anyone..
2 AMD dual core, 2GHz CPUs
12GB of RAM
shared_buffers 3GB
work_mem 64MB
256 check_point segments
10 min checkpoing_timeout
LSI controller with 128MB cache with BBU. Write cache enabled.


Many thanks to all that offered suggestions in the troubleshooting.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] gsoc, oprrest function for text search

Jan Urbański wrote:
> The idea is (quoting a comment)
> /*
> * Traverse the tsquery preorder, calculating selectivity as:

Ekhm.
This should of course read "postorder"...

--
Jan Urbanski
GPG key ID: E583D7D2

ouden estin


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Getting to universal binaries for Darwin

Tom Lane wrote:
> The bad news is that if you only do that, only the arch that you
> actually build on will work. We have configure set up to insert
> various hardware-dependent definitions into pg_config.h and
> ecpg_config.h, and if you don't have the right values visible for
> each compilation, the resulting executables will fail.
>
> You can get around that by hacking up the generated config files
> with #ifdef __i386__ and so on to expose the correct values of
> the hardware-dependent symbols to each build. Of course you have
> to know what the correct values are --- if you don't have a sample
> of each architecture handy to run configure against, it'd be easy
> to miss some things. And even then it's pretty tedious. I am
> not sure if it is possible or worth the trouble to try to automate
> this part better.

It may be less pain to simply config and build for ppc and i386 in separate build directories and
then glue the resulting binaries together with lipo
<http://developer.apple.com/documentation/Darwin/Reference/ManPages/man1/lipo.1.html> to make them
"universal".

Regards,

Adriaan van Os


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Reducing memory usage of insert into select operations?

On Fri, Jul 18, 2008 at 04:48:26PM -0400, Francisco Reyes wrote:
> On 3:55 pm 07/18/08 Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > > AfterTriggerEvents: 10553909248 total in 1268 blocks; 20432 free (6
> > > chunks); 10553888816 used
> >
> > Well, that's definitely your problem ...
>
> What is the overhead for each AfterTriggerEvent?

Can you make them not deferred? Then you don't need the memory either.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
> Please line up in a tree and maintain the heap invariant while
> boarding. Thank you for flying nlogn airlines.

[HACKERS] gsoc, oprrest function for text search

Here's a WIP patch implementing an oprrest function for tsvector @@
tsquery and tsquery @@ tsvector.

The idea is (quoting a comment)
/*
* Traverse the tsquery preorder, calculating selectivity as:
*
* selec(left_oper) * selec(right_oper) in AND nodes,
*
* selec(left_oper) + selec(right_oper) -
* selec(left_oper) * selec(right_oper) in OR nodes,
*
* 1 - select(oper) in NOT nodes
*
* freq[val] in VAL nodes, if the value is in MCELEM
* min(freq[MCELEM]) / 2 in VAL nodes, if it is not
*
*
* Implementation-wise, we sort the MCELEM array to use binary
* search on it.
*/

The patch still has many rough edges, but it applies to HEAD and passes
tests. I'm posting it mostly to get feedback about whether I'm going in
the right direction.

Cheers,
Jan

--
Jan Urbanski
GPG key ID: E583D7D2

ouden estin

[ADMIN] Query

hello there,
i am new to postgresql
i am using postgresql 8.3.3 and i am writing one whole transaction. i want to know how to set auto commit off
because if any error occours i want the whole transaction to be rolled back.
when i tried alter database dbname set autocommit = off
then it gives me error saying autocommit not available
so can anyone suggest how to set autocommit as off so that i can rollback the transaction..
thank you
waiting for your reply
regards 
kartik

Re: [pgsql-fr-generale] Re: Réf. : [pgsql-fr-generale] Probleme Traffic réseau PostgreSQL

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkiBsiUACgkQ+cFYTXJHUVBtNQCgiHHRAMWfEVSxZB+HlT5CJIaU
ZkoAmwZxwNLa+WQVkaH6Lc/zpiLOwGaH
=Sk6T
-----END PGP SIGNATURE-----
Bonjour,

Le Sat, 19 Jul 2008 10:27:10 +0200, Guillaume Lelarge <guillaume@lelarge.info> a écrit :

> > WireShark est vraiment génial, puisqu'il sait formater en clair les
> > messages du protocole PostgreSQL :-)
>
> Tout comme le fait Etherreal à ma connaissance. Mais j'avoue encore une
> fois n'avoir jamais eu la curiosité de regarder ce que cela donnait.

C'est normal, WireShark est le nouveau nom d'Ethereal :)
(cf. http://www.wireshark.org/faq.html#q1.2 )

Mes 2 centimes,

Jonathan

Re: [GENERAL] Backup/Restore of single table in multi TB database

On Fri, 2008-07-18 at 20:25 -0400, Francisco Reyes wrote:

> Does pg_snapclone works mostly on large rows or will it also be faster
> than pg_dump for narrow tables?

It allows you to run your dump in multiple pieces. Thats got nothing to
do with narrow or wide.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [ADMIN] unrecognized data type on dblink

I use 8.03.02.00 version driver.

> On Fri, Jul 18, 2008 at 9:24 AM, Aynur SANCAKLI
> <aynur.sancakli@kamusm.gov.tr> wrote:
>> Hi all,
>> I have a problem with dblink from oracle to postgresql.
>>
>> I cannot read varchar colums from postgresql db, in the trace file the
>> message is in the trace file is : unrecognized data type.
>
> Which Postgres ODBC driver are you using with heterogeneous services?
>
> --
> Jonah H. Harris, Sr. Software Architect | phone: 732.331.1324
> EnterpriseDB Corporation | fax: 732.331.1301
> 499 Thornall Street, 2nd Floor | jonah.harris@enterprisedb.com
> Edison, NJ 08837 | http://www.enterprisedb.com/
>

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [pgadmin-hackers] Dialogs review

Dave Page a écrit :
> On Mon, Jul 14, 2008 at 8:17 PM, Guillaume Lelarge
> <guillaume@lelarge.info> wrote:
>
>> Hmmmm, I see... that I can't do anything till I get my Mac. I will work on
>> it but I don't know now how to fix it.
>
> Understood. I'm really busy right now (yeah, I know I'm starting to
> sound like a broken record with that one!) but if you send over the
> latest version of the patch sometime I'll try out a couple of ideas
> for this and the combo box thing when I can get five minutes..
>

:)

Now that I have a MacMini, that I'm able to build a pgAdmin3.app file,
things should go faster and with less burden. Or so I hope.

BTW, I have the same issue with the two dialogs. I'm statring to work on
this.


--
Guillaume.
http://www.postgresqlfr.org
http://dalibo.com

--
Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers

[pgsql-fr-generale] Re: Réf. : [pgsql-fr-generale] Probleme Traffic réseau PostgreSQL

Bonjour,

philippe.beaudoin@bull.net a écrit :
> [...]
> Comme il n'y avait pas beaucoup de réponse à ton post, j'ai pris mon
> courage à 2 mains et ai fait quelques analyses.

Le manque de réponses me semble dû au fait que personne ne s'est jamais
posé une telle question. Je pense bien que ça a déjà dû traverser
l'esprit de quelqu'un. Mais je ne me rappelle pas avoir lu une
quelconque étude ou mail à ce sujet ici et sur les listes anglophones.

> Pour ne pas avoir à me
> perdre dans les sources de PostgreSQL, j'ai installé WireShark sur mon
> micro et j'ai regardé les messages échangés entre pgAdmin sur mon micro et
> un PostgreSQL sur un serveur Linux.
> WireShark est vraiment génial, puisqu'il sait formater en clair les
> messages du protocole PostgreSQL :-)

Tout comme le fait Etherreal à ma connaissance. Mais j'avoue encore une
fois n'avoir jamais eu la curiosité de regarder ce que cela donnait.

> J'en ai déduit ceci sur la taille des messages échangés (hors couches
> réseau au dessous) :
>
> 1) A l'aller, on a le texte de la requête avec simplement un overhead de 6
> octets.
>
> 2) Au retour, on a un message (découpé éventuellement en plusieurs paquets
> selon la taille) composé de :
> - la description des lignes de la table résultat, dont la longueur en
> octets est égale à :
> 7 + ( 19 * nombre de colonnes) + somme des longueurs des noms des
> colonnes,
> - les lignes de données (voir plus bas),
> - 2 commandes "command completion" et "ready for query" représentant 18
> octets.
> Chaque ligne se compose de :
> - une partie fixe de 7 octets,
> - la liste des colonnes.
> Chaque colonne se compose de :
> - une partie fixe de 4 octets,
> - la valeur de la colonne.
>
> Tout est donc très logique.
>

Oui, en effet.

> 3) Contenu des colonnes :
> Pour les colonnes de type CHAR, on trouve le contenu de la colonne (je n'ai
> pas fait attention aux questions d'encodage qui peuvent j'imagine avoir un
> impact sur la taille physique de chaque caractère). il n'y a pas de
> compression, même pour les blancs à droite.

Exact pour l'encodage, la taille de chaque caractère en dépend. Quant à
la compression, je sais que PostgreSQL stocke parfois en compressant les
données. Pour ce qui est de l'envoi sur le réseau, je n'en sais rien du
tout.

> Pour les colonnes de type VARCHAR, on trouve évidemment la longueur réelle
> du contenu de la colonne et non sa longueur maximale.

Sa longueur maximale se trouve déjà dans la description des colonnes
envoyée au message précédent, non ?

> Pour les colonnes BYTEA, les octets "non imprimables" sont représentés par
> leur séquence d'escape (\nnn).
> Pour les colonnes numériques, la donnée est représentée sous forme de texte
> de longueur variable, c'est à dire la suite des chiffres nécessaires à la
> représentation du nombre.
>

Je suppose que, quand vous parlez de colonnes numériques, vous entendez
le type NUMERIC ? et pas les types int4, float, etc. ? si vous ne parlez
que du types NUMERIC, ça ne m'étonne pas, c'est déjà pas stocké comme un
entier/flottant.

> 4) Les requêtes retournant une ligne et celles retournant plusieurs lignes
> semblent avoir la même structure, quelle que soit le nombre de colonnes.
>
> Voici donc ma compréhension de la structure des messages. Elle est
> peut-être approximative mais ça peut aider. Quelqu'un a peut-être des
> précisions ?
>

En dehors de ce que je viens déjà de dire, non :)

> J'imagine que le dialogue entre pgAdmin et PostgreSQL que j'ai observé est
> le même que celui des autres clients et drivers disponibles. Quelqu'un
> peut-il me confirmer ?
>

Oui dans le sens où pgAdmin s'appuie sur la libpq (la bibliothèque C
d'accès au serveur PostgreSQL), donc tout outil basé sur la libpq fera
de même.

> Pour comprendre pourquoi Oracle qui est un peu moins bon sur 1 ligne
> devient meilleur sur plusieurs lignes, il faudrait faire ce même exercice
> avec Oracle.
> A moins que quelqu'un ait une déjà une idée sur la question...
>

Nope, ma connaissance d'Oracle est pratiquement nulle.

Merci pour ces infos. Même si ce n'est pas particulièrement nouveau,
c'est très intéressant à lire.


--
Guillaume.
http://www.postgresqlfr.org
http://dalibo.com

--
Sent via pgsql-fr-generale mailing list (pgsql-fr-generale@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-fr-generale

Re: [PATCHES] Is autovacuum doing a wraparound-avoiding VACUUM?

Index: src/backend/postmaster/autovacuum.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/postmaster/autovacuum.c,v
retrieving revision 1.81
diff -c -r1.81 autovacuum.c
*** src/backend/postmaster/autovacuum.c 17 Jul 2008 21:02:31 -0000 1.81
--- src/backend/postmaster/autovacuum.c 19 Jul 2008 07:58:33 -0000
***************
*** 2657,2664 ****
/* Report the command and possible options */
if (tab->at_dovacuum)
snprintf(activity, MAX_AUTOVAC_ACTIV_LEN,
! "autovacuum: VACUUM%s",
! tab->at_doanalyze ? " ANALYZE" : "");
else
snprintf(activity, MAX_AUTOVAC_ACTIV_LEN,
"autovacuum: ANALYZE");
--- 2657,2665 ----
/* Report the command and possible options */
if (tab->at_dovacuum)
snprintf(activity, MAX_AUTOVAC_ACTIV_LEN,
! "autovacuum: VACUUM%s%s",
! tab->at_doanalyze ? " ANALYZE" : "",
! tab->at_wraparound ? " (to prevent wraparound)" : "");
else
snprintf(activity, MAX_AUTOVAC_ACTIV_LEN,
"autovacuum: ANALYZE");
Index: src/backend/postmaster/autovacuum.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/postmaster/autovacuum.c,v
retrieving revision 1.71.2.4
diff -c -r1.71.2.4 autovacuum.c
*** src/backend/postmaster/autovacuum.c 17 Jul 2008 21:02:41 -0000 1.71.2.4
--- src/backend/postmaster/autovacuum.c 19 Jul 2008 07:58:43 -0000
***************
*** 291,297 ****
static PgStat_StatTabEntry *get_pgstat_tabentry_relid(Oid relid, bool isshared,
PgStat_StatDBEntry *shared,
PgStat_StatDBEntry *dbentry);
! static void autovac_report_activity(VacuumStmt *vacstmt, Oid relid);
static void avl_sighup_handler(SIGNAL_ARGS);
static void avl_sigusr1_handler(SIGNAL_ARGS);
static void avl_sigterm_handler(SIGNAL_ARGS);
--- 291,297 ----
static PgStat_StatTabEntry *get_pgstat_tabentry_relid(Oid relid, bool isshared,
PgStat_StatDBEntry *shared,
PgStat_StatDBEntry *dbentry);
! static void autovac_report_activity(VacuumStmt *vacstmt, Oid relid, bool for_wraparound);
static void avl_sighup_handler(SIGNAL_ARGS);
static void avl_sigusr1_handler(SIGNAL_ARGS);
static void avl_sigterm_handler(SIGNAL_ARGS);
***************
*** 2633,2639 ****
MemoryContextSwitchTo(old_cxt);

/* Let pgstat know what we're doing */
! autovac_report_activity(&vacstmt, relid);

vacuum(&vacstmt, relids, bstrategy, for_wraparound, true);
}
--- 2633,2639 ----
MemoryContextSwitchTo(old_cxt);

/* Let pgstat know what we're doing */
! autovac_report_activity(&vacstmt, relid, for_wraparound);

vacuum(&vacstmt, relids, bstrategy, for_wraparound, true);
}
***************
*** 2650,2656 ****
* bother to report "<IDLE>" or some such.
*/
static void
! autovac_report_activity(VacuumStmt *vacstmt, Oid relid)
{
char *relname = get_rel_name(relid);
char *nspname = get_namespace_name(get_rel_namespace(relid));
--- 2650,2656 ----
* bother to report "<IDLE>" or some such.
*/
static void
! autovac_report_activity(VacuumStmt *vacstmt, Oid relid, bool for_wraparound)
{
char *relname = get_rel_name(relid);
char *nspname = get_namespace_name(get_rel_namespace(relid));
***************
*** 2661,2668 ****
/* Report the command and possible options */
if (vacstmt->vacuum)
snprintf(activity, MAX_AUTOVAC_ACTIV_LEN,
! "autovacuum: VACUUM%s",
! vacstmt->analyze ? " ANALYZE" : "");
else
snprintf(activity, MAX_AUTOVAC_ACTIV_LEN,
"autovacuum: ANALYZE");
--- 2661,2669 ----
/* Report the command and possible options */
if (vacstmt->vacuum)
snprintf(activity, MAX_AUTOVAC_ACTIV_LEN,
! "autovacuum: VACUUM%s%s",
! vacstmt->analyze ? " ANALYZE" : "",
! for_wraparound ? " (to prevent wraparound)" : "");
else
snprintf(activity, MAX_AUTOVAC_ACTIV_LEN,
"autovacuum: ANALYZE");
On Fri, 2008-07-18 at 01:44 -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > On Thu, 2008-07-17 at 17:10 -0400, Alvaro Herrera wrote:
> >> I don't like your wording though; it feels too verbose (and you're
> >> losing the ANALYZE in case it's doing both things). How about
> >>
> >> snprintf(activity, MAX_AUTOVAC_ACTIV_LEN,
> >> "autovacuum: VACUUM%s%s", vac
> >> tab->at_doanalyze ? " ANALYZE" : "",
> >> tab->at_wraparound ? " (wraparound)" : "");
>
> > Yes, looks good.
>
> May I suggest "(to prevent wraparound)" or something like that?
> Otherwise, +1.
>
> >> You're not proposing it for 8.3 right?
>
> > I think I am. It's an important diagnostic for your other fix.
>
> I agree, this is important for visibility into what's happening.
> The string isn't getting translated so I don't see any big downside
> to applying the patch in back branches.

Patches for 8.3 and CVS HEAD.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

[ADMIN] Database Link

Hi all,

Like oracle dblink, is it possible to connect two database's in
greenplum??? If yes please pass the command how to create the dblink
to connect two databases.

Regards

Govindarajan

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Réf. : [pgsql-fr-generale] Probleme Traffic réseau PostgreSQL

Antony,

Comme il n'y avait pas beaucoup de réponse à ton post, j'ai pris mon
courage à 2 mains et ai fait quelques analyses. Pour ne pas avoir à me
perdre dans les sources de PostgreSQL, j'ai installé WireShark sur mon
micro et j'ai regardé les messages échangés entre pgAdmin sur mon micro et
un PostgreSQL sur un serveur Linux.
WireShark est vraiment génial, puisqu'il sait formater en clair les
messages du protocole PostgreSQL :-)
J'en ai déduit ceci sur la taille des messages échangés (hors couches
réseau au dessous) :

1) A l'aller, on a le texte de la requête avec simplement un overhead de 6
octets.

2) Au retour, on a un message (découpé éventuellement en plusieurs paquets
selon la taille) composé de :
- la description des lignes de la table résultat, dont la longueur en
octets est égale à :
7 + ( 19 * nombre de colonnes) + somme des longueurs des noms des
colonnes,
- les lignes de données (voir plus bas),
- 2 commandes "command completion" et "ready for query" représentant 18
octets.
Chaque ligne se compose de :
- une partie fixe de 7 octets,
- la liste des colonnes.
Chaque colonne se compose de :
- une partie fixe de 4 octets,
- la valeur de la colonne.

Tout est donc très logique.

3) Contenu des colonnes :
Pour les colonnes de type CHAR, on trouve le contenu de la colonne (je n'ai
pas fait attention aux questions d'encodage qui peuvent j'imagine avoir un
impact sur la taille physique de chaque caractère). il n'y a pas de
compression, même pour les blancs à droite.
Pour les colonnes de type VARCHAR, on trouve évidemment la longueur réelle
du contenu de la colonne et non sa longueur maximale.
Pour les colonnes BYTEA, les octets "non imprimables" sont représentés par
leur séquence d'escape (\nnn).
Pour les colonnes numériques, la donnée est représentée sous forme de texte
de longueur variable, c'est à dire la suite des chiffres nécessaires à la
représentation du nombre.

4) Les requêtes retournant une ligne et celles retournant plusieurs lignes
semblent avoir la même structure, quelle que soit le nombre de colonnes.

Voici donc ma compréhension de la structure des messages. Elle est
peut-être approximative mais ça peut aider. Quelqu'un a peut-être des
précisions ?

J'imagine que le dialogue entre pgAdmin et PostgreSQL que j'ai observé est
le même que celui des autres clients et drivers disponibles. Quelqu'un
peut-il me confirmer ?

Pour comprendre pourquoi Oracle qui est un peu moins bon sur 1 ligne
devient meilleur sur plusieurs lignes, il faudrait faire ce même exercice
avec Oracle.
A moins que quelqu'un ait une déjà une idée sur la question...

Bon week-end. Philippe.


Stéphane Schildknecht
<stephane.schildknecht@postgr Pour : "'pgsql-fr-generale@postgresql.org'" <pgsql-fr-generale@postgresql.org>
esqlfr.org> cc : antony.resbeut@bull.net
Envoyé par : Objet : [pgsql-fr-generale] Probleme Traffic réseau PostgreSQL
pgsql-fr-generale-owner@postg
resql.org


11/07/2008 00:51

Bonjour,

Je fais suivre un message d'un souscripteur dont les messages n'arrivent
pas jusqu'à la liste... En attendant de comprendre pourquoi ils
n'arrivent pas sur la liste...

Stéphane Schildknecht

###############

Bonjour,

Je fais des tests réseau de comparaison entre Oracle et PostgreSQL et
j'ai des gros écarts...

Lorsque je fais un "select toto from table where titi='x'" alors
le nombre d'octet de la réponse est identique entre Oracle et PostgreSQL
(parfois meilleur sous PostgreSQL).

Si j'enlève la condition "where" alors PostgreSQL est beaucoup plus
bavard qu'Oracle aussi bien en nombre de packet que sur la taille des
packets. Et c'est pire si l'on fais un vidage "select * from table" brutal.
Est-ce normal ?

Pour exemple :
SELECT id_commande FROM COMMANDE (VARCHAR(28))
SGBD |Temps |Octets |Packets size |Avg Mbit/sec |Packets
PostgreSQL|0'00" |1 896 702 |1105 bytes |67.286 |1716
Oracle |0'00" |1 496 151 |846 bytes |26.168 |1768
Informix |0'00" |1 858 862 |999 bytes |22.724 |1859

SELECT date_sign FROM COMMANDE (DATE)
SGBD |Temps |Octets |Packets size |Avg Mbit/sec |Packets
PostgreSQL|0'00" |868 573 |1117 bytes |18.999 |777
Oracle |0'00" |453 126 |876 bytes |8.245 |517
Informix |0'00" |587 446 |1018 bytes |4.882 |577

SELECT id_commande FROM COMMANDE where ID_COMMANDE =
'200706202054510076503-276501' (VARCHAR(28))
SGBD |Temps |Octets |Packets size |Avg Mbit/sec |Packets
PostgreSQL|0'00" |567 |81 bytes |0.024 |7
Oracle |0'00" |975 |139 bytes |0.043 |7
Informix |0'00" |904 |82 bytes |0.046 |11

SELECT * FROM COMMANDE (46255 lignes)

SGBD |Temps |Octets |Packets size |Avg Mbit/sec |Packets
PostgreSQL|0'13" |16 801 515 |1 008 bytes |9.900 |16 653
Oracle |0'08" |06 889 074 |694 bytes |6.752 |09 920
Informix |0'13" |11 481 870 |967 bytes |7.054 |11 867

Merci de votre réponse ou avis.

Antony Resbeut


--
Sent via pgsql-fr-generale mailing list (pgsql-fr-generale@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-fr-generale

--
Sent via pgsql-fr-generale mailing list (pgsql-fr-generale@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-fr-generale

Re: [HACKERS] phrase search

Sushant,

the problem of phrase search not in implementation, but in the theoretical
basis. tsearch is query rich and phrase search should support all query
operations, so we need algebra for query operations. We need more time
to investigate this problem, but just have no spare time for this.
If you are interesting, you might think in this direction.

Oleg

On Sat, 19 Jul 2008, Sushant Sinha wrote:

> I looked at query operators for tsquery and here are some of the new
> query operators for position based queries. I am just proposing some
> changes and the questions I have.
>
> 1. What is the meaning of such a query operator?
>
> foo #5 bar -> true if the document has word "foo" followed by "bar" at
> 5th position.
>
> foo #<5 bar -> true if document has word "foo" followed by "bar" with in
> 5 positions
>
> foo #>5 bar -> true if document has word "foo" followed by "bar" after 5
> positions
>
> then some other ways it can be used are
> !(foo #<5 bar) -> true if document never has any "foo" followed by bar
> with in 5 positions.
>
> etc .....
>
> 2. How to implement such query operators?
>
> Should we modify QueryItem to include additional distance information or
> is there any other way to accomplish it?
>
> Is the following list sufficient to accomplish this?
> a. Modify to_tsquery
> b. Modify TS_execute in tsvector_op.c to check new operator
>
> Is there anything needed in rewrite subsystem?
>
> 3. Are these valid uses of the operators and if yes what would they
> mean?
>
> foo #5 (bar & cup)
>
> If no then should the operator be applied to only two QI_VAL's?
>
> 4. If the operator only applies to two query items can we create an
> index such that (foo, bar)-> documents[min distance, max distance]
> How difficult it is to implement an index like this?
>
>
> Thanks,
> -Sushant.
>
> On Thu, 2008-06-05 at 19:37 +0400, Teodor Sigaev wrote:
>>> I can add index support and support for arbitrary distance between
>>> lexeme.
>>> It appears to me that supporting arbitrary boolean expression will be
>>> complicated. Can we pull out something from TSQuery?
>>
>> I don't very like an idea to have separated interface for phrase search. Your
>> patch may be a module and used by people who really wants to have a phrase search.
>>
>> Introducing new operator in tsquery allows to use already existing
>> infrastructure of tsquery such as concatenations (&&, ||, !!), rewrite subsystem
>> etc. But new operation/types specially designed for phrase search makes needing
>> to make that work again.
>>
>
>
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Initdb problem on debian mips cobalt: Bus error

Tom Lane wrote:
> Glyn Astill <glynastill@yahoo.co.uk> writes:
>> No. Will recompile with debug info and post back when done.
>
> FWIW, the most likely issue here is the MIPS-specific assembly code in
> src/include/storage/s_lock.h --- I'm not sure how many MIPS platforms
> that's really been exercised on, but it may not work on yours. While
> you're waiting for the rebuild you might try to find a MIPS guru to
> show that code to.

hmm well - lionfish (which is now offline due to a broken power supply)
is actually a cobalt cube too (and is running debian). So if we really
managed to break mipsel it must have happened in the last few months:


http://www.pgbuildfarm.org/cgi-bin/show_history.pl?nm=lionfish&br=HEAD


Stefan

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Friday, July 18, 2008

Re: [HACKERS] phrase search

I looked at query operators for tsquery and here are some of the new
query operators for position based queries. I am just proposing some
changes and the questions I have.

1. What is the meaning of such a query operator?

foo #5 bar -> true if the document has word "foo" followed by "bar" at
5th position.

foo #<5 bar -> true if document has word "foo" followed by "bar" with in
5 positions

foo #>5 bar -> true if document has word "foo" followed by "bar" after 5
positions

then some other ways it can be used are
!(foo #<5 bar) -> true if document never has any "foo" followed by bar
with in 5 positions.

etc .....

2. How to implement such query operators?

Should we modify QueryItem to include additional distance information or
is there any other way to accomplish it?

Is the following list sufficient to accomplish this?
a. Modify to_tsquery
b. Modify TS_execute in tsvector_op.c to check new operator

Is there anything needed in rewrite subsystem?

3. Are these valid uses of the operators and if yes what would they
mean?

foo #5 (bar & cup)

If no then should the operator be applied to only two QI_VAL's?

4. If the operator only applies to two query items can we create an
index such that (foo, bar)-> documents[min distance, max distance]
How difficult it is to implement an index like this?


Thanks,
-Sushant.

On Thu, 2008-06-05 at 19:37 +0400, Teodor Sigaev wrote:
> > I can add index support and support for arbitrary distance between
> > lexeme.
> > It appears to me that supporting arbitrary boolean expression will be
> > complicated. Can we pull out something from TSQuery?
>
> I don't very like an idea to have separated interface for phrase search. Your
> patch may be a module and used by people who really wants to have a phrase search.
>
> Introducing new operator in tsquery allows to use already existing
> infrastructure of tsquery such as concatenations (&&, ||, !!), rewrite subsystem
> etc. But new operation/types specially designed for phrase search makes needing
> to make that work again.
>


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] 10.5 OS X ppc64 problem

Shane Ambler <pgsql@Sheeky.Biz> writes:
> There has been some talk about getting postgres to build as a universal
> binary. The current makefiles don't support the option but a couple of
> people have come up with work arounds. I do believe that there are plans
> to add this to future releases.

FYI, we are making some progress:
http://archives.postgresql.org/pgsql-hackers/2008-07/msg00884.php

I am not sure we'll ever bother to fix the configuration-file issue,
but other than that it does seem to be feasible to build universal
binaries without major hacking on the source code as of CVS HEAD.
(I fixed the postgres.bki issue a few minutes ago.)

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[COMMITTERS] pgsql: Avoid substituting NAMEDATALEN, FLOAT4PASSBYVAL, and

Log Message:
-----------
Avoid substituting NAMEDATALEN, FLOAT4PASSBYVAL, and FLOAT8PASSBYVAL into
the postgres.bki file during build, because we want that file to be entirely
platform- and configuration-independent; else it can't safely be put into
/usr/share on multiarch machines. We can do the substitution during initdb,
instead. FLOAT4PASSBYVAL and FLOAT8PASSBYVAL are new breakage as of 8.4,
while the NAMEDATALEN hazard has been there all along but I guess no one
tripped over it. Noticed while trying to build "universal" OS X binaries.

Modified Files:
--------------
pgsql/src/backend/catalog:
genbki.sh (r1.44 -> r1.45)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/catalog/genbki.sh?r1=1.44&r2=1.45)
pgsql/src/bin/initdb:
initdb.c (r1.157 -> r1.158)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/bin/initdb/initdb.c?r1=1.157&r2=1.158)
pgsql/src/tools/msvc:
Genbki.pm (r1.4 -> r1.5)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/tools/msvc/Genbki.pm?r1=1.4&r2=1.5)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[HACKERS] Getting to universal binaries for Darwin

Awhile back we determined that the big stumbling block for building
Postgres universal binaries for OS X was that we were using "ld"
to produce intermediate SUBSYS.o files, and it didn't want to preserve
multi-architecture components of input .o files. Peter got rid of that
hack recently, so I thought I'd see what it takes to build a universal
binary with CVS HEAD.

The good news is that the tools problem seems to be solved. If you
add something like "-arch i386 -arch ppc" to CFLAGS and build normally,
you get real working multiarch binaries and libraries. (At least it
works for me on OS X 10.5.4 --- no promises about older toolchains.)

The bad news is that if you only do that, only the arch that you
actually build on will work. We have configure set up to insert
various hardware-dependent definitions into pg_config.h and
ecpg_config.h, and if you don't have the right values visible for
each compilation, the resulting executables will fail.

You can get around that by hacking up the generated config files
with #ifdef __i386__ and so on to expose the correct values of
the hardware-dependent symbols to each build. Of course you have
to know what the correct values are --- if you don't have a sample
of each architecture handy to run configure against, it'd be easy
to miss some things. And even then it's pretty tedious. I am
not sure if it is possible or worth the trouble to try to automate
this part better.

The other big problem is that genbki.sh wants to propagate some of those
hardware-dependent symbols into postgres.bki, in particular
FLOAT4PASSBYVAL and FLOAT8PASSBYVAL. This is a complete nonstarter,
because we put postgres.bki under share/ which means it is supposed to
be architecture independent. (I'm really glad I realized this before we
released 8.4, because it'd violate Red Hat's multilib file rules...)
I think we can pretty easily fix that by moving the responsibility
for substituting these values into initdb, though.

Modulo the above problems, I was able to build i386+ppc binaries that
do in fact work on both architectures. I haven't got any 64-bit Apple
machines to play with, so there might be 64-bit issues I missed.
Still, this is a huge step forward compared to what was discussed here:
http://archives.postgresql.org/pgsql-general/2008-02/msg00200.php

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] temp table problem

I can't help suspecting that the two statements in question were run
in different sessions (or at least different transactions?).

...Robert

On Fri, Jul 18, 2008 at 9:11 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> "Abbas" <abbas.butt@enterprisedb.com> writes:
>> I have come across a problem. When you try to access a temp table
>> created via SPI_EXEC, you get a table not found error.
>
>> SPI_EXEC("CREATE TEMP TABLE my_temp_table(first_name text, last_name
>> text)", UTILITY);
>> SPI_EXEC("REVOKE ALL ON TABLE my_temp_table FROM PUBLIC", UTILITY);
>
>> The second statement generates a table not found error, although the
>> first statement was successful.
>
> Works for me ...
>
> ret = SPI_execute("CREATE TEMP TABLE my_temp_table(first_name text, last_name text)", false, 0);
>
> if (ret != SPI_OK_UTILITY)
> elog(ERROR, "SPI_execute(CREATE) returned %d", ret);
>
> ret = SPI_execute("REVOKE ALL ON TABLE my_temp_table FROM PUBLIC", false, 0);
>
> if (ret != SPI_OK_UTILITY)
> elog(ERROR, "SPI_execute(REVOKE) returned %d", ret);
>
> What PG version are you testing? Maybe you need to show a complete
> test case, instead of leaving us to guess at details?
>
> regards, tom lane
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [PERFORM] An "obvious" index not being used

Daniele Varrazzo writes:

> I suspect the foo.account_id statistical data are not used at all in query:
> the query planner can only estimate the number of accounts to look for, not

You mentioned you bumped your default_statistics_target.
What did you increase it to?
My data sets are so "strange" that anything less than 350 gives many bad
plans.

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: [HACKERS] Load spikes on 8.1.11

On Fri, Jul 18, 2008 at 7:15 PM, David Fetter <david@fetter.org> wrote:
On Fri, Jul 18, 2008 at 10:15:42AM +0530, Gurjeet Singh wrote:
> On Fri, Jul 18, 2008 at 10:05 AM, Gurjeet Singh <singh.gurjeet@gmail.com>
> wrote:
>
> > On Fri, Jul 18, 2008 at 9:58 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >
> >> "Gurjeet Singh" <singh.gurjeet@gmail.com> writes:
> >> >     During these spikes, in the 'top' sessions we see the 'idle' PG
> >> > processes consuming between 2 and 5 % CPU, and since the box has 8 CPUS
> >> (2
> >> > sockets and each CPU is a quad core Intel Xeon processors) and somewhere
> >> > around 200 Postgres processes, the load spikes to above 200; and it does
> >> > this very sharply.
> >>
> >> This looks like heavy contention for a spinlock.  You need to get a
> >> higher-level analysis of what's happening before anyone can say much
> >> more than that.
> >>
> >> Note that 8.1 is pretty much ancient history as far as scalability to
> >> 8-core hardware goes.  You should probably consider updating to 8.3
> >> before investing too much time in tracking down what's happening.
> >> If you can still show the problem on 8.3 then there would be some
> >> interest in fixing it ...
> >
> >
> > Upgrading is on the cards, but not as high priority as I would like it to
> > be! This is a production box, and we desperatly need some respite from these
> > spikes.
> >
> > Can you please elaborate on what high level diagnosis would you need?
> >
> > I just ran DROP SCHEMA _<slony schema> CASCADE; and it spiked again, on a
> > very low loaded box!!
> >
> > Thanks for all you help.
> >
>
> Would reducing the number of connections on the DB help in reducing the
> spike?

Just generally, reducing the number of connections to the DB will help
in reducing resource consumption.

Will try this option, at least in the next schema upgrade or when setting up Slony.



When you first get a chance, use or set up a test environment where
you can test the upgrade to 8.3.latest.

Based on the thread above, we seem to be moving towards greater consensus on upgrade. One of the major hurdles in our environment's upgrade is the loss of implicit casts in 8.3.

Following is the environment we have:

select version();
                                                  version
-----------------------------------------------------------------------------------------------------------
 PostgreSQL 8.1.11 on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-9)
(1 row)

I cannot see oprofile installed on this box, so will try to get that installed and get you guys some more details when this happens next.

Thanks,
--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

Re: [PERFORM] 3ware vs Areca

Jeffrey Baker writes:

> Their firmware is, frankly, garbage. In more than one instance we
> have had the card panic when a disk fails, which is obviously counter
> to the entire purpose of a RAID.

I have had simmilar problems with 3ware 9550 and 9650 cards.
Undre FreeBSD I have seen constant crashes under heavy loads.
Used to think it was just FreeBSD, but saw a thread on StorageReview where
the same was happening under Linux.

> controllers from our database server and replaced them with HP P800s.

How is that working out?
Which RAID level? SAS/SATA?

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

[pgsql-www] How to contribute to site?

I am looking at https://pgweb.postgresql.org and don't see any pointers on
how one contributes to the pg doc project.

Also checked http://wiki.postgresql.org/wiki/Developer_FAQ and the wiki in
general.

Any URLs or any pointers on how to contribute?
I particular I want to provide examples for this page:
http://www.postgresql.org/docs/8.3/interactive/ddl-partitioning.html

I spent the better part of an afternoon figuring out how to make a dynamic
function to send data to the proper partition.

I recall there was a presentation in a PDF that had a sample, but could not
find it anymore. Neither could find any examples.

Figure I could share what I have so others have a starting point when doing
partitions.

--
Sent via pgsql-www mailing list (pgsql-www@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

Re: [GENERAL] UPDATE runs slow in a transaction

Viktor Rosenfeld <rosenfel@informatik.hu-berlin.de> writes:
> Postgres is indeed selecting a bad plan. Turns out that the index I
> created to speed up the UPDATE isn't used inside a transaction block.

That doesn't make any sense to me, and in fact I cannot replicate any
such behavior here. What PG version are you running, exactly?

The exact test case I tried is attached --- it's just your original
incomplete example with some dummy data created beforehand. I get
a plan using the tmp_id index in all supported PG versions.

regards, tom lane


drop table _struct, _rank;
create table _struct(token_index int, id int);
create table _rank(struct_ref int, pre int, post int);

insert into _struct select i, i from generate_series(1,1000) g(i);
insert into _rank select i, i, i from generate_series(1,1000) g(i);
analyze _struct;
analyze _rank;

begin;

-- add columns left_token, right_token and copy values from token_index
ALTER TABLE _struct ADD left_token integer;
ALTER TABLE _struct ADD right_token integer;
UPDATE _struct SET left_token = token_index;
UPDATE _struct SET right_token = token_index;

-- set left, right values for non-terminals
-- (use temporary table to get rid of joins between struct and rank)
CREATE TABLE tmp AS
SELECT r.pre, r.post, s.id, s.left_token, s.right_token
FROM _rank r, _struct s
WHERE r.struct_ref = s.id;

CREATE INDEX idx_tmp_pre_post ON tmp (pre, post);

UPDATE tmp SET left_token = (SELECT min(t2.left_token) FROM tmp t2 WHERE t2.pre >= tmp.pre AND t2.pre <= tmp.post);
UPDATE tmp SET right_token = (SELECT max(t2.right_token) FROM tmp t2 WHERE t2.pre >= tmp.pre AND t2.pre <= tmp.post);

-- copy left, right values for everything
CREATE INDEX tmp_id ON tmp (id);
--analyze tmp;
explain UPDATE _struct SET left_token = (SELECT DISTINCT left_token FROM tmp WHERE _struct.id = tmp.id);
-- the UPDATE above takes ages when called within a transaction
UPDATE _struct SET right_token = (SELECT DISTINCT right_token FROM tmp WHERE _struct.id = tmp.id);

-- clean up
DROP TABLE tmp;

rollback;

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] temp table problem

"Abbas" <abbas.butt@enterprisedb.com> writes:
> I have come across a problem. When you try to access a temp table
> created via SPI_EXEC, you get a table not found error.

> SPI_EXEC("CREATE TEMP TABLE my_temp_table(first_name text, last_name
> text)", UTILITY);
> SPI_EXEC("REVOKE ALL ON TABLE my_temp_table FROM PUBLIC", UTILITY);

> The second statement generates a table not found error, although the
> first statement was successful.

Works for me ...

ret = SPI_execute("CREATE TEMP TABLE my_temp_table(first_name text, last_name text)", false, 0);

if (ret != SPI_OK_UTILITY)
elog(ERROR, "SPI_execute(CREATE) returned %d", ret);

ret = SPI_execute("REVOKE ALL ON TABLE my_temp_table FROM PUBLIC", false, 0);

if (ret != SPI_OK_UTILITY)
elog(ERROR, "SPI_execute(REVOKE) returned %d", ret);

What PG version are you testing? Maybe you need to show a complete
test case, instead of leaving us to guess at details?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[GENERAL] using regexp_matches and array manipulation

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Can anyone give me an example of how to use regexp_matches and use the
captured values?

For instance, if I have a delimited string "a,b,c" and I want to put
each letter into a variable so I can subsequently use those variables in
an insert statement, how would I do that? I know regexp_matches returns
a text array, but how do I assign the value to an array and then access
those values?

leaselog=# select regexp_matches('a,b,c','(.*),(.*),(.*)');

~ regexp_matches
- ----------------
~ {a,b,c}
(1 row)


I've tried select into, but that just created a table and didn't put the
values into an array variable.
leaselog=# select regexp_matches('a,b,c','(.*),(.*),(.*)') into foo;


Thanks in advance!


Rgds,

Bret
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)

iD8DBQFIgT2T/PgQIGRJuUcRAvMGAJ9VRNfc5ZZsFtS2LG8VJgPNNnL1wwCfewlf
Jih6ReqSTj6Pp9Ya3B2uMn8=
=HbPn
-----END PGP SIGNATURE-----


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [pgsql-es-ayuda] problema con dump de una bd

--- El jue 17-jul-08, Carlos Mendez <lucas1850@gmail.com> escribió:

> De: Carlos Mendez <lucas1850@gmail.com>
> Asunto: [pgsql-es-ayuda] problema con dump de una bd
> A: pgsql-es-ayuda@postgresql.org
> Fecha: jueves, 17 julio, 2008, 12:15 am
> Hola, que tal,
>
> hace poco utilice el phppgadmin para hacer un dump de una
> bd y todo
> correcto, pero hoy no pude hacerlo el phppgadmin solo me
> descargaba archivos
> vacios dum.sql con 0 Kb, bueno entonces trate de usar el
> pg_dump desde la
> linea de comando pero tampoco me dice algo como:
> bash: commad not found
> solo quiero volcar la bd a un archivo.sql para construir
> otra bd, pero no me
> funcionan ni el phpadmin ni el pg_dump, voy a seguir
> investigando
> Gracias de antemano por la ayuda,
> saludos.
>
> PS: Uso FC3, pgsql 8.3.3

Fijate que en la version 8.3.3 el nombre de la bd va despues de los parametros
Ejemplo

pg_dump -U postgres -E UTF8 -c otras opciones.... nombre_de_bd > archivo.sql

-E ("opcional para llevarla al encoding que necesitemos en otro server)

Atte.
Gabriel Colina
-c ("opcional para que haga drop antes de crear la nueva")


____________________________________________________________________________________
Yahoo! MTV Blog & Rock &gt;¡Cuéntanos tu historia, inspira una canción y gánate un viaje a los Premios MTV! Participa aquí http://mtvla.yahoo.com/
--
TIP 2: puedes desuscribirte de todas las listas simultáneamente
(envía "unregister TuDirecciónDeCorreo" a majordomo@postgresql.org)

Re: [JDBC] Problem with java infinity mapping to postgres real column type

"Johan Miller" <johanmiller@gmail.com> writes:
> On Thu, Jul 17, 2008 at 08:22, Kris Jurka <jurka@ejurka.com> wrote:
>> I see this in the 8.3 release notes:
>>
>> Fix float4/float8 to handle Infinity and NAN (Not A Number) consistently
>> (Bruce) The code formerly was not consistent about distinguishing Infinity
>> from overflow conditions.

> What would be the options to handle this on earlier version where it is not
> fixed? Would it be a possibility that this will get changed in the 8.1 and
> 8.2 versions also?

I believe the conclusion was that that change shouldn't be back-patched
because it would be a behavioral change that might break applications.

regards, tom lane

--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

Re: [GENERAL] Backup/Restore of single table in multi TB database

Simon Riggs wrote:
> Have a look at pg_snapclone. It's specifically designed to significantly
> improve dump times for very large objects.
>
> http://pgfoundry.org/projects/snapclone/
>
Also, in case the original poster is not aware, by default pg_dump
allows to backup single tables.
Just add -t <table name>.

Does pg_snapclone works mostly on large rows or will it also be faster
than pg_dump for narrow tables?

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[pgsql-es-ayuda] Definir campos xml

  Hola, quiero hacer una aplicación, usando el soprte nativo para XML que trae Postgresql 8.3.3 . La pregunta es: ¿cómo defino los campos que usa XML para que el sistema logre distinguir cuales son: núméricos, textos o de otros tipos, o se guardan todos como texto?

 

  Por ejemplo:

                        

CREATE TABLE test (

id int4 NOT NULL,

xml text,

CONSTRAINT pk PRIMARY KEY (id)

);

 

  Yo sé, por lo que he visto que xml test, puede rebicir varios campos, donde se define el esquema para XML, perdón por la ignorancia :(

 

  Cualquier ayuda será bien recibida.




Enviado desde Correo Yahoo!
La bandeja de entrada más inteligente.

Re: [GENERAL] tsearch dictionary list?

On Fri, 18 Jul 2008, Daniel Chiaramello wrote:

> Hi all.
>
> I'm new to postgresql world, and I have to extend an existing product for
> Thailand - that product has some features based on tsearch, and I was
> wondering if there is an existing dictionary for that language... I failed to
> find any reference of such dictionary on the web, and of course I don't speak
> thailandese at all!

read documentation first and you'll be surprised. In brief,
tsearch supports all dictionaries of OpenOffice
http://wiki.services.openoffice.org/wiki/Dictionaries


>
> In fact, is there somewhere a list of existing tsearch dictionaries? Are
> japanese, chinese, or other "exotic" languages supported by tsearch?
>
> Thanks for your attention,
> Daniel Chiaramello
>
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Reducing memory usage of insert into select operations?

On 4:55 pm 07/18/08 Tom Lane <tgl@sss.pgh.pa.us> wrote:
> The only thing I can think of is that you had a huge number of rows
> with all the same hash value, so that there wasn't any way to split
> the batch into smaller sections. What are the join keys exactly in
> this query, and what can you tell us about their data distributions?

I can't put actual table or column names so I am putting the actual select
and explain, with all names changed..

insert into customer_transactions
(record_id, date, type, amount, ids, groupid)
select
ca.record_id, coh.date, coh.type, coh.amount, coh.ids, ids.groupid
from
customer_original_historical coh,
cards ca,
customer_ids ids
where
ca.natural_key = coh.natural_key
and ids.ids = coh.ids
and coh.yearmo > '200703';

Hash Join (cost=712213.57..27293913.33 rows=234402352 width=24)
Hash Cond: (coh.id = ids.id)
-> Hash Join (cost=551387.26..18799378.16 rows=234402352
width=22)
Hash Cond: (coh.user_id = ca.user_id)
-> Seq Scan on customer_original_historical coh
(cost=0.00..6702501.40 rows=234402352 width=47)
Filter: (yearmo > '200703'::bpchar)
-> Hash (cost=268355.67..268355.67 rows=14637567 width=32)
-> Seq Scan on cards ca
(cost=0.00..268355.67 rows=14637567 width=32)
-> Hash (cost=77883.25..77883.25 rows=5055525 width=6)
-> Seq Scan on customer_ids ids
(cost=0.00..77883.25 rows=5055525 width=6)

There was a single table, customer_original_historical, which was using a
natural key with a text field.

Most queries used the customer_original_historical by itself or joined
against a single other table which we shoudl call "area".

The new schema I am testing is to split the one single file into 12 files
per month.

In addition I replaced the natural keys with a synthetic integer key.
I also replaced the "area" table with a customer_ids table which only has
two columns: synthetic key for historical and a region.

In order to have 12 tables per month I grouped all the regions into 12
groups. Queries are usually within a single region so what I am trying to
benchmark is if dividing 24 months of data into 24 sets of 12 regions will
perform better than a single large file.

The distribution of the joins is:
There are about 1000,000 unique natural keys. Each natural key has in
average 15 rows per month.
ids are regions where the natural_keys are. Figure 10s of thousands of
natural_keys to an id.

Is that along the lines of what you were looking for?


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [ADMIN] answer my question, please!!!

riakdr@gmail.com ("ria ulfa") writes:
> i want to copy table from file with command:
>      COPY log FROM '/tmp/logfile.txt';
> but there is error:
>      ERROR: could not open file "/tmp/logfile.txt" for reading: Permission denied
> what can i do for repair this error??
> thanks for the answer..

It's worth observing that the louder the demands are in the subject
line, the less likely that people are to *want* to volunteer
assistance.

The problem here is that COPY accesses data as whatever Unix user is
running the database, typically "postgres" or "pgsql".

Presumably that user hasn't rights to access the file
"/tmp/logfile.txt". Perhaps that needs to be changed.
--
select 'cbbrowne' || '@' || 'linuxfinances.info';
http://cbbrowne.com/info/lsf.html
Rules of the Evil Overlord #145. "My dungeon cell decor will not
feature exposed pipes. While they add to the gloomy atmosphere, they
are good conductors of vibrations and a lot of prisoners know Morse
code." <http://www.eviloverlord.com/>

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [HACKERS] Postgres-R: primary key patches

markus@bluegap.ch (Markus Wanner) writes:
> Hello Chris,
>
> chris wrote:
>> Slony-I does the same, with the "variation" that it permits the option
>> of using a "candidate primary key," namely an index that is unique+NOT
>> NULL.
>>
>> If it is possible to support that broader notion, that might make
>> addition of these sorts of logic more widely useful.
>
> Well, yeah, that's technically not much different, so it would
> probably be very easy to extend Postgres-R to work on any arbitrary
> Index.
>
> But what do we have primary keys for, in the first place? Isn't it
> exactly the *primay* key into the table, which you want to use for
> replication? Or do we need an additional per-table configuration
> option for that? A REPLICATION KEY besides the PRIMARY KEY?

I agree with you that tables are *supposed* to have primary keys;
that's proper design, and if tables are missing them, then something
is definitely broken.

Sometimes, unfortunately, people make errors in design, and we wind up
needing to accomodate situations that are "less than perfect."

The "happy happenstance" is that, in modern versions of PostgreSQL, a
unique index may be added in the background so that this may be
rectified without outage if you can live with a "candidate primary
key" rather than a true PRIMARY KEY.

It seems to me that this extension can cover over a number of "design
sins," which looks like a very kind accomodation where it is surely
preferable to design it in earlier rather than later.

>> I know Jan Wieck has in mind the idea of adding an interface to enable
>> doing highly efficient IUD (Insert/Update/Delete) via generating a way
>> to do direct heap updates, which would be *enormously* more efficient
>> than the present need (in Slony-I, for instance) to parse, plan and
>> execute thousands of IUD statements. For UPDATE/DELETE to work
>> requires utilizing (candidate) primary keys, so there is some
>> seemingly relevant similarity there.
>
> Definitely. The remote backend does exactly that for Postgres-R: it
> takes a change set, which consists of one or more tuple collections,
> and then applies these collections. See ExecProcessCollection() in
> execMain.c.
>
> (Although, I'm still less than thrilled about the internal storage
> format of these tuple collections. That can certainly be improved and
> simplified.)

You may want to have a chat with Jan; he's got some thoughts on a more
general purpose mechanism that would be good for this as well as for
(we think) extremely efficient bulk data loading.
--
select 'cbbrowne' || '@' || 'linuxfinances.info';
http://cbbrowne.com/info/lsf.html
Rules of the Evil Overlord #145. "My dungeon cell decor will not
feature exposed pipes. While they add to the gloomy atmosphere, they
are good conductors of vibrations and a lot of prisoners know Morse
code." <http://www.eviloverlord.com/>

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Thursday, July 17, 2008

Re: [HACKERS] TABLE-function patch vs plpgsql

Hello

The core of problems is in standard that doesn't know RETURN NEXT
statement and knows only RETURN TABLE statement - so PL/pgPSM or SQL
doesn't have problem. I am not sure about PL/pgSQL, but I thing so
using TABLE attribs as OUT variables is maybe too simple solution -
there isn't any progress to current state, and where OUT variables are
typically source of mistakes.

Maybe we can use some well defined implicit record, maybe NEW (or
RESULT, ROW_RESULT, ROW, TABLE_ROW, ...) like trigger - some like

create or replace function foo(i int) returns table(a int, b int) as $$
begin
for j in 1..i loop
new.a := j; new.b := j+1;
return next new; -- maybe only RETURN NEXT???
end loop;
end;
$$ language plpgsql

Regards
Pavel Stehule

2008/7/18 Tom Lane <tgl@sss.pgh.pa.us>:
> I've been working on the TABLE-function patch, and I am coming to the
> conclusion that it's really a bad idea for plpgsql to not associate
> variables with output columns --- that is, I think we should make
> RETURNS TABLE columns semantically just the same as OUT parameters.
> Here are some reasons:
>
> 1. It's ludicrous to argue that "standards compliance" requires the
> behavior-as-submitted. plpgsql is not specified by the SQL standard.
>
> 2. Not having the parameter names available means that you don't have
> access to their types either, which is a big problem for polymorphic
> functions. Read the last couple paragraphs of section 38.3.1:
> http://developer.postgresql.org/pgdocs/postgres/plpgsql-declarations.html#PLPGSQL-DECLARATION-ALIASES
> as well as the following 38.3.2. How would you do those things with
> a polymorphic TABLE column?
>
> 3. Not treating the parameters as assignable variables makes RETURN NEXT
> nearly worthless in a TABLE function. Since they're not assignable,
> you can't use the parameterless form of RETURN NEXT (which'd return
> the current values of the variables). The only alternative available
> is to return a record or row variable; but there's no convenient way
> to declare such a variable, since after all the whole point here is
> that the function's output rowtype is anonymous.
>
> 4. It's a whole lot easier to explain things if we can just say that
> OUT parameters and TABLE parameters work alike. This is especially
> true when they actually *are* alike for all the other available PLs.
>
> If we insist on the current definition then we are eventually going to
> need to kluge up some solutions to #2 and #3, which seems like make-work
> to me when we already have smooth solutions to these problems for
> OUT parameters.
>
> Comments?
>
> For the archives, here is the patch as I currently have it (with the
> no-plpgsql-variables behavior). But unless I hear a good argument
> to the contrary, I'm going to change that part before committing.
>
> regards, tom lane
>
>

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Load spikes on 8.1.11

On Fri, Jul 18, 2008 at 10:21 AM, Andrew Sullivan <ajs@commandprompt.com> wrote:
On Fri, Jul 18, 2008 at 10:05:33AM +0530, Gurjeet Singh wrote:

> I just ran DROP SCHEMA _<slony schema> CASCADE; and it spiked again, on a
> very low loaded box!!

Ah, well, if slony is involved, then you have possible locking
problems in the database _also_ to contend with, along with the
spinlock problems.  This will for sure cause spikes.

You need to tell us more about what you're doing.  And I bet some of
it belongs on the slony lists.


I am in the eye of the storm right now.

Just started INIT cluster Slonik command and that spiked too.. for more than 10 minutes now!!


--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

Re: [NOVICE] tracking down idle transactions in pg_locks

Thanks for your help, I was getting rather confident this app was to
blame, it just didn't make sense to me that it would show up on an
unrelated table. This gives me more information to contact the developer
with.

W are running 8.3, I have been able to reproduce the problem, it
occurs after an insert has occurred.


Tom Lane wrote:
> David Gardner <david@gardnerit.net> writes:
>
>> I've been tracking down a problem with idle transactions being left open
>> and not being able to vacuum one of my tables. I think I have tracked it
>> down to proprietary app, the problem seems to be that it leaves "idle
>> transactions" open.
>>
>
>
>> However the table that we are having problems vacuuming (probably just
>> because it is the most active table) is in the same db, but a different
>> scheme not accessed by that app. Could this app still be causing the
>> problem?
>>
>
> Different schema won't help --- VACUUM assumes that *any* other
> transaction in the same database might potentially be able to look at
> the target table later, so it can't remove rows that that transaction
> might be able to "see".
>
> The most common cause of this problem is apps that issue "COMMIT; BEGIN"
> and then go to sleep. If that's what this app is doing (if you don't
> know, turn on statement logging and find out...) then updating to a more
> recent PG version might help. In 8.3 a transaction doesn't block VACUUM's
> row reclamation until it's done something more than just BEGIN.
>
> regards, tom lane
>


--
David Gardner


--
Sent via pgsql-novice mailing list (pgsql-novice@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-novice