Thursday, August 7, 2008

Re: [HACKERS] patch: Add columns via CREATE OR REPLACE VIEW

ALTER VIEW does not sound useful for me.
CREATE OR REPLACE VIEW should create or replace view and only thing that should be same is the name of the view. It's up to Postgres to invalidate all plans and up to developer to make sure that all places where his view is used will stay still working. All this discussion about matching up columns and AI seems totally useless to me :)

On Fri, Aug 8, 2008 at 4:41 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> Well, my feeling is that if we are inventing a new feature we ought not
> paint ourselves into a corner by failing to consider what will happen
> when obvious extensions to the feature are attempted.  Whether the
> present patch is self-consistent is not the question --- the question
> is do we have a self-consistent vision of how we will later do the
> other stuff like renaming, changing column type, etc.

If we can work out that design, I think that's great.  However, it
doesn't actually 100% matter whether we know the one true way that we
will definitely implement those features - it only matters that none
of the things we might choose are inconsistent with what we're doing
now.

In order to avoid being AI-complete, REPLACE VIEW needs some kind of
straightforward algorithm for matching up the old and new target
lists.  AFAICS, the only thing to decide here is what you want to use
as the key.  There are three possibilities that I can think of: [1]
name, [2] position, [3] both name and position.

It's axiomatic that REPLACE VIEW can't be given the capability to make
any modification that involves changing the key field, so in [1] you
can't rename columns, in [2] you can't reorder columns, and in [3] you
can't do either.  Furthermore, in [2], you also can't support dropping
columns, because a drop is indistinguishable from renaming and
retyping every column from the point of the drop onwards.  Therefore,
the maximum set of operations REPLACE VIEW can potentially support in
each scenario are:

[1] add column, change type, drop column, reorder columns
[2] add column, change type, rename
[3] add column, change type, drop column

The actual set of operations supported may be less either because of
implementation limitations or because you don't want to provide users
with a foot-gun.  ISTM that allowing REPLACE VIEW to do renames in
scenario [2] can be pretty much rejected outright as a violation of
the principle of least surprise - there is an enormous danger of
someone simultaneously renaming and retyping a whole series of columns
when they instead intended to drop a column.  Similarly, in scenario
[1] or [3], ISTM that allowing someone to drop columns using REPLACE
VIEW is something of a foot-gun unless we are in scenario [1] and
reordering columns is also implemented, because users who don't RTFM
will try to reorder columns and it will succeed and fail erratically
according to whether there are dependencies that prevent dropping and
re-adding whatever subset of columns need to be shuffled to create the
same effect as would be produced by reordering.  However, in any
scenario, I can't see how adding columns or changing column types is
likely to produce any confusion or user-unexpected behavior.  Perhaps
I'm missing something?

Personally, I favor scenario [1].  I hardly ever rename database
columns, and I don't mind needing to ALTER VIEW RENAME COLUMN on those
rare occasions when I do, but I add new columns to my tables (which
then also need to be added to my views) on a regular basis.  If I
could keep groups of related columns together in the table and view
definitions without having to drop and recreate the objects, that
would be awesome.  But I'm not sure it's worth the amount of
implementation that would be required to get there, especially if all
of that implementation would need to be done by me (and
double-especially if none of it would likely be included in -core).

Of course, as I said before, nothing we do in REPLACE VIEW precludes
having a powerful implementation of ALTER VIEW.  But I think the
coding to make ALTER VIEW do these operations is a lot trickier,
because you have to deal with modifying the query that's already in
place piecemeal as you make your changes to the view.  It's not that
it can't be done, but I doubt it can be done in an 8K patch, and as
mentioned upthread, it certainly can't be done in a fully general
way... you will still frequently need to CREATE OR REPLACE VIEW
afterwards.  To put that another way, ALTER TABLE is a complicated
beast because you have to worry about how you're going to handle the
existing data, and ALTER VIEW will be a complicated beast for the
analogous reason that you need to worry about handing the existing
rewrite rule.  But at the moment when a REPLACE VIEW command is
executed, that problem goes away, because now you have the query in
your hand and just need to make the relation match it without breaking
any of the dependencies.

...Robert

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] restoring one table?

On Thu, Aug 7, 2008 at 4:34 PM, Brian Maguire <bmaguire@vantage.com> wrote:
> We need to restore one table from a backup. What is the proper way to do this?
>
> Our backup command looks like this:
>
> pg_dump -C -Fc -S postgresql mydatabase > today.backup.sqlc
>
> I am using PostgreSQL 8.1.

You have read this page, right:

http://www.postgresql.org/docs/8.1/interactive/app-pgrestore.html

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [ADMIN] [HACKERS] pg_restore -d cipa /cipa/RAJASTHAN/RAJASTHAN/CIPABACKUP01_08_2008.TAR pg_restore: [archiver] out of memory

Thanx for info I will take care of it in future.

I have tried with all ulimit settings but still getting the same error. I am trying to restore this ba ckup at server side on so i tried with shared_buffer and shmmax,

Why this error occured as the same backup day one before it is restoring fine but this particular is not getting restored.

Hope for getting any resolution.

Amit
+91-9818450022




On Wed, Aug 6, 2008 at 6:21 PM, Andrew Dunstan <andrew@dunslane.net> wrote:


Amit jain wrote:
Dear All,

 I am having a backup file in tar format. While restoring it through pg_restore instantly i am getting an following error.
*Command - pg_restore -d cipa /cipa/RAJASTHAN/RAJASTHAN/CIPABACKUP01_08_2008.TAR
Error  - pg_restore: [archiver] out of memory
*I have increased shmmax , maintenace work memory but still i am getting the same error.

2. I have make all log_parameter to debug5 but still getting a single error -line while restoring not getting any log or debug info about it.

Kindly help me out from this absurd behaviour of postgreSQL restoration process.
Any help would be highly appreciated. Thanx in advance



Please do not cross-post to multiple lists like this. (Also, adding addresses of individual hackers is not good netiquette). The -hackers list is quite inappropriate for usage questions.

It looks like the problem you have is that the client is running out of memory, so changing the server's memory settings seems unlikely to have any beneficial effect. Maybe you need to look at your ulimit settings.

cheers

andrew

[NOVICE] Calling an external command via triggers

Hi All,

Is it possible for me to call a certain command in the terminal via a trigger?

Say for example, after every INSERT in a certain table, I would be calling a script `php load.php` ?


Thanks,

Ridvan

--
リヅバン バルヨス
ridvan.baluyos@qualservcentral.com
http://ridvan.baluyos.net
http://www.onxiam.com/people/rbaluyos

Registered Linux User #439466
Registered Ubuntu User #16034
Q: Have you heard of the Object-Oriented way to get wealthy?
A: Inheritance.

[ADMIN] Problems with connection

Hi to all!

I have following systems:
1. Local machine with Windows XP SP2 and PostgreSQL 8.2 - test system.
2. Work server in local network (FreeBSD + PostgreSQL 8.2) - work
system.

I write client with Delphi 7 IDE and use Zeos Lib 6.6.2
(http://zeos.firmos.at/) for database access.

When I tried to connect to server I recieve error message: "SQL Error:
fe_sendauth: authentication type 5 not supported."
After searching in internet I found solution: set authentication
method "password" instead of "md5".
In file pg_hba.conf on local machine I wrote:

# TYPE DATABASE USER CIDR-ADDRESS METHOD
host all all 127.0.0.1/32 password

and on server:
# TYPE DATABASE USER CIDR-ADDRESS METHOD
host all all 192.168.3.0/24 password

My IP-address in local network is 192.168.3.16...

After that I can to connect to test PostgreSQL server on localhost,
and connection works fine, but attempt to connect to work server finishes by
same error message "SQL Error: fe_sendauth: authentication type 5 not
supported."

I tried to place string in pg_hba.conf on first and last positions,
wrote names of databases and users instead of "all" but it doesn't
work.

As I understand, my main problem is to force PostgreSQL server allow
me to connect with password authentication method, isn't it? And how I
can do that?

P. S. Sorry for poor english...


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [BUGS] Oder by not working

As Tom pointed out, the order depends on the locale.

Blanco, Jose wrote:
> So is there no way to make oder by work with names?
>
> -----Original Message-----
> From: Heikki Linnakangas [mailto:heikki@enterprisedb.com]
> Sent: Thursday, August 07, 2008 3:00 AM
> To: Blanco, Jose
> Cc: pgsql-bugs@postgresql.org
> Subject: Re: [BUGS] Oder by not working
>
> Heikki Linnakangas wrote:
>> Unlike on some systems, in PostgreSQL "ORDER BY 1" means order by the
>> constant value "1", not the first column. Try "ORDER BY name".
>
> Oh, that's of course not correct at all. "ORDER BY 1" does indeed mean
> order by first column. I don't know what I was thinking,
>


--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Re: [PERFORM] Query Plan choice with timestamps

On 07/ago/08, at 23:01, Tom Lane wrote:

> Giorgio Valoti <giorgio_v@mac.com> writes:
>> On 07/ago/08, at 17:50, Tom Lane wrote:
>>> These numbers seem pretty bogus: there is hardly any scenario in
>>> which a
>>> full-table indexscan should be costed as significantly cheaper
>>> than a
>>> seqscan. Have you put in silly values for random_page_cost?
>
>> No,
>
> I looked at it more closely and realized that the cost discrepancy is
> from the evaluation of the function: having to evaluate a SQL or
> plpgsql
> function 247736 times more than explains the cost estimate
> differential
> compared to a query that involves no function call. Some experiments
> here suggest that it hardly matters whether the query uses indexscan
> or
> seqscan because the time is dominated by the function calls anyway.

I see, thank you Tom. Could it be a good idea adding some notes about
it in <http://www.postgresql.org/docs/8.3/interactive/indexes-expressional.html
>? As you said, since the function call dominates the query cost, in
this case, I think there's no point to use an index expression.

Ciao
--
Giorgio Valoti
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

[pgsql-jobs] Senior PostgreSQL DBA

 

 

Senior PostgreSQL DBA

 

Position summary

We have an immediate need for a Senior PostgreSQL DBA who can lead and provide the direction for database technology and administration for our enterprise databases as we migrate to PostgreSQL.  In this position you will be the primary support for all development, test and production database environments to ensure high availability, reliability, appropriate capacity, scalability, backup, recoverability, performance, and 24x7 operation.

Required Skills/Experience

Ø      Minimum 5 years experience PostgreSQL DBA experience, and 2+ years as a DBA for large enterprise systems (+1 TB).

Ø      Strong physical database design experience.

Ø      Expertise in configuring, implementing and supporting high availability strategies with database systems including clustering, and replication.

Ø      Mastery of SQL (ANSI, PL/SQL).

Ø      Experience in software development using Perl, shell, or other scripting languages.

Ø      Solid background using UNIX/Linux operating systems.

Ø      Ability to perform capacity planning which supports the forecasting and purchasing processes.

Ø      Ability to implement Database security safeguards and audit them.

Ø      Ability to work in a high availability production environment where unplanned downtime is not acceptable.

Ø      Ability to communicate ideas and designs to all stakeholders, including non-technical audiences.


Ø      Bachelor's degree in Computer Science or related field; equivalent combinations of education and experience will be considered in lieu of a degree.

Ø      Minimum five years of progressive experience performing PostgreSQL administration in a production environment; experience should include three or more years of database administration in an enterprise level environment with very large (> 1 terabyte) databases.

 

 

Location: Portland, OR

Type: Direct Hire/Permanent, Contract-Hire

Target Start Date: ASAP / 2 weeks notice

Compensation: Salary (BOE), plus excellent benefits package (PTO, 100% employee paid med/dent/vis/RX/life, retirement plan, educational allowance, great facilities)

 

Please send MS Word version of resume to – recruiter@intersoftinc.com

 

Founded in 1985 and based in Lake Oswego, OR, we work with a number of large, medium, and small companies by supporting their need for software solutions and qualified technical professionals as well as providing consultancy and turnkey solutions.  Some of the areas we have particular strengths in are software development, SQA, embedded systems, and hardware design.  We partner with our clients and employees to make sure each engagement is mutually beneficial.  We offer all of our employees competitive compensation, medical, dental, 401k, and PTO.    

 

 

Re: [HACKERS] Visibility Groups

On Thu, 2008-08-07 at 14:18 +0100, Richard Huxton wrote:
> Simon Riggs wrote:
> > On Thu, 2008-08-07 at 12:55 +0200, Jochem van Dieten wrote:
> >> On Thu, Aug 7, 2008 at 12:38 PM, Simon Riggs wrote:
> >>> I propose creating "Visibility Groups" that *explicitly* limit the
> >>> ability of a transaction to access data outside its visibility group(s).
> >> Doesn't every transaction need to access data from the catalogs?
> >> Wouldn't the inclusion of a catalogs visibility group in every
> >> transaction negate any potential benefits?
> >
> > True, but I don't see the catalogs as frequently updated data. The
> > objective is to isolate frequently updated tables from long running
> > statements that don't need to access them.
> >
> > Tables can be in multiple visibility groups, perhaps that wasn't clear.
> > When we seek to vacuum a table, we take the lowest xmin of any group it
> > was in when we took snapshot.
>
> I'm not sure if "visibility group" is the best name for this - I had to
> go away and think through what you meant about that last bit. Have I got
> this right?
>
> So - a "visibility group" is attached to a transaction.

Perhaps visibility_scope might be better name. See below.

> My long-running transaction T0 can restrict itself to <catalogues> and
> table "event_log".
>
> Various other transactions T1..Tn make no promises about what they are
> going to access. They all share the "null visibility group".

OK, good example.

> A table "user_emails" is in the "null visibility group" and can be
> vacuumed based on whatever the lowest xid of T1..Tn is.
>
> Table "event_log" is in both groups and can only be vacuumed based on
> T0..Tn (presumably T0 is the oldest, since that's the point of the
> exercise).
>
> An attempt to write to user_emails by T0 will fail with an error.

All above correct

The point of doing this is that *if* T0 becomes the oldest transaction
it will *not* interfere with removal of rows on "user_emails".

> An attempt to read from user_emails by T0 will be allowed?

No, reads must also be excluded otherwise MVCC will be violated.

> What happens if I'm in ISOLATION LEVEL SERIALIZABLE? Presumably the read
> is disallowed then too?

No, that's not relevant. That is your choice about how often you update
your snapshot of the database. The visibility group refers to the
*scope* of the snapshot, so the two things are orthogonal.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Create Table Dinamic

Read about dynamic sql in Postgres documentation (EXECUTE statement):

http://www.postgresql.org/docs/8.2/static/plpgsql-statements.html

Igor

-----Original Message-----
From: Anderson dos Santos Donda [mailto:andersondonda@gmail.com]
Sent: Wednesday, August 06, 2008 9:10 PM
To: pgsql-general@postgresql.org
Subject: Create Table Dinamic

Hello All!

Its my first time here in maillist and I started work with postgre on
last moth.

My questions is: Threre is a way to create tables dinamic?

Example:

To create a table we use CREATE TABLE TableName ......

In my db, I have many tables with diferents names but with same colums

Example:

TableOne ( id int, name text );
TableTwo ( id int, name text );
TableThree ( id int, name text );

So, I created a function to create me this tables with diferents names

CREATE OR REPLACE FUNCTION MakeTables ( NameTable text ) RETURNS VOID $$
BEGIN
CREATE TABLE NameTable ( id int, name text ); END; $$ LANGUAGE
'plpgsql';

But, the plpgsql or postgre don't accept this..

So, How can I create a table with my function?

Thanks for any helps!!!

PS : If somebody want knows why I need to create this function, is
because in my db have 1000 tables with the some colums, and each time I
have a new client, I need to create this tables manually.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] bytea encode performance issues

On Thu, Aug 7, 2008 at 1:16 AM, Sim Zacks <sim@compulab.co.il> wrote:
>
>> I don't quite follow that...the whole point of utf8 encoded database
>> is so that you can use text functions and operators without the bytea
>> treatment. As long as your client encoding is set up properly (so
>> that data coming in and out is computed to utf8), then you should be
>> ok. Dropping to ascii is usually not the solution. Your data
>> inputting application should set the client encoding properly and
>> coerce data into the unicode text type...it's really the only
>> solution.
>>
> Email does not always follow a specific character set. I have tried
> converting the data that comes in to utf-8 and it does not always work.
> We receive Hebrew emails which come in mostly 2 flavors, UTF-8 and
> windows-1255. Unfortunately, they are not compatible with one another.
> SQL-ASCII and ASCII are different as someone on the list pointed out to
> me. According to the documentation, SQL-ASCII makes no assumption about
> encoding, so you can throw in any encoding you want.

no, you can't! SQL-ASCII means that the database treats everything
like ascii. This means that any operation that deals with text could
(and in the case of Hebrew, almost certianly will) be broken. Simple
things like getting the length of a string will be wrong. If you are
accepting unicode input, you absolutely must be using a unicode
encoded backend.

If you are accepting text of different encodings from the client, you
basically have two choices:
a) set client_encoding on the fly to whatever text the client is encoded in
b) pick an encoding (utf8) and convert all text to that before sending
it to the database (preferred)

you pretty much have to go with option 'b' if you are accepting any
text for which there is no supported client encoding translation.

merlin

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [PERFORM] Plz Heeeelp! performance settings

The performance problem is really only on the insertion and even more on
the treatment for the aggregation.

To treat the 3000 entrances and to insert, or update the tables it needs
10 minutes.

As I told you I inject 14000 query every 2 minutes, and it needs 10
minutes to treat 3000 of those query.

As you can easly understand it's a big narrow section.

I'm not doing the treatment in ones, cause I can't, but all is managed
by procedure.

> That wasn't a yes/no question. Please choose one of:
> Are you updating 6Go per week? most of update
> Are you adding 6Go per week? less of injection,

This action depend if the data are already present in the database.


>
> OK. I assume you're happy with the plans you are getting on these
> queries, since you've not provided any information about them.

The plan seems ok as it use index as well.
here is the plan :

explain analyse SELECT "insertUpdateTracks"(137,2605, 852, ('2008-08-06
19:28:54'::text)::date,3,'dailydisplay',2,NULL);
INFO: method 1
QUERY PLAN
------------------------------------------------------------------------------------
Result (cost=0.00..0.01 rows=1 width=0) (actual time=1.151..1.151
rows=1 loops=1)
Total runtime: 1.160 ms
(2 lignes)

Has you can see the runtime processs for an update in this table.

multiplying this per 10000, it is too long.

regards

david


Richard Huxton a écrit :
> dforums wrote:
>> vmstat is giving :
>> procs -----------memory---------- ---swap-- -----io---- --system--
>> ----cpu----
>> r b swpd free buff cache si so bi bo in cs us
>> sy id wa
>> 0 2 1540 47388 41684 7578976 0 0 131 259 0 1 9
>> 3 82 7
>
> This system is practically idle. Either you're not measuring it at a
> useful time, or there isn't a performance problem.
>
>> > > But
>> >> if I use a second machine to replicate the database, I escape this
>> >> problem isn't it ?
>> > You reduce the chance of a single failure causing disaster.
>> Not clear this reply. It's scare me ....
>
> If server A fails, you still have server B. If server A fails so that
> replication stops working and you don't notice, server B won't help any
> more.
>
>> > What do you mean by "take 6Go per week"? You update/delete that much
>> > data? It's growing by that amount each week?
>> YES
>
> That wasn't a yes/no question. Please choose one of:
> Are you updating 6Go per week?
> Are you adding 6Go per week?
>
>> > I'm not sure what "15000 request per 2 minutes and empty it into 10
>> min"
>> > means.
>> I insert 15000 datas every 2 min and delete 15000 every 10 min in
>> those tables
>> >
>> > Do you have 7500 requests per minute?
>> should be that, But in fact I'm not treating the datas in real time,
>> and I buffer the datas and push the data into the database every 2 min
>> > Are these updates?
>> during the delete the data are aggregated in other tables which make
>> updates
>
> OK, so every 2 minutes you run one big query that adds 15000 rows.
> Every 10 minutes you run one big query that deletes 15000 rows.
>
>> > To the "temporary storage"?
>>
>> > What is this "temporary storage" - an ordinary table?
>> Yes, I thied to use temporary tables but I never been able to connect
>> this tables over 2 different session/connection, seems that is a
>> functionnality of postgresql, or a misunderstanding from me.
>
> That's correct - temporary tables are private to a backend (connection).
>
>> > > I'm making some update or select on tables including more than 20
>> > > millions of entrance.
>> >
>> > Again, I'm not sure what this means.
>>
>> To aggregate the data, I have to check the presence of others
>> information that are stores in 2 tables which includes 24 millions of
>> entrance.
>
> OK. I assume you're happy with the plans you are getting on these
> queries, since you've not provided any information about them.
>
>> > Oh - *important* - which version of PostgreSQL are you running?
>> 8.1.11
>> > Is an upgrade practical?
>> We are working of trying to upgrade to 8.3.3, but we are not yet ready
>> for such migration
>
> OK
>
>> > Looking at your postgresql.conf settings:
>> >
>> > max_connections = 624
>> >
>> > That's an odd number.
>> Now we could decrease this number, it's not so much usefull for now.
>> we could decrease is to 350.
>
> I don't believe you've got 350 active connections either. It will be
> easier to help if you can provide some useful information.
>
>> > effective_cache_size = 625000
>> >
>> > That's around 5GB - is that roughly the amount of memory used for
>> > caching (what does free -m say for buffers/cache)?
>> total used free shared buffers cached
>> Mem: 7984 7828 156 0 38 7349
>> -/+ buffers/cache: 440 7544
>> Swap: 509 1 508
>
> Not far off - free is showing 7349MB cached. You're not running 350
> clients there though - you're only using 440MB of RAM.
>
>
> I don't see anything to show a performance problem from these emails.
>

--
<http://www.1st-affiliation.fr>

*David Bigand
*Président Directeur Générale*
*51 chemin des moulins
73000 CHAMBERY - FRANCE

Web : htttp://www.1st-affiliation.fr
Email : david@1st-affiliation.com
Tel. : +33 479 696 685
Mob. : +33 666 583 836
Skype : firstaffiliation_support


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: [HACKERS] Visibility Groups

On Thu, 2008-08-07 at 13:30 +0100, Gregory Stark wrote:

> Hm, so backing up a bit from the specific proposed interface, the key here is
> being able to explicitly mark which tables your transaction will need in the
> future?

Think of it as a promise to touch nothing except a specific set of
tables. Whether you lock/access the full set of tables in the visibility
group is up to you.

Visibility groups would likely have overhead, so specifying them more
loosely would allow reasonable administration, yet retain benefit.

> Is it always just a handful of heavily updated tables that you want to
> protect?

Possibly. It seems easier to specify what you might touch, since
security may prevent you even knowing about the existence of other
tables.

> In that case we could have a lock type which means "I'll never need
> to lock this object". Then a session could issue "LOCK TABLE foo IN
> INACCESSIBLE MODE" or something like that. That requires people to hack up
> their pg_dump or replication script though which might be awkward.

Possibly

The main point is this information needs to be available on PGPROC, so
that new snapshots can see the visibility groups and then calculate
OldestXmin for each object as a result.

> Perhaps the way to do that would be to preemptively take locks on all the
> objects that you'll need, then have a command to indicate you won't need any
> further objects beyond those.

I seem to recall we track which objects we have already locked. Perhaps
we would use the explicit visibility group to mark all objects we are
allowed to lock, or mark those to avoid, depending upon which is smaller
list. Then any attempt to lock a new object would be straightforwardly
refused. We must do that cheaply and so that precludes any use of shared
resources such as the lock table.

If you'd like to run with this, I doubt I will have time to do anything
with it this release.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Visibility Groups

Simon Riggs wrote:
> On Thu, 2008-08-07 at 12:55 +0200, Jochem van Dieten wrote:
>> On Thu, Aug 7, 2008 at 12:38 PM, Simon Riggs wrote:
>>> I propose creating "Visibility Groups" that *explicitly* limit the
>>> ability of a transaction to access data outside its visibility group(s).
>> Doesn't every transaction need to access data from the catalogs?
>> Wouldn't the inclusion of a catalogs visibility group in every
>> transaction negate any potential benefits?
>
> True, but I don't see the catalogs as frequently updated data. The
> objective is to isolate frequently updated tables from long running
> statements that don't need to access them.
>
> Tables can be in multiple visibility groups, perhaps that wasn't clear.
> When we seek to vacuum a table, we take the lowest xmin of any group it
> was in when we took snapshot.

I'm not sure if "visibility group" is the best name for this - I had to
go away and think through what you meant about that last bit. Have I got
this right?

So - a "visibility group" is attached to a transaction.

My long-running transaction T0 can restrict itself to <catalogues> and
table "event_log".

Various other transactions T1..Tn make no promises about what they are
going to access. They all share the "null visibility group".

A table "user_emails" is in the "null visibility group" and can be
vacuumed based on whatever the lowest xid of T1..Tn is.

Table "event_log" is in both groups and can only be vacuumed based on
T0..Tn (presumably T0 is the oldest, since that's the point of the
exercise).

An attempt to write to user_emails by T0 will fail with an error.

An attempt to read from user_emails by T0 will be allowed?

What happens if I'm in ISOLATION LEVEL SERIALIZABLE? Presumably the read
is disallowed then too?

--
Richard Huxton
Archonet Ltd

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[GENERAL] Need info on installer

Hi,
Greetings!!!

We have developed a software which uses PostgreSQL as one of the component.
We would like to delever PostgreSQL along with our software as a single package. So we have to create an installer which installs PostgreSQL with all pre-install configuration first and then our software.

Could you help me how to create a single installer? could you send PostgreSQL install script.

Thanks
Ram
Monster

[BUGS] Re: BUG #4343: upper, lower, initcap do not work with umlaute? (important!)

On Aug 6, 5:06 pm, kaeptn.hadd...@gmail.com ("paull") wrote:
> The following bug has been logged online:
>
> Bug reference:      4343
> Logged by:          paull
> Email address:      kaeptn.hadd...@gmail.com
> PostgreSQL version: 8.2? not sure
> Operating system:   ubuntu
> Description:        upper, lower, initcap do not work with umlaute?
> (important!)
> Details:
>
> Hello
>
> Sorry; I forgot to mention, that the Database is encoded in SQLASCII, so I
> suspect, that this is the source of the problem. Probably noone can fix it
> then.
> Sorry for creating unnecessary reports.
>
> Best regards
>
> ---------------------------------------
>
> I suspect this is no news, but then again, maybe noone ever bothered to
> report?
>
> I suspect that pgsql functions upper, lower and initcap do not support
> umlaute. If I query in phppgadmin like this:
>
> select
> initcap('xxx'),upper('xxx'),initcap('öÖöÖäÄäÄä'),upper('öÖöÖä­
> äÄä')
>
> the result is this:
> Xxx     XXX     öÖöÖäÄäÄä      öÖöÖäÄäÄä
>
> Hope this helps you
> Best regards
>
> --
> Sent via pgsql-bugs mailing list (pgsql-b...@postgresql.org)
> To make changes to your subscription:http://www.postgresql.org/mailpref/pgsql-bugs

see BUG #4319

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Re: [PERFORM] Query Plan choice with timestamps

Giorgio Valoti wrote:
>
> On 07/ago/08, at 10:35, Richard Huxton wrote:
>
>> Giorgio Valoti wrote:
>>> Hi, I have a timestamptz field that I want to use with a query, but I
>>> don't need the full timestamp resolution, so I've created a
>>> day_trunc(timestamptz) immutable function which I'll use with the
>>> query and with a new index:
>>> logs=> create index test_idx on blackbox (day_trunc(ts));
>>> However, the query plan doesn't use the index:
>>
>> Does it use it ever? e.g. with
>> SELECT * FROM blackbox WHERE day_trunk(ts) = '...'
>
> It's used:
[snip]

OK - so the index is working.

If you disable seq-scans before running the query, does it use it then?

SET enable_seqscan = off;

> logs=>explain select count(*) from blackbox group by day_trunc(ts) order
> by day_trunc(ts);
> QUERY PLAN
>
------------------------------------------------------------------------------------------

>
> GroupAggregate (cost=98431.58..119773.92 rows=74226 width=8)

In particular:
1. Is the estimated cost more or less than 119773.92?
2. How does that match the actual time taken?

--
Richard Huxton
Archonet Ltd

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: [BUGS] BUG #4342: upper, lower, initcap do not work with umlaute?

On Aug 6, 4:48 pm, kaeptn.hadd...@gmail.com ("paull") wrote:
> The following bug has been logged online:
>
> Bug reference:      4342
> Logged by:          paull
> Email address:      kaeptn.hadd...@gmail.com
> PostgreSQL version: 8.2? not sure
> Operating system:   ubuntu
> Description:        upper, lower, initcap do not work with umlaute?
> Details:
>
> Hello
>
> I suspect this is no news, but then again, maybe noone ever bothered to
> report?
>
> I suspect that pgsql functions upper, lower and initcap do not support
> umlaute. If I query in phppgadmin like this:
>
> select
> initcap('xxx'),upper('xxx'),initcap('öÖöÖäÄäÄä'),upper('öÖöÖä­
> äÄä')
>
> the result is this:
> Xxx     XXX     öÖöÖäÄäÄä      öÖöÖäÄäÄä
>
> Hope this helps you
> Best regards
>
> --
> Sent via pgsql-bugs mailing list (pgsql-b...@postgresql.org)
> To make changes to your subscription:http://www.postgresql.org/mailpref/pgsql-bugs

locale of the database should be de_DE.utf8 (if the database is with
encoding UNICODE) or de_DE in case the encoding is LATIN1.

check it with

show lc_ctype
show database_encoding

You cannot officially change the locale of the database so
theoretically you can buckup and restore the database to the correctly
initialized database.

I have managed to chenge the control file and reindex the database and
do not see any problems yet.

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Re: [PERFORM] Query Plan choice with timestamps

On 07/ago/08, at 10:35, Richard Huxton wrote:

> Giorgio Valoti wrote:
>> Hi, I have a timestamptz field that I want to use with a query, but
>> I don't need the full timestamp resolution, so I've created a
>> day_trunc(timestamptz) immutable function which I'll use with the
>> query and with a new index:
>> logs=> create index test_idx on blackbox (day_trunc(ts));
>> However, the query plan doesn't use the index:
>
> Does it use it ever? e.g. with
> SELECT * FROM blackbox WHERE day_trunk(ts) = '...'

It's used:

logs=> explain select * from blackbox where day_trunc(ts) =
day_trunc(now());
QUERY PLAN
---------------------------------------------------------------------------
Bitmap Heap Scan on blackbox (cost=22.38..3998.43 rows=1239
width=264)
Recheck Cond: (day_trunc(ts) = day_trunc(now()))
-> Bitmap Index Scan on date_idx (cost=0.00..22.07 rows=1239
width=0)
Index Cond: (day_trunc(ts) = day_trunc(now()))

--
Giorgio Valoti
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: [HACKERS] Visibility Groups

"Simon Riggs" <simon@2ndquadrant.com> writes:

> Currently, we calculate a single OldestXmin across all snapshots on the
> assumption that any transaction might access any table.
>
> I propose creating "Visibility Groups" that *explicitly* limit the
> ability of a transaction to access data outside its visibility group(s).
> By default, visibility_groups would be NULL, implying potential access
> to all tables.
>
> Once set, any attempt to lock an object outside of a transactions
> defined visibility_groups will result in an error:
> ERROR attempt to lock table outside of visibility group(s): foo
> HINT you need to set a different value for visibility_groups
> A transaction can only ever reduce or restrict its visibility_groups, it
> cannot reset or add visibility groups.

Hm, so backing up a bit from the specific proposed interface, the key here is
being able to explicitly mark which tables your transaction will need in the
future?

Is it always just a handful of heavily updated tables that you want to
protect? In that case we could have a lock type which means "I'll never need
to lock this object". Then a session could issue "LOCK TABLE foo IN
INACCESSIBLE MODE" or something like that. That requires people to hack up
their pg_dump or replication script though which might be awkward.

Perhaps the way to do that would be to preemptively take locks on all the
objects that you'll need, then have a command to indicate you won't need any
further objects beyond those.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's Slony Replication support!

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[BUGS] BUG #4345: ERROR: OUT OF MEMORY

The following bug has been logged online:

Bug reference: 4345
Logged by: Glaucio Pedro
Email address: gpamaster@gmail.com
PostgreSQL version: 8.2
Operating system: Linux
Description: ERROR: OUT OF MEMORY
Details:

Dear Fellow,

I have been trying to restore a file through the pgAdmin (Version:1.8.2)and
the following message has been exhibited (although the metioned file´s size
is roughly 20kb):
"pg_restore [arquive] out of memory"


What is the solution for this problem?

Best Regards,

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

[HACKERS] For what should pg_stop_backup wait?

Hi

In HEAD, pg_stop_backup waits until the history file has been archived.
But, in order to ensure that the last wal file was archived, pg_stop_backup
should wait until not only the history file but also the backup stopping wal
file has been archived, I think.

Because the alphabetic order of the history file and the backup stopping
wal file is not constant.

If the backup starting wal file is the same as the stopping one (that is,
any wal files were not switched during backup), the history file whose
name consists of the starting one is behind the stopping one.
Otherwise, the backup stopping wal file is behind the history file.

Is this worth fixing?

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] patch: Add columns via CREATE OR REPLACE VIEW

Index: doc/src/sgml/ref/create_view.sgml
===================================================================
RCS file: /projects/cvsroot/pgsql/doc/src/sgml/ref/create_view.sgml,v
retrieving revision 1.36
diff -c -r1.36 create_view.sgml
*** doc/src/sgml/ref/create_view.sgml 3 Jul 2007 01:30:35 -0000 1.36
--- doc/src/sgml/ref/create_view.sgml 7 Aug 2008 11:22:33 -0000
***************
*** 36,44 ****

<para>
<command>CREATE OR REPLACE VIEW</command> is similar, but if a view
! of the same name already exists, it is replaced. You can only replace
! a view with a new query that generates the identical set of columns
! (i.e., same column names and data types).
</para>

<para>
--- 36,45 ----

<para>
<command>CREATE OR REPLACE VIEW</command> is similar, but if a view
! of the same name already exists, it is replaced. The new query must
! generate all of the same columns that were generated by the original query
! in the same order and with the same data types, but may add additional
! columns to the end of the list.
</para>

<para>
Index: src/backend/commands/tablecmds.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/commands/tablecmds.c,v
retrieving revision 1.261
diff -c -r1.261 tablecmds.c
*** src/backend/commands/tablecmds.c 16 Jul 2008 19:33:25 -0000 1.261
--- src/backend/commands/tablecmds.c 7 Aug 2008 11:22:39 -0000
***************
*** 2327,2332 ****
--- 2327,2338 ----
ATPrepAddColumn(wqueue, rel, recurse, cmd);
pass = AT_PASS_ADD_COL;
break;
+ case AT_AddColumnToView: /* add column via CREATE OR REPLACE VIEW */
+ ATSimplePermissions(rel, true);
+ /* Performs own recursion */
+ ATPrepAddColumn(wqueue, rel, recurse, cmd);
+ pass = AT_PASS_ADD_COL;
+ break;
case AT_ColumnDefault: /* ALTER COLUMN DEFAULT */

/*
***************
*** 2548,2553 ****
--- 2554,2560 ----
switch (cmd->subtype)
{
case AT_AddColumn: /* ADD COLUMN */
+ case AT_AddColumnToView: /* add column via CREATE OR REPLACE VIEW */
ATExecAddColumn(tab, rel, (ColumnDef *) cmd->def);
break;
case AT_ColumnDefault: /* ALTER COLUMN DEFAULT */
Index: src/backend/commands/view.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/commands/view.c,v
retrieving revision 1.106
diff -c -r1.106 view.c
*** src/backend/commands/view.c 19 Jun 2008 00:46:04 -0000 1.106
--- src/backend/commands/view.c 7 Aug 2008 11:22:39 -0000
***************
*** 174,181 ****
Assert(relation->istemp == rel->rd_istemp);

/*
* Create a tuple descriptor to compare against the existing view, and
! * verify it matches.
*/
descriptor = BuildDescForRelation(attrList);
checkViewTupleDesc(descriptor, rel->rd_att);
--- 174,206 ----
Assert(relation->istemp == rel->rd_istemp);

/*
+ * If new attributes have been added, we must modify the pre-existing
+ * view.
+ */
+ if (list_length(attrList) > rel->rd_att->natts) {
+ List *atcmds = NIL;
+ ListCell *c;
+ int skip = rel->rd_att->natts;
+
+ foreach(c, attrList) {
+ AlterTableCmd *atcmd;
+
+ if (skip > 0) {
+ --skip;
+ continue;
+ }
+ atcmd = makeNode(AlterTableCmd);
+ atcmd->subtype = AT_AddColumnToView;
+ atcmd->def = lfirst(c);
+ atcmds = lappend(atcmds, atcmd);
+ }
+ AlterTableInternal(viewOid, atcmds, true);
+ }
+
+ /*
* Create a tuple descriptor to compare against the existing view, and
! * verify that the old column list is an initial prefix of the new
! * column list.
*/
descriptor = BuildDescForRelation(attrList);
checkViewTupleDesc(descriptor, rel->rd_att);
***************
*** 220,232 ****
{
int i;

! if (newdesc->natts != olddesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
! errmsg("cannot change number of columns in view")));
/* we can ignore tdhasoid */

! for (i = 0; i < newdesc->natts; i++)
{
Form_pg_attribute newattr = newdesc->attrs[i];
Form_pg_attribute oldattr = olddesc->attrs[i];
--- 245,257 ----
{
int i;

! if (newdesc->natts < olddesc->natts)
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
! errmsg("cannot drop columns from view")));
/* we can ignore tdhasoid */

! for (i = 0; i < olddesc->natts; i++)
{
Form_pg_attribute newattr = newdesc->attrs[i];
Form_pg_attribute oldattr = olddesc->attrs[i];
***************
*** 235,241 ****
if (newattr->attisdropped != oldattr->attisdropped)
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
! errmsg("cannot change number of columns in view")));

if (strcmp(NameStr(newattr->attname), NameStr(oldattr->attname)) != 0)
ereport(ERROR,
--- 260,266 ----
if (newattr->attisdropped != oldattr->attisdropped)
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
! errmsg("cannot drop columns from view")));

if (strcmp(NameStr(newattr->attname), NameStr(oldattr->attname)) != 0)
ereport(ERROR,
Index: src/backend/parser/parse_utilcmd.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/parser/parse_utilcmd.c,v
retrieving revision 2.14
diff -c -r2.14 parse_utilcmd.c
*** src/backend/parser/parse_utilcmd.c 16 Jul 2008 01:30:22 -0000 2.14
--- src/backend/parser/parse_utilcmd.c 7 Aug 2008 11:22:41 -0000
***************
*** 1719,1724 ****
--- 1719,1725 ----
switch (cmd->subtype)
{
case AT_AddColumn:
+ case AT_AddColumnToView:
{
ColumnDef *def = (ColumnDef *) cmd->def;

Index: src/include/nodes/parsenodes.h
===================================================================
RCS file: /projects/cvsroot/pgsql/src/include/nodes/parsenodes.h,v
retrieving revision 1.371
diff -c -r1.371 parsenodes.h
*** src/include/nodes/parsenodes.h 7 Aug 2008 01:11:51 -0000 1.371
--- src/include/nodes/parsenodes.h 7 Aug 2008 11:22:43 -0000
***************
*** 907,912 ****
--- 907,913 ----
typedef enum AlterTableType
{
AT_AddColumn, /* add column */
+ AT_AddColumnToView, /* implicitly via CREATE OR REPLACE VIEW */
AT_ColumnDefault, /* alter column default */
AT_DropNotNull, /* alter column drop not null */
AT_SetNotNull, /* alter column set not null */
Index: src/test/regress/expected/create_view.out
===================================================================
RCS file: /projects/cvsroot/pgsql/src/test/regress/expected/create_view.out,v
retrieving revision 1.13
diff -c -r1.13 create_view.out
*** src/test/regress/expected/create_view.out 11 Jun 2008 21:53:49 -0000 1.13
--- src/test/regress/expected/create_view.out 7 Aug 2008 11:22:45 -0000
***************
*** 49,63 ****
-- should fail
CREATE OR REPLACE VIEW viewtest AS
SELECT a FROM viewtest_tbl WHERE a <> 20;
! ERROR: cannot change number of columns in view
-- should fail
CREATE OR REPLACE VIEW viewtest AS
SELECT 1, * FROM viewtest_tbl;
! ERROR: cannot change number of columns in view
-- should fail
CREATE OR REPLACE VIEW viewtest AS
SELECT a, b::numeric FROM viewtest_tbl;
ERROR: cannot change data type of view column "b"
DROP VIEW viewtest;
DROP TABLE viewtest_tbl;
-- tests for temporary views
--- 49,66 ----
-- should fail
CREATE OR REPLACE VIEW viewtest AS
SELECT a FROM viewtest_tbl WHERE a <> 20;
! ERROR: cannot drop columns from view
-- should fail
CREATE OR REPLACE VIEW viewtest AS
SELECT 1, * FROM viewtest_tbl;
! ERROR: column "b" of relation "viewtest" already exists
-- should fail
CREATE OR REPLACE VIEW viewtest AS
SELECT a, b::numeric FROM viewtest_tbl;
ERROR: cannot change data type of view column "b"
+ -- should work
+ CREATE OR REPLACE VIEW viewtest AS
+ SELECT a, b, 0 AS c FROM viewtest_tbl;
DROP VIEW viewtest;
DROP TABLE viewtest_tbl;
-- tests for temporary views
Index: src/test/regress/sql/create_view.sql
===================================================================
RCS file: /projects/cvsroot/pgsql/src/test/regress/sql/create_view.sql,v
retrieving revision 1.7
diff -c -r1.7 create_view.sql
*** src/test/regress/sql/create_view.sql 7 Apr 2005 01:51:41 -0000 1.7
--- src/test/regress/sql/create_view.sql 7 Aug 2008 11:22:45 -0000
***************
*** 61,66 ****
--- 61,70 ----
CREATE OR REPLACE VIEW viewtest AS
SELECT a, b::numeric FROM viewtest_tbl;

+ -- should work
+ CREATE OR REPLACE VIEW viewtest AS
+ SELECT a, b, 0 AS c FROM viewtest_tbl;
+
DROP VIEW viewtest;
DROP TABLE viewtest_tbl;

Here's a patch that allows CREATE OR REPLACE VIEW to add new columns
to an existing view.

Any feedback would be appreciated, especially if it meant that I could
fix any problems before the next commitfest.

Thanks,

...Robert

Re: [HACKERS] Visibility Groups

On Thu, 2008-08-07 at 12:55 +0200, Jochem van Dieten wrote:
> On Thu, Aug 7, 2008 at 12:38 PM, Simon Riggs wrote:
> > I propose creating "Visibility Groups" that *explicitly* limit the
> > ability of a transaction to access data outside its visibility group(s).
>
> Doesn't every transaction need to access data from the catalogs?
> Wouldn't the inclusion of a catalogs visibility group in every
> transaction negate any potential benefits?

True, but I don't see the catalogs as frequently updated data. The
objective is to isolate frequently updated tables from long running
statements that don't need to access them.

Tables can be in multiple visibility groups, perhaps that wasn't clear.
When we seek to vacuum a table, we take the lowest xmin of any group it
was in when we took snapshot.

e.g. Long running statement accesses table L, so must access table L and
catalog tables only. We set this explicitly. Frequently updated table F
is accessed by general transactions that have set no visibility group,
i.e. the implicit group is "all tables".

So catalog and table L would be in in two groups, while F in only one.
As a result, the xmin used for table F will be later than the one used
for table L because the long running statement's transaction is not
included in the calculation of the xmin for table F. The transaction
accessing L has explicitly defined the limit of its access, so removing
rows from F is possible without breaking MVCC.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] looking for psql without server-installation

hallo magnus,

thanks for your answer.

> Just run the "make" and "make install" commands in subdirs
> only. You will need to do it in src/interfaces/libpq and
> src/bin/psql to get a working psql.

that is the answer i was looking for.

> But you can certainly have control over the version on the
> client even if you use binary packages like .RPMs. Just
> decide which version you install. The OS may come with a
> pre-selected version by default, but for most OSes there are
> RPMs or DEBs available for other versions as well that you
> can install.

the available versions are very different for every os, every version of
the os, and so on. so i don't prefer this method but sometimes (or for
other people maybe often) it works

best regards
christian


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[HACKERS] Infrastructure changes for recovery

I would like to propose some changes to the infrastructure for recovery.
These changes are beneficial in themselves, but also form the basis for
other work we might later contemplate.

Currently
* the startup process performs restartpoints during recovery
* the death of the startup process is tied directly to the change of
state in the postmaster following recovery

I propose to
* have startup process signal postmaster when it starts Redo phase (if
it starts it)
* have startup process signal postmaster again when it has completed
recovery, so that the change of state is via explicit signal rather than
death of the child process

Decoupling things in this way allows us to
1. arrange for the bgwriter to start during Redo, so it can:
i) clean dirty blocks for the startup process
ii) perform restartpoints in background
Both of these aspects will increase performance of recovery

2. provide a starting point for other changes in both startup process
and postmaster. These would include
i) have startup process do other work after startup completes, such as
executing transactions to rebuild damaged indexes, etc

ii) have postmaster allow connections while Redo is taking place, as one
part of allowing query access to standby database

The above two points have not been discussed and require separate
justification. However, any work on them is impossible without these
infrastructure changes.

These changes are part of a general strategy of moving in beneficial
steps towards various other goals, rather than attempting to create a
super-patch on 1 Nov that conflicts with other patches incoming at that
time. These parts are likely to conflict with synch replication work, so
I want to resolve as much as possible on Sept Commitfest.

The patch would include the required changes for bgwriter also.

Any objections/alterations to these proposals, please?

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] looking for psql without server-installation

hi tomasz,
thanks for your answers. unfortunately i misworded my problem again. i have a very clear problem and i am looking for a solution of this problem (if it is basic or not) and i am not using RedHatEL/CentOS/Fedora. i am looking for a solution which is applicable to every linux os and not only for some. so here again my problem:

i am looking for the possibility to install a psql client (without a server) FROM THE SOURCE. my usecase is: one server with a postgresql-server (self compiled) and several workstations with psql-clients, also preferred self compiled.
i know that is possible to compile at every workstation the sourcecode of the server to get a psql-client. but the side effect is, that in this case i have a potential server at every workstation and in my opinion that is a very large footprint. i have to compile psql for myself because otherwise (installing binaries with packaging tools) i have no (or minimal) control about the version of the client.
i think that is a use case which can be applied to the needs of many people (for example: oracle offers also client-only-packages which are containing sql+, ...)
regards and please excuse my unclear question
christian
p.s. i don't want to change to RedHatEL/CentOS/Fedora

> -----Ursprüngliche Nachricht-----
> Von: Tomasz Ostrowski [mailto:tometzky@batory.org.pl]
> Gesendet: Donnerstag, 7. August 2008 13:01
> An: Strobl, Christian
> Cc: pgsql-general@postgresql.org
> Betreff: Re: AW: looking for psql without server-installation
>
> On 2008-08-07 11:49, Christian.Strobl@dlr.de wrote:
>
> > maybe i misworded my problem. i am looking for the possibility to
> > install a psql client (without a server) out of the sources.
>
> It is not a basic problem - it is your solution for a
> problem, which maybe does have another, better solution.
>
> > i have to compile psql for myself because otherwise (installing
> > binaries with packaging tools) i have no (or minimal) control about
> > the version of the client.
>
> You can use official yum repository and you'll have total
> control of client version. It works for RedHatEL/CentOS/Fedora:
> http://yum.pgsqlrpms.org/reporpms/repoview/
>
> Regards
> Tometzky
> --
> ...although Eating Honey was a very good thing to do, there
> was a moment just before you began to eat it which was better
> than when you were...
> Winnie the Pooh
>

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[GENERAL] How to get many data at once?

Hi, if I have such a table:

t_ref_id     t_data
--------------------
1             'abc'
2             '321'
1             'ddd'
2             'xyz'
9             '777'
...


I want to get data with a special t_ref_id:

SELECT t_data FROM THETABLE WHERE t_ref_id = '1';

I must use a while loop to extract the data (I'm using PHP):

$rows = array();
while (($row = pgsql_fetch_assoc($result) !== false) {
    $rows[] = $row;
}

And if there are many matched rows, such as many hundreds or thousands of rows, I think such a loop maybe inefficient.

How to do this in a more efficient way?

Thank you!



--
ddh

Re: [GENERAL] looking for psql without server-installation

On 2008-08-07 11:49, Christian.Strobl@dlr.de wrote:

> maybe i misworded my problem. i am looking for the possibility to
> install a psql client (without a server) out of the sources.

It is not a basic problem - it is your solution for a problem, which
maybe does have another, better solution.

> i have to compile psql for myself because otherwise (installing
> binaries with packaging tools) i have no (or minimal) control about
> the version of the client.

You can use official yum repository and you'll have total control of
client version. It works for RedHatEL/CentOS/Fedora:
http://yum.pgsqlrpms.org/reporpms/repoview/

Regards
Tometzky
--
...although Eating Honey was a very good thing to do, there was a
moment just before you began to eat it which was better than when you
were...
Winnie the Pooh

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] looking for psql without server-installation

Christian.Strobl@dlr.de wrote:
> maybe i misworded my problem. i am looking for the possibility to install a psql client (without a server) out of the sources. my usecase is: one server with a postgresql-server (self compiled) and several workstations with psql-clients, also preferred self compiled.
> i know that is possible to compile at every workstation the sourcecode of the server to get a psql-client. but the side effect is, that in this case i have a potential server at every workstation and in my opinion that is a very large footprint. i have to compile psql for myself because otherwise (installing binaries with packaging tools) i have no (or minimal) control about the version of the client.
> i think that is a use case which can be applied to the needs of many people (for example: oracle offers also client-only-packages which are containing sql+, ...)
> hoping for a solution

Just run the "make" and "make install" commands in subdirs only. You
will need to do it in src/interfaces/libpq and src/bin/psql to get a
working psql.

But you can certainly have control over the version on the client even
if you use binary packages like .RPMs. Just decide which version you
install. The OS may come with a pre-selected version by default, but for
most OSes there are RPMs or DEBs available for other versions as well
that you can install.

//Magnus


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] Visibility Groups

On Thu, Aug 7, 2008 at 12:38 PM, Simon Riggs wrote:
> I propose creating "Visibility Groups" that *explicitly* limit the
> ability of a transaction to access data outside its visibility group(s).

Doesn't every transaction need to access data from the catalogs?
Wouldn't the inclusion of a catalogs visibility group in every
transaction negate any potential benefits?


> Once set, any attempt to lock an object outside of a transactions
> defined visibility_groups will result in an error:

Or is this about locking and not about data access?

Jochem

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [PERFORM] Plz Heeeelp! performance settings

dforums wrote:
> vmstat is giving :
> procs -----------memory---------- ---swap-- -----io---- --system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy
> id wa
> 0 2 1540 47388 41684 7578976 0 0 131 259 0 1 9
> 3 82 7

This system is practically idle. Either you're not measuring it at a
useful time, or there isn't a performance problem.

> > > But
> >> if I use a second machine to replicate the database, I escape this
> >> problem isn't it ?
> > You reduce the chance of a single failure causing disaster.
> Not clear this reply. It's scare me ....

If server A fails, you still have server B. If server A fails so that
replication stops working and you don't notice, server B won't help any
more.

> > What do you mean by "take 6Go per week"? You update/delete that much
> > data? It's growing by that amount each week?
> YES

That wasn't a yes/no question. Please choose one of:
Are you updating 6Go per week?
Are you adding 6Go per week?

> > I'm not sure what "15000 request per 2 minutes and empty it into 10 min"
> > means.
> I insert 15000 datas every 2 min and delete 15000 every 10 min in those
> tables
> >
> > Do you have 7500 requests per minute?
> should be that, But in fact I'm not treating the datas in real time, and
> I buffer the datas and push the data into the database every 2 min
> > Are these updates?
> during the delete the data are aggregated in other tables which make
> updates

OK, so every 2 minutes you run one big query that adds 15000 rows.
Every 10 minutes you run one big query that deletes 15000 rows.

> > To the "temporary storage"?
>
> > What is this "temporary storage" - an ordinary table?
> Yes, I thied to use temporary tables but I never been able to connect
> this tables over 2 different session/connection, seems that is a
> functionnality of postgresql, or a misunderstanding from me.

That's correct - temporary tables are private to a backend (connection).

> > > I'm making some update or select on tables including more than 20
> > > millions of entrance.
> >
> > Again, I'm not sure what this means.
>
> To aggregate the data, I have to check the presence of others
> information that are stores in 2 tables which includes 24 millions of
> entrance.

OK. I assume you're happy with the plans you are getting on these
queries, since you've not provided any information about them.

> > Oh - *important* - which version of PostgreSQL are you running?
> 8.1.11
> > Is an upgrade practical?
> We are working of trying to upgrade to 8.3.3, but we are not yet ready
> for such migration

OK

> > Looking at your postgresql.conf settings:
> >
> > max_connections = 624
> >
> > That's an odd number.
> Now we could decrease this number, it's not so much usefull for now. we
> could decrease is to 350.

I don't believe you've got 350 active connections either. It will be
easier to help if you can provide some useful information.

> > effective_cache_size = 625000
> >
> > That's around 5GB - is that roughly the amount of memory used for
> > caching (what does free -m say for buffers/cache)?
> total used free shared buffers cached
> Mem: 7984 7828 156 0 38 7349
> -/+ buffers/cache: 440 7544
> Swap: 509 1 508

Not far off - free is showing 7349MB cached. You're not running 350
clients there though - you're only using 440MB of RAM.


I don't see anything to show a performance problem from these emails.

--
Richard Huxton
Archonet Ltd

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

[HACKERS] Visibility Groups

Currently, we calculate a single OldestXmin across all snapshots on the
assumption that any transaction might access any table.

I propose creating "Visibility Groups" that *explicitly* limit the
ability of a transaction to access data outside its visibility group(s).
By default, visibility_groups would be NULL, implying potential access
to all tables.

Once set, any attempt to lock an object outside of a transactions
defined visibility_groups will result in an error:
ERROR attempt to lock table outside of visibility group(s): foo
HINT you need to set a different value for visibility_groups
A transaction can only ever reduce or restrict its visibility_groups, it
cannot reset or add visibility groups.

This would give us the ability to explicitly prevent long running
statements from interfering with updates of critical tables, when those
tables are not accessed.

The visibility_groups list would be held on each PGPROC, so that a
snapshot will be able to calculate an xmin for each visibility group.
When checking visibility of rows using HeapTupleSatisfiesVacuum() we
would use the oldest xmin for the visibility group of the table, rather
than the single global value.

If no visibility groups are in use then everything happens just as it
does now, with only a single "if" test in GetSnapshotData() and
HeapTupleSatisfiesVacuum().

Patch would require providing info through to HeapTupleSatisfiesVacuum()
to allow it to resolve the appropriate xmin to use for visibility
checks, since it will have more than one to choose from. Various ways of
doing that.

We might like the visibility group to be set automatically but that
seems like a harder problem. I do not propose to solve that here. This
general idea has been proposed before, but we always get hung up on our
inability to automatically determine the visibility group. Let's just do
this explicitly, so we can get benefit in the cases where we know very
clearly which tables we'll access and more importantly, which we won't.

How do we specify visibility groups? Well various ways, various syntax,
so that is up for discussion and debate. This might be the place the
concept falls down, maybe where it starts having wings.

* SET visibility_groups = table, schema.*
* define visibility groups using a function:
create_visibility_group('name', 'table/schema list')
* specify them via ALTER TABLE etc

This idea is connected somewhat to replication, so floating it now to
see how viable a concept people think this is. I'm floating the idea in
a fairly neutral way in the hope that it leads others to even
better/more workable proposals, possibly now, possibly over the next few
years.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[BUGS] BUG #4344: initdb -L timezone directory

The following bug has been logged online:

Bug reference: 4344
Logged by: Domingo Alvarez Duarte
Email address: mingodad@gmail.com
PostgreSQL version: 8.3.3
Operating system: windows
Description: initdb -L timezone directory
Details:

When specifing a non default location for input files to initialize the
database cluster with the "-L" option this option is not transfered to
timezone functions and initdb fail because it tries to find timezone files
in the hardcoded path settings done at compile time
"/usr/local/share/timezone" for example.

I expected initdb to use "-L" value to timezone too and for all other files
that is expected to be on share directory.

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Re: [ADMIN] Managing connections

C K wrote:
> Thanks for this information. We are migrating our application from
> mysql to postgresql and use MS Access/ODBC for front end.
Mystery explained. Access is notorious for opening a connection for
everything. Visual Basic used to do the same thing. I think the dot net
stuff now uses a connection pool.
> When I tested on windows, PG starts it's new instance for each
> application that uses ODBC to connect.
Remember it's the app that calls for the connection through ODBC.
Postgres simply responds to the call.
> Even we open multiple tables from a single file, it use only one
> instance, but when another application with same username even from
> same machine is started, another PG instance is started. Now we are
> testing for other applications and impact on memory.
>
> Regards,
> CPK
>
>
> On Thu, Aug 7, 2008 at 3:09 PM, H. Hall <hhall1001@reedyriver.com
> <mailto:hhall1001@reedyriver.com>> wrote:
>
> C K wrote:
>
> Dear Friends,
> I want to know if I open a table to view the data a new
> connection is opened. If I open another table/execute a query
> does a another new connection is opened? As per manual, I know
> that for a database cluster and the same port a postgresql
> instance runs. But if we run multiple SQL statements from same
> client machine with same username, does postgresql creates a
> new instance?
>
> How can connections to the db are handled is determined by the
> application program that accesses the database. Some application
> programs create a connection for each user and re-use it each time
> the user accesses the database. The app is also responsible for
> closing the connection when the user signs off.
>
> Other apps create a pool of connections and lease a connection on
> demand to each operation that accesses the db. When the operation
> is complete, the connection is returned to the pool. This is the
> most efficient way to use connections and I would recommend this
> approach for just about any program.
>
> We created our own connection pool program because we wanted a
> connection pool that we could use to access any db and because we
> considered it to be a key technology. It's a good thing that we
> did because we started out using MS SQLServer for our SaaS
> applications and migrated to Postgres. Migration was a breeze.
>
> Postgresql offers a connection pool which I have not used but
> appears to be highly regarded. You can find info about it here:
> http://pgpool.projects.postgresql.org/
>
> Some apps are not well designed and don't close connections or
> un-lease connections when they should. I call this phenomena
> "leaking connections". You can detect this by having everyone sign
> off the application and see how many connections are still open.
>
> Cheers,
> HH
>
> If yes, how to manage memory? (As per my observations,
> normally each postgresql instance takes 5-10 MB. So if we have
> to manage simultenious 1000 connections, it will require
> 1000*5MB=5GB + system memory requirements (approx. 1 GB) =
> total 6 GB (min.) Does this is correct?)
> Please give the details. I am using Win XP Pro with PostgreSQL
> 8.3.
> Thanks
> CPK
>
> --
> Keep your Environment clean and green.
>
> --------------------------------
>
>
>
>
>
> --
> H. Hall
> ReedyRiver Group LLC
> http://www.reedyriver.com
>
>
>
>
> --
> Keep your Environment clean and green.
>
> --------------------------------
>
>
>


--
H. Hall
ReedyRiver Group LLC
http://www.reedyriver.com


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [SQL] How to creat tables using record ID in for loop

Oh, I checked the function. There are some syntax errors. Right code
listed below:

CREATE OR REPLACE FUNCTION cr_tbls_by_staid() RETURNS INTEGER AS
$$
DECLARE
stid INTEGER;
q TEXT;
BEGIN
FOR stid IN SELECT staid FROM mytest LOOP
q:= 'CREATE TABLE "s' || staid || '" (staid varchar(50), val real,
dt date);';
RAISE NOTICE 'query is: %', q; --for debug
EXECUTE q;
END LOOP;
RETURN 1;
END;
$$ LANGUAGE plpgsql;

Also I forget to change first column definition in last CREATE TABLE
query. Edited variant is, for sure:
CREATE TABLE sta_descs (staid INTEGER, val real, dt date) WITHOUT OIDS;

--
Best regards. Yuri.
mailto: yuragal@gmail.com

--
Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-sql

Re: [PERFORM] file system and raid performance

Mark Kirkwood schrieb:
> Mark Kirkwood wrote:
>> You are right, it does (I may be recalling performance from my other
>> machine that has a 3Ware card - this was a couple of years ago...)
>> Anyway, I'm thinking for the Hardware raid tests they may need to be
>> specified.
>>
>>
>
> FWIW - of course this somewhat academic given that the single disk xfs
> test failed! I'm puzzled - having a Gentoo system of similar
> configuration (2.6.25-gentoo-r6) and running the fio tests a little
> modified for my config (2 cpu PIII 2G RAM with 4x ATA disks RAID0 and
> all xfs filesystems - I changed sizes of files to 4G and no. processes
> to 4) all tests that failed on Marks HP work on my Supermicro P2TDER +
> Promise TX4000. In fact the performance is pretty reasonable on the
> old girl as well (seq read is 142Mb/s and the random read/write is
> 12.7/12.0 Mb/s).
>
> I certainly would like to see some more info on why the xfs tests were
> failing - as on most systems I've encountered xfs is a great performer.
>
> regards
>
> Mark
>
I can second this, we use XFS on nearly all our database servers, and
never encountered the problems mentioned.


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: [ADMIN] Managing connections

Thanks for this information. We are migrating our application from mysql to postgresql and use MS Access/ODBC for front end. When I tested on windows, PG starts it's new instance for each application that uses ODBC to connect. Even we open multiple tables from a single file, it use only one instance, but when another application with same username even from same machine is started, another PG instance is started. Now we are testing for other applications and impact on memory.

Regards,
CPK


On Thu, Aug 7, 2008 at 3:09 PM, H. Hall <hhall1001@reedyriver.com> wrote:
C K wrote:
Dear Friends,
I want to know if I open a table to view the data a new connection is opened. If I open another table/execute a query does a another new connection is opened? As per manual, I know that for a database cluster and the same port a postgresql instance runs. But if we run multiple SQL statements from same client machine with same username, does postgresql creates a new instance?
How can connections to the db are handled is determined by the application program that accesses the database. Some application programs create a connection for each user and re-use it each time the user accesses the database. The app is also responsible for closing the connection when the user signs off.

Other apps create a pool of connections and lease a connection on demand to each operation that accesses the db. When the operation is complete, the connection is returned to the pool.  This is the most efficient way to use connections and I would recommend this approach for just about any program.

We created our own connection pool program because we wanted a connection pool that we could use to access any db and because we considered it to be a key technology.  It's a good thing that we did because we started out using MS SQLServer for our SaaS applications and migrated to Postgres.  Migration was a breeze.

Postgresql offers a connection pool which I have not used but appears to be highly regarded. You can find info about it here: http://pgpool.projects.postgresql.org/

Some apps are not well designed and don't close connections or un-lease connections when they should. I call this phenomena "leaking connections". You can detect this by having everyone sign off the application and see how many connections are still open.

Cheers,
HH
If yes, how to manage memory? (As per my observations, normally each postgresql instance takes 5-10 MB. So if we have to manage simultenious 1000 connections, it will require 1000*5MB=5GB + system memory requirements (approx. 1 GB) = total 6 GB (min.) Does this is correct?)
Please give the details. I am using Win XP Pro with PostgreSQL 8.3.
Thanks
CPK

--
Keep your Environment clean and green.

--------------------------------





--
H. Hall
ReedyRiver Group LLC
http://www.reedyriver.com




--
Keep your Environment clean and green.