Tuesday, September 30, 2008

Re: [GENERAL] Why Does UPDATE Take So Long?

Doesn't look like that's the problem. I moved my table over to another
Linux box running PG 8.3 and update performance was pretty bad there as
well. In the time that PG 8.3 was struggling with update there I created
a copy of my table on my PG 8.1 machine and inserted all columns with
one containing the altered values I wanted and that took less than two
minutes. Meanwhile, a half-hour later, my PG 8.3 machine was still
thrashing away trying to update that one column that's not even part of
any index..

Something is really wrong with UPDATE in PostgreSQL I think.


Scott Marlowe wrote:
> On Tue, Sep 30, 2008 at 1:00 PM, Bill Thoen <bthoen@gisnet.com> wrote:
>
>> Working with PG 8.1 I'm trying to update a char(4) column, and it's taking a
>> very long time; 15 minutes so far and no end in sight. From the explain, it
>> doesn't seem like it should take that long, and this column is not indexed.
>> Sure, there's 2.7 million records but it only takes a few minutes to scan
>> the whole file. Is there some special overhead I should be aware of with an
>> UPDATE? I VACUUMed and ANALYZEd first, too.
>>
>> Or am I just expecting too much?
>>
>
> The problem is that on older versions of pgsql, the db had to update
> each index for each row updated as well as the rows. The latest
> version, with a low enough fill factor, can update non-indedexed
> fields by using the free space in each page and not have to hit the
> indexes. But on 8.1 you don't get that optimization.
>
>


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[lapug] Attn:LAPUGers - Greenplum requesting Topics For Discussion

On Tue, Sep 30, 2008 at 1:21 PM, Rui Maximo <rmaximo@greenplum.com> wrote:

> As we get closer to that date, could you share logistics (address
> and time)?

Rui,

A reservation was created for 11/19/08 at 7pm, in:

Steele 214 in Caltech.
370 South Holliston Avenue,
Pasadena, CA 91106

Google map:
http://maps.google.com/maps/ms?ie=UTF8&t=h&hl=en&msa=0&ll=34.139186,-118.123265&spn=0.001592,0.002403&z=19&msid=114588631836793408373.00045822d679f33626b04


Caltech map:
http://www.caltech.edu/map/Caltech-map-2007-11-14.pdf

LAPUGers,

> Since I have a rare opportunity to be able to connect directly with your
> audience beforehand, I would encourage you to please voice what you would
> like to hear about Greenplum. This will help me come as prepared to answer
> your questions.
>
> Thank you,
> Rui Maximo
> Senior Product Manager
> Greenplum Inc.


Please reply to Rui regarding any Data warehouse, greenplum, OLAP
question that you might like him to address.

--
Regards,
Richard Broersma Jr.

Visit the Los Angeles PostgreSQL Users Group (LAPUG)
http://pugs.postgresql.org/lapug

--
Sent via lapug mailing list (lapug@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/lapug

Re: [HACKERS] Block-level CRC checks

Alvaro Herrera wrote:
> A customer of ours has been having trouble with corrupted data for some
> time. Of course, we've almost always blamed hardware (and we've seen
> RAID controllers have their firmware upgraded, among other actions), but
> the useful thing to know is when corruption has happened, and where.
>
> So we've been tasked with adding CRCs to data files.

Maybe a stupid question, but what I/O subsystems corrupt data and fail
to report it?

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [pgsql-www] Finding some bug statistics..

Chander Ganesan wrote:
> Magnus Hagander wrote:
> >
> > We don't have a bug tracker, thus there is nothing to gather statistics
> > from. We have a web form that is really just a sequence in a database
> > that generates a bug id, and then remails the whole form to pgsql-bugs.
> >
> > For discussions of why we don't have one, see about a billion mails in
> > the archives over the past 10 years or so :-(
> >
> The general policy is a 72 hour bug fix, right? Do we have any
> ideas/numbers as to the currently "open" number of bugs (identified but
> not fixed yet)? That would be just as useful in some regard.

As far as I am concerned, the TODO list has every known bug against CVS
HEAD, with lots of feature requests in there too.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

--
Sent via pgsql-www mailing list (pgsql-www@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

[pgsql-es-ayuda] RE: [pgsql-es-ayuda] Validación de CUIT - Argentina

Bueno gente he encontrado un código hecho en Oracle y lo adapte a plpsql

Les dejo el codigo de ejemplo:
----------------------------------------------------------------------------
--------------------
CREATE OR REPLACE FUNCTION "public"."validar_cuit" (bigint) RETURNS varchar
AS
$body$
DECLARE
RES BIGINT;
DIG BIGINT;
NUM BIGINT;
CUIT ALIAS FOR $1;

BEGIN
IF LENGTH(CUIT) != 11 OR SUBSTR(CUIT, 1, 2) = '00' THEN
RETURN 0;
END IF;
RES = 0;
FOR I IN 1..10 LOOP
NUM := (SUBSTR(CUIT, I, 1));
IF (I = 1 OR I = 7) THEN RES := RES + NUM * 5;
ELSIF (I = 2 OR I = 8) THEN RES := RES + NUM * 4;
ELSIF (I = 3 OR I = 9) THEN RES := RES + NUM * 3;
ELSIF (I = 4 OR I = 10) THEN RES := RES + NUM * 2;
ELSIF (I = 5) THEN RES := RES + NUM * 7;
ELSIF (I = 6) THEN RES := RES + NUM * 6;
END IF;
END LOOP;
DIG := 11 - MOD(RES,11);
IF DIG = 11 THEN
DIG := 0;
END IF;

IF DIG = (SUBSTR(CUIT,11,1)) THEN
RETURN 1;
ELSE
RETURN 0;
END IF;
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER;

COMMENT ON FUNCTION "public"."validar_cuit"(bigint)
IS 'Valida el CUIT segun formato 99999999999
Devuelve 0 para los CUIT No Válidos
Devuelve 1 para los CUIT Válidos';
----------------------------------------------------------------------------
--------------------

-----Mensaje original-----
De: pgsql-es-ayuda-owner@postgresql.org
[mailto:pgsql-es-ayuda-owner@postgresql.org] En nombre de Damian Culotta
Enviado el: Martes, 30 de Septiembre de 2008 05:28 p.m.
Para: pgsql-es-ayuda@postgresql.org
Asunto: Re: [pgsql-es-ayuda] Validación de CUIT - Argentina

2008/9/30 Daniel Ferrer <daniel.ferrer@ctd.com.ar>:
> Buenas Tardes a todos:
> Alguien tiene armada alguna función para validar el
> CUIT (Clave Única de Identificación Tributaria) de Argentina.
>
> Desde ya les agradezco.
>
> Atte.
> ___________________________
> ASC Daniel Ferrer
> Gerente de Sistemas - CTD SRL
> Rosario - Argentina
> mailto:daniel.ferrer@ctd.com.ar
> ___________________________
>
> --
> TIP 7: no olvides aumentar la configuración del "free space map"
>

Salvo que se haya actualizado el asunto, hasta donde recuerdo, no está
definida formalmente la función para calcular la validez de un CUIT.
Si bien hay funciones dando vueltas (yo no tengo encima ninguna en
este momento) que servirían para hacerlo, no son "oficiles".
Ojalá me esté equivocando y alquien pueda darte la referncia a un
instructivo de la AFIP.
Saludos.
--
TIP 6: ¿Has buscado en los archivos de nuestra lista de correo?
http://archives.postgresql.org/pgsql-es-ayuda

--
TIP 9: visita nuestro canal de IRC #postgresql-es en irc.freenode.net

Re: [pgsql-es-ayuda] Validación de CUIT - Argentina

2008/9/30 Daniel Ferrer <daniel.ferrer@ctd.com.ar>:
> Buenas Tardes a todos:
> Alguien tiene armada alguna función para validar el
> CUIT (Clave Única de Identificación Tributaria) de Argentina.
>
> Desde ya les agradezco.
>
> Atte.
> ___________________________
> ASC Daniel Ferrer
> Gerente de Sistemas - CTD SRL
> Rosario - Argentina
> mailto:daniel.ferrer@ctd.com.ar
> ___________________________
>
> --
> TIP 7: no olvides aumentar la configuración del "free space map"
>

Salvo que se haya actualizado el asunto, hasta donde recuerdo, no está
definida formalmente la función para calcular la validez de un CUIT.
Si bien hay funciones dando vueltas (yo no tengo encima ninguna en
este momento) que servirían para hacerlo, no son "oficiles".
Ojalá me esté equivocando y alquien pueda darte la referncia a un
instructivo de la AFIP.
Saludos.
--
TIP 6: ¿Has buscado en los archivos de nuestra lista de correo?
http://archives.postgresql.org/pgsql-es-ayuda

Re: [pgsql-es-ayuda] wxwidgets libpq error de codificacion de caracteres

Podrias pegar la salida del error o el log?
Fijate el enconding de la base también, en especial si compilaste el codigo.

El día 30 de septiembre de 2008 17:21, Pedro Mateo
<pedromateoa@gmail.com> escribió:
> hola a todos
>
> en DEBIAN
> estoy haciendo un programa en c++ con wxwidgets y con librerias de C para
> conectarse a postgres (libpq )
>
> en vez de tomar como std::string estoy usando wxString en todo el programa
> pues tiene mas funcionalidades y es la clase que maneja las cadenas en las
> wxwidgets
>
> hay una funcion de libpq que se llama PQprepared que en uno de sus
> parametros recibe un arreglo de cadenas, con un arreglo int que contiene los
> diferentes tamanos de cada cadena
>
> yo creo el arreglo de cadenas y voy copiando de instancias de wxstring para
> llenarlos, cosa que me esta saliendo bien
>
> el programa compila correctamente, pero cuando lo ejecuto me da un error de
> codificacion de caracteres
>
> parece que wxstring no maneja el misma codificacion que trabaja postgres que
> usa uft8
>
> no se mucho de esos asuntos pero imaginaba que al instalar las librerias C
> para conectase a postgres y las librerias wxwidgets asi como el posgres
> estas utilizarian los locales de DEBIAN que son uno do_es.utf8 (para
> republica dominicana ) o algo asi
>
> en fin
> agradeceria la ayuda, pues estoy trancado
>
>
>
--
TIP 8: explain analyze es tu amigo

[pgsql-es-ayuda] wxwidgets libpq error de codificacion de caracteres

hola a todos
 
en DEBIAN
estoy haciendo un programa en c++ con wxwidgets y con librerias  de C para conectarse a postgres (libpq )
 
en vez de tomar como std::string estoy usando wxString en todo el programa pues tiene mas funcionalidades y es la clase que maneja las cadenas en las wxwidgets
 
hay una funcion de libpq que se llama PQprepared que en uno de sus parametros recibe un arreglo de cadenas, con un arreglo int que contiene los diferentes tamanos de cada cadena
 
yo creo el arreglo de cadenas y voy copiando de instancias de wxstring para llenarlos, cosa que me esta saliendo bien
 
el programa compila correctamente, pero cuando lo ejecuto me da un error de codificacion de caracteres
 
parece que wxstring no maneja el misma codificacion que trabaja postgres que usa uft8
 
no se mucho de esos asuntos pero imaginaba que al instalar las librerias C para conectase a postgres y las librerias wxwidgets asi como el posgres estas utilizarian los locales de DEBIAN que son uno do_es.utf8 (para republica dominicana ) o algo asi
 
en fin
agradeceria la ayuda, pues estoy trancado
 
 
 

Re: [pgsql-es-ayuda] Insertar imagen tipo bytea

Claro como el agua! Gracias Alvaro!


El día 30 de septiembre de 2008 17:03, Alvaro Herrera
<alvherre@alvh.no-ip.org> escribió:
> postgres Emanuel CALVO FRANCO escribió:
>
>> entonces ... cual seria la funcion que inserta la imagen directamente
>> en la tabla?
>> o mejor dicho, que me permita leer una imagen y acoplarla la la query
>> de inserción.
>> o la respuesta es: 'tenes que insertarlo desde la aplicacion, parseas
>> la instrucción y listo'
>
> Efectivamente :-)
>
> Algunos lenguajes tienen soporte para expandir "placeholders" en la
> consulta, como mostraba el amigo que hizo la pregunta. En PHP (!) se ve
> algo así:
>
> $imagen = file_get_contents("/home/john/sarah_palin.png");
> pg_query("insert into friends (photo) values (?)", $imagen)
>
> (En realidad no sé si pg_query admite hacer eso; sé que PDO puede, pero
> espero que se entienda la idea).
>
> --
> Alvaro Herrera Valdivia, Chile ICBM: S 39º 48' 55.3", W 73º 15' 24.7"
> "La conclusión que podemos sacar de esos estudios es que
> no podemos sacar ninguna conclusión de ellos" (Tanenbaum)
>
--
TIP 10: no uses HTML en tu pregunta, seguro que quien responda no podrá leerlo

Re: [pgsql-es-ayuda] Insertar imagen tipo bytea

postgres Emanuel CALVO FRANCO escribió:

> entonces ... cual seria la funcion que inserta la imagen directamente
> en la tabla?
> o mejor dicho, que me permita leer una imagen y acoplarla la la query
> de inserción.
> o la respuesta es: 'tenes que insertarlo desde la aplicacion, parseas
> la instrucción y listo'

Efectivamente :-)

Algunos lenguajes tienen soporte para expandir "placeholders" en la
consulta, como mostraba el amigo que hizo la pregunta. En PHP (!) se ve
algo así:

$imagen = file_get_contents("/home/john/sarah_palin.png");
pg_query("insert into friends (photo) values (?)", $imagen)

(En realidad no sé si pg_query admite hacer eso; sé que PDO puede, pero
espero que se entienda la idea).

--
Alvaro Herrera Valdivia, Chile ICBM: S 39º 48' 55.3", W 73º 15' 24.7"
"La conclusión que podemos sacar de esos estudios es que
no podemos sacar ninguna conclusión de ellos" (Tanenbaum)
--
TIP 6: ¿Has buscado en los archivos de nuestra lista de correo?
http://archives.postgresql.org/pgsql-es-ayuda

Re: [pgsql-es-ayuda] Insertar imagen tipo bytea

copio a la lista porque le erré a la direccion....


entonces ... cual seria la funcion que inserta la imagen directamente
en la tabla?
o mejor dicho, que me permita leer una imagen y acoplarla la la query
de inserción.
o la respuesta es: 'tenes que insertarlo desde la aplicacion, parseas
la instrucción y listo'
perdón si le estoy pifiando peor todavia... ;P

El día 30 de septiembre de 2008 16:52, postgres Emanuel CALVO FRANCO
<postgres.arg@gmail.com> escribió:
> enteonces ... cual seria la funcion que inserta la imagen directamente
> en la tabla?
> o mejor dicho, que me permita leer una imagen y acoplarla la la query
> de inserción.
> o la respuesta es: 'tenes que insertarlo desde la aplicacion, parseas
> la instrucción y listo'
> perdón si le estoy pifiando peor todavia... ;P
>
> 2008/9/30 Alvaro Herrera <alvherre@alvh.no-ip.org>:
>> postgres Emanuel CALVO FRANCO escribió:
>>> Entonces le quedaria usar
>>> lo_read(PGconn *conn, int fd, char *buf, size_t len);
>>> y lo_open ?
>>>
>>> por lo que si bien import lo que hace es devolver un oid, las funciones
>>> que levantan los large objects utilizan estas funciones. o le estoy
>>> pifiando fiero?
>>
>> Le estas pifiando, porque las funciones lo_* funcionan con large
>> objects (crean, abren, cierran, escriben). La clave de mi mensaje
>> anterior es que bytea no es "large objects". No necesitas abrir y
>> cerrar un bytea, por ejemplo.
>>
>> --
>> Alvaro Herrera http://www.amazon.com/gp/registry/3BP7BYG9PUGI8
>> "Escucha y olvidarás; ve y recordarás; haz y entenderás" (Confucio)
>>
>
--
TIP 3: Si encontraste la respuesta a tu problema, publícala, otros te lo agradecerán

Re: [GENERAL] Why Does UPDATE Take So Long?

On Tue, Sep 30, 2008 at 1:00 PM, Bill Thoen <bthoen@gisnet.com> wrote:
> Working with PG 8.1 I'm trying to update a char(4) column, and it's taking a
> very long time; 15 minutes so far and no end in sight. From the explain, it
> doesn't seem like it should take that long, and this column is not indexed.
> Sure, there's 2.7 million records but it only takes a few minutes to scan
> the whole file. Is there some special overhead I should be aware of with an
> UPDATE? I VACUUMed and ANALYZEd first, too.
>
> Or am I just expecting too much?

The problem is that on older versions of pgsql, the db had to update
each index for each row updated as well as the rows. The latest
version, with a low enough fill factor, can update non-indedexed
fields by using the free space in each page and not have to hit the
indexes. But on 8.1 you don't get that optimization.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] Common Table Expressions (WITH RECURSIVE) patch

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Le 30 sept. 08 à 20:03, Tom Lane a écrit :
> set_read_position(tupstore, &local_read_position);
> tuple = tuplestore_gettuple(tupstore, ...);
> get_read_position(tupstore, &local_read_position);
>
> rather than just tuplestore_gettuple. The set/get functions will be
> cheap enough that this is no big deal. (Or maybe we should just
> provide a wrapper function that does this sequence?)

It seems to me to share some ideas with the MemoryContext concept:
what about a TupstoreContext associated with tuplestore, you get a
common default one if you don't register your own, and use
tuplestore_gettuple(MyTupstoreContext, ...);

Maybe some other API would benefit from the idea?

Regards,
- --
dim


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkjigF4ACgkQlBXRlnbh1bkycQCgqs/+JBOd0SiN4xvKwLgEgi9F
BOYAoLm0Se6zs8cEAnoTlH6de7pLLh/l
=kzm1
-----END PGP SIGNATURE-----

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Alias name from subquery

Thanks to Scott and Taras for pointing me to the crosstab functions. I
only had a quick look but they seem very helpful!

Kind regards,

Felix

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Why Does UPDATE Take So Long?

On Tue, 2008-09-30 at 13:00 -0600, Bill Thoen wrote:
> Working with PG 8.1 I'm trying to update a char(4) column, and it's
> taking a very long time; 15 minutes so far and no end in sight. From the
> explain, it doesn't seem like it should take that long, and this column
> is not indexed. Sure, there's 2.7 million records but it only takes a
> few minutes to scan the whole file. Is there some special overhead I
> should be aware of with an UPDATE? I VACUUMed and ANALYZEd first, too.
>

In PostgreSQL, think of an UPDATE as an INSERT+DELETE. It's got to write
the new versions of the tuples, and it has to keep the old versions
until there are no more transactions that might reference those old
versions. Imagine if you canceled the query halfway through, for
example. Also, it has to create new index entries for the same reason,
which is expensive.

There are some optimizations in 8.3 for when the same tuple gets updated
many times, but that won't help you in this case.

Regards,
Jeff Davis


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [pgsql-es-ayuda] Insertar imagen tipo bytea

postgres Emanuel CALVO FRANCO escribió:
> Entonces le quedaria usar
> lo_read(PGconn *conn, int fd, char *buf, size_t len);
> y lo_open ?
>
> por lo que si bien import lo que hace es devolver un oid, las funciones
> que levantan los large objects utilizan estas funciones. o le estoy
> pifiando fiero?

Le estas pifiando, porque las funciones lo_* funcionan con large
objects (crean, abren, cierran, escriben). La clave de mi mensaje
anterior es que bytea no es "large objects". No necesitas abrir y
cerrar un bytea, por ejemplo.

--
Alvaro Herrera http://www.amazon.com/gp/registry/3BP7BYG9PUGI8
"Escucha y olvidarás; ve y recordarás; haz y entenderás" (Confucio)
--
TIP 7: no olvides aumentar la configuración del "free space map"

Re: [pgsql-es-ayuda] Insertar imagen tipo bytea

Entonces le quedaria usar
lo_read(PGconn *conn, int fd, char *buf, size_t len);
y lo_open ?

por lo que si bien import lo que hace es devolver un oid, las funciones
que levantan los large objects utilizan estas funciones. o le estoy
pifiando fiero?


El día 30 de septiembre de 2008 12:18, Alvaro Herrera
<alvherre@alvh.no-ip.org> escribió:
> Linder Poclaba Lazaro escribió:
>
> Hola!
>
>> [TopLink Warning]: 2008.09.30 02:23:41.962--UnitOfWork(11595950)--Exception
>> [TOPLINK-4002] (Oracle TopLink Essentials - 2.0.1 (Build b09d-fcs
>> (12/06/2007))): oracle.toplink.essentials.exceptions.DatabaseException
>> Internal Exception: org.postgresql.util.PSQLException: ERROR: column
>> "imagen" is of type bytea but expression is of type oid
>> Error Code: 0
>> Call: INSERT INTO public.joya (estado, cliente_idcliente, cantidad,
>> peso_otros, imagen, descripcion, peso_bruto, inscripciones, tasacion,
>> idpreforma_joya, kilataje_idkilataje, tipo_joya_idtipo_joya) VALUES (?, ?,
>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
>> bind => [muy bueno, 0, 1, 1.3, null, anillo de oro, 44.0, regalo,
>> 4.0, 13, 1, 1]
>
> Estás mezclando las cosas. No debes confundir el concepto de un large
> object con una columna de tipo bytea.
>
> Una columna de tipo bytea lleva la imagen misma, es decir la secuencia
> bytes que la componen. Una columna de tipo oid lleva un OID (es decir
> un identificador numérico), que es una referencia a un "large object".
> En este último caso los bytes que forman la imagen se guardan en otra
> tabla (pg_largeobject), y debes importarlos usando lo_import.
>
> Básicamente, ese "null" que tienes ahí debe ir declarado como tipo
> bytea, no como oid.
>
> --
> Alvaro Herrera http://www.amazon.com/gp/registry/5ZYLFMCVHXC
> "La realidad se compone de muchos sueños, todos ellos diferentes,
> pero en cierto aspecto, parecidos..." (Yo, hablando de sueños eróticos)
> --
> TIP 6: ¿Has buscado en los archivos de nuestra lista de correo?
> http://archives.postgresql.org/pgsql-es-ayuda
>
--
TIP 6: ¿Has buscado en los archivos de nuestra lista de correo?
http://archives.postgresql.org/pgsql-es-ayuda

Re: [ADMIN] What process clears the logs?

Carol Walter wrote:
> Greetings,
>
> As you may be aware, we experienced a problem last week with pg_clogs
> that had been deleted, through human error it appears. What process
> will clear or delete the pg_clogs? I've been all over the
> documentation and I'm not finding a reference to this.
>
PostgreSQL takes care of this itself. I don't know where it is
documented but IIRC from old messages, the number of files in pg_xlog
should level out at about two-times your checkpoint_settings value.

I'd have to do some digging as I don't recall for sure, but I think that
if you are using archive_command setting to archive WAL files the
archive command does not return "success", then the WAL file will not be
removed/reused.

Cheers,
Steve


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [HACKERS] Block-level CRC checks

On Tue, 30 Sep 2008, Heikki Linnakangas wrote:

> Doesn't some filesystems include a per-block CRC, which would achieve the
> same thing? ZFS?

Yes, there is a popular advoacy piece for ZFS with a high-level view of
why and how they implement that at
http://blogs.sun.com/bonwick/entry/zfs_end_to_end_data The guarantees are
stronger than what you can get if you just put a CRC in the block itself.
I'd never really thought too hard about putting this in the database
knowing that ZFS is available for environments where this is a concern,
but it certainly would be a nice addition.

The best analysis I've ever seen that makes a case for OS or higher level
disk checksums of some sort, by looking at the myriad ways that disks and
disk arrays fail in the real world, is in
http://www.usenix.org/event/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf
(there is a shorter version that hits the high points of that at
http://www.usenix.org/publications/login/2008-06/openpdfs/bairavasundaram.pdf
)

One really interesting bit in there I'd never seen before is that they
find real data that supports the stand that enterprise drives are
significantly more reliable than consumer ones. While general failure
rates aren't that different, "SATA disks have an order of magnitude higher
probability of developing checksum mismatches than Fibre Channel disks. We
find that 0.66% of SATA disks develop at least one mismatch during the
first 17 months in the field, whereas only 0.06% of Fibre Channel disks
develop a mismatch during that time."

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Why Does UPDATE Take So Long?

Bill Thoen <bthoen@gisnet.com> schrieb:

> Working with PG 8.1 I'm trying to update a char(4) column, and it's
> taking a very long time; 15 minutes so far and no end in sight. From the
> explain, it doesn't seem like it should take that long, and this column
> is not indexed. Sure, there's 2.7 million records but it only takes a
> few minutes to scan the whole file. Is there some special overhead I
> should be aware of with an UPDATE? I VACUUMed and ANALYZEd first, too.
>
> Or am I just expecting too much?
>
> Here's the explain:
> explain UPDATE farms SET prog_year='2007';
> QUERY PLAN
> ----------------------------------------------------------------
> Seq Scan on farms (cost=0.00..59144.07 rows=2772207 width=54)
> (1 row)

Please provide us an EXPLAIN ANALYSE. But without a WHERE-condition a
seq-scan are logical, and PG has to rewrite the whole table and the
transaction-log.


Andreas
--
Really, I'm not out to destroy Microsoft. That will just be a completely
unintentional side effect. (Linus Torvalds)
"If I was god, I would recompile penguin with --enable-fly." (unknown)
Kaufbach, Saxony, Germany, Europe. N 51.05082°, E 13.56889°

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] Block-level CRC checks

> A customer of ours has been having trouble with corrupted data for some
> time. Of course, we've almost always blamed hardware (and we've seen
> RAID controllers have their firmware upgraded, among other actions), but
> the useful thing to know is when corruption has happened, and where.

That is an important statement, to know when it happens not necessarily to
be able to recover the block or where in the block it is corrupt. Is that
correct?

>
> So we've been tasked with adding CRCs to data files.

CRC or checksum? If the objective is merely general "detection" there
should be some latitude in choosing the methodology for performance.

>
> The idea is that these CRCs are going to be checked just after reading
> files from disk, and calculated just before writing it. They are
> just a protection against the storage layer going mad; they are not
> intended to protect against faulty RAM, CPU or kernel.

It will actually find faults in all if it. If the CPU can't add and/or a
RAM location lost a bit, this will blow up just as easily as a bad block.
It may cause "false identification" of an error, but it will keep a bad
system from hiding.

>
> This code would be run-time or compile-time configurable. I'm not
> absolutely sure which yet; the problem with run-time is what to do if
> the user restarts the server with the setting flipped. It would have
> almost no impact on users who don't enable it.

CPU capacity on modern hardware within a small area of RAM is practically
infinite when compared to any sort of I/O.
>
> The implementation I'm envisioning requires the use of a new relation
> fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
> for each block. FlushBuffer would calculate the checksum and store it
> in the CRC fork; ReadBuffer_common would read the page, calculate the
> checksum, and compare it to the one stored in the CRC fork.

Hell, all that is needed is a long or a short checksum value in the block.
I mean, if you just want a sanity test, it doesn't take much. Using a
second relation creates confusion. If there is a CRC discrepancy between
two different blocks, who's wrong? You need a third "control" to know. If
the block knows its CRC or checksum and that is in error, the block is
bad.

>
> A buffer's io_in_progress lock protects the buffer's CRC. We read and
> pin the CRC page before acquiring the lock, to avoid having two buffer
> IO operations in flight.
>
> I'd like to submit this for 8.4, but I want to ensure that -hackers at
> large approve of this feature before starting serious coding.
>
> Opinions?

If its fast enough, its a good idea. It could be very helpful in
protecting users data.

>
> --
> Alvaro Herrera
> http://www.CommandPrompt.com/
> PostgreSQL Replication, Consulting, Custom Development, 24x7 support
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [GENERAL] Why Does UPDATE Take So Long?

On Tuesday 30 September 2008, Bill Thoen <bthoen@gisnet.com> wrote:
> Working with PG 8.1 I'm trying to update a char(4) column, and it's
> taking a very long time; 15 minutes so far and no end in sight. From the
> explain, it doesn't seem like it should take that long, and this column
> is not indexed. Sure, there's 2.7 million records but it only takes a
> few minutes to scan the whole file. Is there some special overhead I
> should be aware of with an UPDATE? I VACUUMed and ANALYZEd first, too.

update creates new rows for all affected rows. If the table is indexed, it
creates new index rows for all affected rows in every index. Slow updates
is a common PostgreSQL complaint.

--
Alan

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] Block-level CRC checks

Alvaro Herrera wrote:
> Initially I'm aiming at a CRC32 sum
> for each block. FlushBuffer would calculate the checksum and store it
> in the CRC fork; ReadBuffer_common would read the page, calculate the
> checksum, and compare it to the one stored in the CRC fork.

There's one fundamental problem with that, related to the way our hint
bits are written.

Currently, hint bit updates are not WAL-logged, and thus no full page
write is done when only hint bits are changed. Imagine what happens if
hint bits are updated on a page, but there's no other changes, and we
crash so that only one half of the new page version makes it to disk (=
torn page). The CRC would not match, even though the page is actually valid.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[GENERAL] Why Does UPDATE Take So Long?

Working with PG 8.1 I'm trying to update a char(4) column, and it's
taking a very long time; 15 minutes so far and no end in sight. From the
explain, it doesn't seem like it should take that long, and this column
is not indexed. Sure, there's 2.7 million records but it only takes a
few minutes to scan the whole file. Is there some special overhead I
should be aware of with an UPDATE? I VACUUMed and ANALYZEd first, too.

Or am I just expecting too much?

Here's the explain:
explain UPDATE farms SET prog_year='2007';
QUERY PLAN
----------------------------------------------------------------
Seq Scan on farms (cost=0.00..59144.07 rows=2772207 width=54)
(1 row)


TIA,
- Bill Thoen


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [HACKERS] Block-level CRC checks

Hello Alvaro,

some random thoughts while reading your proposal follow...

Alvaro Herrera wrote:
> So we've been tasked with adding CRCs to data files.

Disks get larger and relative reliability shrinks, it seems. So I agree
that this is a worthwhile thing to have. But shouldn't that be the job
of the filesystem? Think of ZFS or the upcoming BTRFS.

> The idea is that these CRCs are going to be checked just after reading
> files from disk, and calculated just before writing it. They are
> just a protection against the storage layer going mad; they are not
> intended to protect against faulty RAM, CPU or kernel.

That sounds reasonable if we do it from Postgres.

> This code would be run-time or compile-time configurable. I'm not
> absolutely sure which yet; the problem with run-time is what to do if
> the user restarts the server with the setting flipped. It would have
> almost no impact on users who don't enable it.

I'd say calculating a CRC is close enough to be considered "no impact".
A single core of a modern CPU easily reaches way above 200 MiB/s
throughput for CRC32 today. See [1].

Maybe consider Adler-32 which is 3-4x faster [2], also part of zlib and
AFAIK about equally safe for 8k blocks and above.

> The implementation I'm envisioning requires the use of a new relation
> fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
> for each block. FlushBuffer would calculate the checksum and store it
> in the CRC fork; ReadBuffer_common would read the page, calculate the
> checksum, and compare it to the one stored in the CRC fork.

Huh? Aren't CRCs normally stored as part of the block they are supposed
to protect? Or how do you expect to ensure the data from the CRC
relation fork is correct? How about crash safety (a data block written,
but not its CRC block or vice versa)?

Wouldn't that double the amount of seeking required for writes?

> I'd like to submit this for 8.4, but I want to ensure that -hackers at
> large approve of this feature before starting serious coding.

Very cool!

Regards

Markus Wanner

[1]: Crypto++ benchmarks:
http://www.cryptopp.com/benchmarks.html

[2]: Wikipedia about hash functions:
http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [ADMIN] turning of pg_xlog

Tom Lane wrote:
> Chander Ganesan <chander@otg-nc.com> writes:
>
>> You should also understand that if you set checkpoint_segments to a
>> small number, its still possible that PostgreSQL might use more than
>> that number for a large transaction.
>>
>
> "Large transactions" have nothing to do with this. You are confusing
> Postgres' implementation with Oracle's.
>
> regards, tom lane
Okay. I guess my use of the word "transaction" here was erroneous -
probably comes from my oracle background :-( . However, the fact
remains that its possible to exceed checkpoint_segments in certain
scenarios (I think the docs refer to it as a peak in log creation or
something like that), which - if it fills up the disk that the WALs are
on, can result in issues... And one should be wary of that when trying
to limit the number of WALs in order to meet limited disk space
requirements.

--
Chander Ganesan
Open Technology Group, Inc.
One Copley Parkway, Suite 210
Morrisville, NC 27560
919-463-0999/877-258-8987
http://www.otg-nc.com


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [HACKERS] Block-level CRC checks

On Tue, Sep 30, 2008 at 2:49 PM, Joshua Drake <jd@commandprompt.com> wrote:
> On Tue, 30 Sep 2008 14:33:04 -0400
> "Jonah H. Harris" <jonah.harris@gmail.com> wrote:
>
>> > I'd like to submit this for 8.4, but I want to ensure that -hackers
>> > at large approve of this feature before starting serious coding.
>>
>> IMHO, this is a functionality that should be enabled by default (as it
>> is on most other RDBMS). It would've prevented severe corruption in
>
> What other RDMS have it enabled by default?

Oracle and (I belive) SQL Server >= 2005

--
Jonah H. Harris, Senior DBA
myYearbook.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [ADMIN] turning of pg_xlog

I might have to disagree here. I personally experienced a scenario
where I had 500 MB of file space defined for my WAL log files and then
attempted (not thinking it through thoroughly) to perform a COPY of a
very large (1.8 GB) dataset and kept adding WAL files until my file
system filled up and then the instance crashed.

-----Original Message-----
From: pgsql-admin-owner@postgresql.org
[mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Tom Lane
Sent: Tuesday, September 30, 2008 1:45 PM
To: Chander Ganesan
Cc: Jonny; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] turning of pg_xlog

Chander Ganesan <chander@otg-nc.com> writes:
> You should also understand that if you set checkpoint_segments to a
> small number, its still possible that PostgreSQL might use more than
> that number for a large transaction.

"Large transactions" have nothing to do with this. You are confusing
Postgres' implementation with Oracle's.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [HACKERS] Block-level CRC checks

On Tue, 30 Sep 2008 14:33:04 -0400
"Jonah H. Harris" <jonah.harris@gmail.com> wrote:

> > I'd like to submit this for 8.4, but I want to ensure that -hackers
> > at large approve of this feature before starting serious coding.
>
> IMHO, this is a functionality that should be enabled by default (as it
> is on most other RDBMS). It would've prevented severe corruption in

What other RDMS have it enabled by default?

Sincerely,

Joshua D. Drake


--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Block-level CRC checks

Alvaro Herrera wrote:
> A customer of ours has been having trouble with corrupted data for some
> time. Of course, we've almost always blamed hardware (and we've seen
> RAID controllers have their firmware upgraded, among other actions), but
> the useful thing to know is when corruption has happened, and where.
>
> So we've been tasked with adding CRCs to data files.
>
> The idea is that these CRCs are going to be checked just after reading
> files from disk, and calculated just before writing it. They are
> just a protection against the storage layer going mad; they are not
> intended to protect against faulty RAM, CPU or kernel.

This has been suggested before, and the usual objection is precisely
that it only protects from errors in the storage layer, giving a false
sense of security.

Doesn't some filesystems include a per-block CRC, which would achieve
the same thing? ZFS?

> This code would be run-time or compile-time configurable. I'm not
> absolutely sure which yet; the problem with run-time is what to do if
> the user restarts the server with the setting flipped. It would have
> almost no impact on users who don't enable it.

Yeah, seems like it would need to be compile-time or initdb-time
configurable.

> The implementation I'm envisioning requires the use of a new relation
> fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
> for each block. FlushBuffer would calculate the checksum and store it
> in the CRC fork; ReadBuffer_common would read the page, calculate the
> checksum, and compare it to the one stored in the CRC fork.

Surely it would be much simpler to just add a field to the page header.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [ADMIN] turning of pg_xlog

Chander Ganesan <chander@otg-nc.com> writes:
> You should also understand that if you set checkpoint_segments to a
> small number, its still possible that PostgreSQL might use more than
> that number for a large transaction.

"Large transactions" have nothing to do with this. You are confusing
Postgres' implementation with Oracle's.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [HACKERS] Block-level CRC checks

Alvaro Herrera <alvherre@commandprompt.com> writes:
> The implementation I'm envisioning requires the use of a new relation
> fork to store the per-block CRCs.

That seems bizarre, and expensive, and if you lose one block of the CRC
fork you lose confidence in a LOT of data. Why not keep the CRCs in the
page headers?

> A buffer's io_in_progress lock protects the buffer's CRC.

Unfortunately, it doesn't. See hint bits.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[pgsql-es-ayuda] Validación de CUIT - Argentina

Buenas Tardes a todos:
Alguien tiene armada alguna función para validar el
CUIT (Clave Única de Identificación Tributaria) de Argentina.

Desde ya les agradezco.

Atte.
___________________________
ASC Daniel Ferrer
Gerente de Sistemas - CTD SRL
Rosario - Argentina
mailto:daniel.ferrer@ctd.com.ar
___________________________

--
TIP 7: no olvides aumentar la configuración del "free space map"

[HACKERS] pg_hba options parsing

*** a/src/backend/libpq/auth.c
--- b/src/backend/libpq/auth.c
***************
*** 126,134 **** char *pg_krb_realm = NULL;
* MIT Kerberos authentication system - protocol version 5
*----------------------------------------------------------------
*/
- static int pg_krb5_recvauth(Port *port);
-
#ifdef KRB5

#include <krb5.h>
/* Some old versions of Kerberos do not include <com_err.h> in <krb5.h> */
--- 126,133 ----
* MIT Kerberos authentication system - protocol version 5
*----------------------------------------------------------------
*/
#ifdef KRB5
+ static int pg_krb5_recvauth(Port *port);

#include <krb5.h>
/* Some old versions of Kerberos do not include <com_err.h> in <krb5.h> */
***************
*** 150,163 **** static krb5_principal pg_krb5_server;
* GSSAPI Authentication
*----------------------------------------------------------------
*/
- static int pg_GSS_recvauth(Port *port);
-
#ifdef ENABLE_GSS
#if defined(HAVE_GSSAPI_H)
#include <gssapi.h>
#else
#include <gssapi/gssapi.h>
#endif
#endif /* ENABLE_GSS */


--- 149,162 ----
* GSSAPI Authentication
*----------------------------------------------------------------
*/
#ifdef ENABLE_GSS
#if defined(HAVE_GSSAPI_H)
#include <gssapi.h>
#else
#include <gssapi/gssapi.h>
#endif
+
+ static int pg_GSS_recvauth(Port *port);
#endif /* ENABLE_GSS */


***************
*** 165,176 **** static int pg_GSS_recvauth(Port *port);
* SSPI Authentication
*----------------------------------------------------------------
*/
- static int pg_SSPI_recvauth(Port *port);
-
#ifdef ENABLE_SSPI
typedef SECURITY_STATUS
(WINAPI * QUERY_SECURITY_CONTEXT_TOKEN_FN) (
PCtxtHandle, void **);
#endif


--- 164,174 ----
* SSPI Authentication
*----------------------------------------------------------------
*/
#ifdef ENABLE_SSPI
typedef SECURITY_STATUS
(WINAPI * QUERY_SECURITY_CONTEXT_TOKEN_FN) (
PCtxtHandle, void **);
+ static int pg_SSPI_recvauth(Port *port);
#endif


***************
*** 236,251 **** auth_failed(Port *port, int status)
case uaPassword:
errstr = gettext_noop("password authentication failed for user \"%s\"");
break;
- #ifdef USE_PAM
case uaPAM:
errstr = gettext_noop("PAM authentication failed for user \"%s\"");
break;
- #endif /* USE_PAM */
- #ifdef USE_LDAP
case uaLDAP:
errstr = gettext_noop("LDAP authentication failed for user \"%s\"");
break;
- #endif /* USE_LDAP */
default:
errstr = gettext_noop("authentication failed for user \"%s\": invalid authentication method");
break;
--- 234,245 ----
***************
*** 316,333 **** ClientAuthentication(Port *port)
--- 310,339 ----
}

case uaKrb5:
+ #ifdef KRB5
sendAuthRequest(port, AUTH_REQ_KRB5);
status = pg_krb5_recvauth(port);
+ #else
+ Assert(false);
+ #endif
break;

case uaGSS:
+ #ifdef ENABLE_GSS
sendAuthRequest(port, AUTH_REQ_GSS);
status = pg_GSS_recvauth(port);
+ #else
+ Assert(false);
+ #endif
break;

case uaSSPI:
+ #ifdef ENABLE_SSPI
sendAuthRequest(port, AUTH_REQ_SSPI);
status = pg_SSPI_recvauth(port);
+ #else
+ Assert(false);
+ #endif
break;

case uaIdent:
***************
*** 377,393 **** ClientAuthentication(Port *port)
status = recv_and_check_password_packet(port);
break;

- #ifdef USE_PAM
case uaPAM:
pam_port_cludge = port;
status = CheckPAMAuth(port, port->user_name, "");
break;
#endif /* USE_PAM */

- #ifdef USE_LDAP
case uaLDAP:
status = CheckLDAPAuth(port);
break;
#endif

case uaTrust:
--- 383,403 ----
status = recv_and_check_password_packet(port);
break;

case uaPAM:
+ #ifdef USE_PAM
pam_port_cludge = port;
status = CheckPAMAuth(port, port->user_name, "");
+ #else
+ Assert(false);
break;
#endif /* USE_PAM */

case uaLDAP:
+ #ifdef USE_LDAP
status = CheckLDAPAuth(port);
break;
+ #else
+ Assert(false);
#endif

case uaTrust:
***************
*** 713,731 **** pg_krb5_recvauth(Port *port)
return STATUS_ERROR;
}

! if (pg_krb_caseins_users)
! ret = pg_strncasecmp(port->user_name, kusername, SM_DATABASE_USER);
! else
! ret = strncmp(port->user_name, kusername, SM_DATABASE_USER);
! if (ret)
! {
! ereport(LOG,
! (errmsg("unexpected Kerberos user name received from client (received \"%s\", expected \"%s\")",
! port->user_name, kusername)));
! ret = STATUS_ERROR;
! }
! else
! ret = STATUS_OK;

krb5_free_ticket(pg_krb5_context, ticket);
krb5_auth_con_free(pg_krb5_context, auth_context);
--- 723,730 ----
return STATUS_ERROR;
}

! ret = check_usermap(port->hba->usermap, port->user_name, kusername,
! pg_krb_caseins_users);

krb5_free_ticket(pg_krb5_context, ticket);
krb5_auth_con_free(pg_krb5_context, auth_context);
***************
*** 733,748 **** pg_krb5_recvauth(Port *port)

return ret;
}
- #else
-
- static int
- pg_krb5_recvauth(Port *port)
- {
- ereport(LOG,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("Kerberos 5 not implemented on this server")));
- return STATUS_ERROR;
- }
#endif /* KRB5 */


--- 732,737 ----
***************
*** 1020,1057 **** pg_GSS_recvauth(Port *port)
return STATUS_ERROR;
}

! if (pg_krb_caseins_users)
! ret = pg_strcasecmp(port->user_name, gbuf.value);
! else
! ret = strcmp(port->user_name, gbuf.value);
!
! if (ret)
! {
! /* GSS name and PGUSER are not equivalent */
! elog(DEBUG2,
! "provided username (%s) and GSSAPI username (%s) don't match",
! port->user_name, (char *) gbuf.value);
!
! gss_release_buffer(&lmin_s, &gbuf);
! return STATUS_ERROR;
! }

gss_release_buffer(&lmin_s, &gbuf);

return STATUS_OK;
}
-
- #else /* no ENABLE_GSS */
-
- static int
- pg_GSS_recvauth(Port *port)
- {
- ereport(LOG,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("GSSAPI not implemented on this server")));
- return STATUS_ERROR;
- }
-
#endif /* ENABLE_GSS */


--- 1009,1021 ----
return STATUS_ERROR;
}

! ret = check_usermap(port->hba->usermap, port->user_name, gbuf.value,
! pg_krb_caseins_users);

gss_release_buffer(&lmin_s, &gbuf);

return STATUS_OK;
}
#endif /* ENABLE_GSS */


***************
*** 1328,1357 **** pg_SSPI_recvauth(Port *port)
* We have the username (without domain/realm) in accountname, compare to
* the supplied value. In SSPI, always compare case insensitive.
*/
! if (pg_strcasecmp(port->user_name, accountname))
! {
! /* GSS name and PGUSER are not equivalent */
! elog(DEBUG2,
! "provided username (%s) and SSPI username (%s) don't match",
! port->user_name, accountname);
!
! return STATUS_ERROR;
! }
!
! return STATUS_OK;
}
-
- #else /* no ENABLE_SSPI */
-
- static int
- pg_SSPI_recvauth(Port *port)
- {
- ereport(LOG,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("SSPI not implemented on this server")));
- return STATUS_ERROR;
- }
-
#endif /* ENABLE_SSPI */


--- 1292,1299 ----
* We have the username (without domain/realm) in accountname, compare to
* the supplied value. In SSPI, always compare case insensitive.
*/
! return check_usermap(port->hba->usermap, port->user_name, accountname, true);
}
#endif /* ENABLE_SSPI */


***************
*** 1795,1808 **** authident(hbaPort *port)
return STATUS_ERROR;
}

! ereport(DEBUG2,
! (errmsg("Ident protocol identifies remote user as \"%s\"",
! ident_user)));
!
! if (check_ident_usermap(port->hba->usermap, port->user_name, ident_user))
! return STATUS_OK;
! else
! return STATUS_ERROR;
}


--- 1737,1743 ----
return STATUS_ERROR;
}

! return check_usermap(port->hba->usermap, port->user_name, ident_user, false);
}


***************
*** 1913,1920 **** CheckPAMAuth(Port *port, char *user, char *password)
* not allocated */

/* Optionally, one can set the service name in pg_hba.conf */
! if (port->hba->auth_arg && port->hba->auth_arg[0] != '\0')
! retval = pam_start(port->hba->auth_arg, "pgsql@",
&pam_passw_conv, &pamh);
else
retval = pam_start(PGSQL_PAM_SERVICE, "pgsql@",
--- 1848,1855 ----
* not allocated */

/* Optionally, one can set the service name in pg_hba.conf */
! if (port->hba->pamservice && port->hba->pamservice[0] != '\0')
! retval = pam_start(port->hba->pamservice, "pgsql@",
&pam_passw_conv, &pamh);
else
retval = pam_start(PGSQL_PAM_SERVICE, "pgsql@",
***************
*** 2000,2075 **** static int
CheckLDAPAuth(Port *port)
{
char *passwd;
- char server[128];
- char basedn[128];
- char prefix[128];
- char suffix[128];
LDAP *ldap;
- bool ssl = false;
int r;
int ldapversion = LDAP_VERSION3;
- int ldapport = LDAP_PORT;
char fulluser[NAMEDATALEN + 256 + 1];

! if (!port->hba->auth_arg || port->hba->auth_arg[0] == '\0')
{
ereport(LOG,
! (errmsg("LDAP configuration URL not specified")));
return STATUS_ERROR;
}

! /*
! * Crack the LDAP url. We do a very trivial parse:
! *
! * ldap[s]://<server>[:<port>]/<basedn>[;prefix[;suffix]]
! *
! * This code originally used "%127s" for the suffix, but that doesn't
! * work for embedded whitespace. We know that tokens formed by
! * hba.c won't include newlines, so we can use a "not newline" scanset
! * instead.
! */
!
! server[0] = '\0';
! basedn[0] = '\0';
! prefix[0] = '\0';
! suffix[0] = '\0';
!
! /* ldap, including port number */
! r = sscanf(port->hba->auth_arg,
! "ldap://%127[^:]:%d/%127[^;];%127[^;];%127[^\n]",
! server, &ldapport, basedn, prefix, suffix);
! if (r < 3)
! {
! /* ldaps, including port number */
! r = sscanf(port->hba->auth_arg,
! "ldaps://%127[^:]:%d/%127[^;];%127[^;];%127[^\n]",
! server, &ldapport, basedn, prefix, suffix);
! if (r >= 3)
! ssl = true;
! }
! if (r < 3)
! {
! /* ldap, no port number */
! r = sscanf(port->hba->auth_arg,
! "ldap://%127[^/]/%127[^;];%127[^;];%127[^\n]",
! server, basedn, prefix, suffix);
! }
! if (r < 2)
! {
! /* ldaps, no port number */
! r = sscanf(port->hba->auth_arg,
! "ldaps://%127[^/]/%127[^;];%127[^;];%127[^\n]",
! server, basedn, prefix, suffix);
! if (r >= 2)
! ssl = true;
! }
! if (r < 2)
! {
! ereport(LOG,
! (errmsg("invalid LDAP URL: \"%s\"",
! port->hba->auth_arg)));
! return STATUS_ERROR;
! }

sendAuthRequest(port, AUTH_REQ_PASSWORD);

--- 1935,1954 ----
CheckLDAPAuth(Port *port)
{
char *passwd;
LDAP *ldap;
int r;
int ldapversion = LDAP_VERSION3;
char fulluser[NAMEDATALEN + 256 + 1];

! if (!port->hba->ldapserver|| port->hba->ldapserver[0] == '\0')
{
ereport(LOG,
! (errmsg("LDAP server not specified")));
return STATUS_ERROR;
}

! if (port->hba->ldapport == 0)
! port->hba->ldapport = LDAP_PORT;

sendAuthRequest(port, AUTH_REQ_PASSWORD);

***************
*** 2077,2083 **** CheckLDAPAuth(Port *port)
if (passwd == NULL)
return STATUS_EOF; /* client wouldn't send password */

! ldap = ldap_init(server, ldapport);
if (!ldap)
{
#ifndef WIN32
--- 1956,1962 ----
if (passwd == NULL)
return STATUS_EOF; /* client wouldn't send password */

! ldap = ldap_init(port->hba->ldapserver, port->hba->ldapport);
if (!ldap)
{
#ifndef WIN32
***************
*** 2100,2106 **** CheckLDAPAuth(Port *port)
return STATUS_ERROR;
}

! if (ssl)
{
#ifndef WIN32
if ((r = ldap_start_tls_s(ldap, NULL, NULL)) != LDAP_SUCCESS)
--- 1979,1985 ----
return STATUS_ERROR;
}

! if (port->hba->ldaptls)
{
#ifndef WIN32
if ((r = ldap_start_tls_s(ldap, NULL, NULL)) != LDAP_SUCCESS)
***************
*** 2155,2161 **** CheckLDAPAuth(Port *port)
}

snprintf(fulluser, sizeof(fulluser), "%s%s%s",
! prefix, port->user_name, suffix);
fulluser[sizeof(fulluser) - 1] = '\0';

r = ldap_simple_bind_s(ldap, fulluser, passwd);
--- 2034,2042 ----
}

snprintf(fulluser, sizeof(fulluser), "%s%s%s",
! port->hba->ldapprefix?port->hba->ldapprefix:"",
! port->user_name,
! port->hba->ldapsuffix?port->hba->ldapsuffix:"");
fulluser[sizeof(fulluser) - 1] = '\0';

r = ldap_simple_bind_s(ldap, fulluser, passwd);
***************
*** 2165,2171 **** CheckLDAPAuth(Port *port)
{
ereport(LOG,
(errmsg("LDAP login failed for user \"%s\" on server \"%s\": error code %d",
! fulluser, server, r)));
return STATUS_ERROR;
}

--- 2046,2052 ----
{
ereport(LOG,
(errmsg("LDAP login failed for user \"%s\" on server \"%s\": error code %d",
! fulluser, port->hba->ldapserver, r)));
return STATUS_ERROR;
}

*** a/src/backend/libpq/hba.c
--- b/src/backend/libpq/hba.c
***************
*** 565,570 **** check_db(const char *dbname, const char *role, char *param_str)
--- 565,606 ----


/*
+ * Macros used to check and report on invalid configuration options.
+ * INVALID_AUTH_OPTION = reports when an option is specified for a method where it's
+ * not supported.
+ * REQUIRE_AUTH_OPTION = same as INVALID_AUTH_OPTION, except it also checks if the
+ * method is actually the one specified. Used as a shortcut when
+ * the option is only valid for one authentication method.
+ * MANDATORY_AUTH_ARG = check if a required option is set for an authentication method,
+ * reporting error if it's not.
+ */
+ #define INVALID_AUTH_OPTION(optname, validmethods) do {\
+ ereport(LOG, \
+ (errcode(ERRCODE_CONFIG_FILE_ERROR), \
+ errmsg("authentication option '%s' is only valid for authentication methods '%s'", \
+ optname, validmethods), \
+ errcontext("line %d of configuration file \"%s\"", \
+ line_num, HbaFileName))); \
+ goto hba_other_error; \
+ } while (0)
+
+ #define REQUIRE_AUTH_OPTION(methodval, optname, validmethods) \
+ if (parsedline->auth_method != methodval) \
+ INVALID_AUTH_OPTION("ldaptls", "ldap")
+
+ #define MANDATORY_AUTH_ARG(argvar, argname, authname) \
+ if (argvar == NULL) {\
+ ereport(LOG, \
+ (errcode(ERRCODE_CONFIG_FILE_ERROR), \
+ errmsg("authentication method '%s' requires argument '%s' to be set", \
+ authname, argname), \
+ errcontext("line %d of configuration file \"%s\"", \
+ line_num, HbaFileName))); \
+ goto hba_other_error; \
+ } while (0);
+
+
+ /*
* Parse one line in the hba config file and store the result in
* a HbaLine structure.
*/
***************
*** 801,838 **** parse_hba_line(List *line, int line_num, HbaLine *parsedline)
goto hba_other_error;
}

! /* Get the authentication argument token, if any */
! line_item = lnext(line_item);
! if (line_item)
{
token = lfirst(line_item);
- parsedline->auth_arg= pstrdup(token);
- }

! /*
! * Backwards compatible format of ident authentication - support "naked" ident map
! * name, as well as "sameuser"/"samerole"
! */
! if (parsedline->auth_method == uaIdent)
! {
! if (parsedline->auth_arg && strlen(parsedline->auth_arg))
{
! if (strcmp(parsedline->auth_arg, "sameuser\n") == 0 ||
! strcmp(parsedline->auth_arg, "samerole\n") == 0)
{
! /* This is now the default */
! pfree(parsedline->auth_arg);
! parsedline->auth_arg = NULL;
! parsedline->usermap = NULL;
}
else
{
! /* Specific ident map specified */
! parsedline->usermap = parsedline->auth_arg;
! parsedline->auth_arg = NULL;
}
}
}

return true;

--- 837,938 ----
goto hba_other_error;
}

! /* Parse remaining arguments */
! while ((line_item = lnext(line_item)) != NULL)
{
+ char *c;
+
token = lfirst(line_item);

! c = strchr(token, '=');
! if (c == NULL)
{
! /*
! * Got something that's not a name=value pair.
! *
! * XXX: attempt to do some backwards compatible parsing here?
! */
! ereport(LOG,
! (errcode(ERRCODE_CONFIG_FILE_ERROR),
! errmsg("authentication option not in name=value format: %s", token),
! errcontext("line %d of configuration file \"%s\"",
! line_num, HbaFileName)));
! goto hba_other_error;
! }
! else
! {
! *c++ = '\0'; /* token now holds "name", c holds "value" */
! if (strcmp(token, "map") == 0)
! {
! if (parsedline->auth_method != uaIdent &&
! parsedline->auth_method != uaKrb5 &&
! parsedline->auth_method != uaGSS &&
! parsedline->auth_method != uaSSPI)
! INVALID_AUTH_OPTION("map", "ident, krb5, gssapi and sspi");
! parsedline->usermap = pstrdup(c);
! }
! else if (strcmp(token, "pamservice") == 0)
! {
! REQUIRE_AUTH_OPTION(uaPAM, "pamservice", "pam");
! parsedline->pamservice = pstrdup(c);
! }
! else if (strcmp(token, "ldaptls") == 0)
! {
! REQUIRE_AUTH_OPTION(uaLDAP, "ldaptls", "ldap");
! if (strcmp(c, "1") == 0)
! parsedline->ldaptls = true;
! else
! parsedline->ldaptls = false;
! }
! else if (strcmp(token, "ldapserver") == 0)
! {
! REQUIRE_AUTH_OPTION(uaLDAP, "ldapserver", "ldap");
! parsedline->ldapserver = pstrdup(c);
! }
! else if (strcmp(token, "ldapport") == 0)
! {
! REQUIRE_AUTH_OPTION(uaLDAP, "ldapport", "ldap");
! parsedline->ldapport = atoi(c);
! if (parsedline->ldapport == 0)
! {
! ereport(LOG,
! (errcode(ERRCODE_CONFIG_FILE_ERROR),
! errmsg("invalid ldap port '%s'", c),
! errcontext("line %d of configuration file \"%s\"",
! line_num, HbaFileName)));
! goto hba_other_error;
! }
! }
! else if (strcmp(token, "ldapprefix") == 0)
{
! REQUIRE_AUTH_OPTION(uaLDAP, "ldapprefix", "ldap");
! parsedline->ldapprefix = pstrdup(c);
! }
! else if (strcmp(token, "ldapsuffix") == 0)
! {
! REQUIRE_AUTH_OPTION(uaLDAP, "ldapsuffix", "ldap");
! parsedline->ldapsuffix = pstrdup(c);
}
else
{
! ereport(LOG,
! (errcode(ERRCODE_CONFIG_FILE_ERROR),
! errmsg("unknown authentication option name '%s'", token),
! errcontext("line %d of configuration file \"%s\"",
! line_num, HbaFileName)));
! goto hba_other_error;
}
}
}
+
+ /*
+ * Check if the selected authentication method has any mandatory arguments that
+ * are not set.
+ */
+ if (parsedline->auth_method == uaLDAP)
+ {
+ MANDATORY_AUTH_ARG(parsedline->ldapserver, "ldapserver", "ldap");
+ }

return true;

***************
*** 1018,1025 **** free_hba_record(HbaLine *record)
pfree(record->database);
if (record->role)
pfree(record->role);
! if (record->auth_arg)
! pfree(record->auth_arg);
}

/*
--- 1118,1131 ----
pfree(record->database);
if (record->role)
pfree(record->role);
! if (record->pamservice)
! pfree(record->pamservice);
! if (record->ldapserver)
! pfree(record->ldapserver);
! if (record->ldapprefix)
! pfree(record->ldapprefix);
! if (record->ldapsuffix)
! pfree(record->ldapsuffix);
}

/*
***************
*** 1150,1156 **** read_pg_database_line(FILE *fp, char *dbname, Oid *dboid,
static void
parse_ident_usermap(List *line, int line_number, const char *usermap_name,
const char *pg_role, const char *ident_user,
! bool *found_p, bool *error_p)
{
ListCell *line_item;
char *token;
--- 1256,1262 ----
static void
parse_ident_usermap(List *line, int line_number, const char *usermap_name,
const char *pg_role, const char *ident_user,
! bool case_insensitive, bool *found_p, bool *error_p)
{
ListCell *line_item;
char *token;
***************
*** 1183,1192 **** parse_ident_usermap(List *line, int line_number, const char *usermap_name,
file_pgrole = token;

/* Match? */
! if (strcmp(file_map, usermap_name) == 0 &&
! strcmp(file_pgrole, pg_role) == 0 &&
! strcmp(file_ident_user, ident_user) == 0)
! *found_p = true;

return;

--- 1289,1308 ----
file_pgrole = token;

/* Match? */
! if (case_insensitive)
! {
! if (strcmp(file_map, usermap_name) == 0 &&
! pg_strcasecmp(file_pgrole, pg_role) == 0 &&
! pg_strcasecmp(file_ident_user, ident_user) == 0)
! *found_p = true;
! }
! else
! {
! if (strcmp(file_map, usermap_name) == 0 &&
! strcmp(file_pgrole, pg_role) == 0 &&
! strcmp(file_ident_user, ident_user) == 0)
! *found_p = true;
! }

return;

***************
*** 1210,1231 **** ident_syntax:
* file. That's an implied map where "pgrole" must be identical to
* "ident_user" in order to be authorized.
*
! * Iff authorized, return true.
*/
! bool
! check_ident_usermap(const char *usermap_name,
const char *pg_role,
! const char *ident_user)
{
bool found_entry = false,
error = false;

if (usermap_name == NULL || usermap_name[0] == '\0')
{
! if (strcmp(pg_role, ident_user) == 0)
! found_entry = true;
! else
! found_entry = false;
}
else
{
--- 1326,1357 ----
* file. That's an implied map where "pgrole" must be identical to
* "ident_user" in order to be authorized.
*
! * Iff authorized, return STATUS_OK, otherwise return STATUS_ERROR.
*/
! int
! check_usermap(const char *usermap_name,
const char *pg_role,
! const char *auth_user,
! bool case_insensitive)
{
bool found_entry = false,
error = false;

if (usermap_name == NULL || usermap_name[0] == '\0')
{
! if (case_insensitive)
! {
! if (pg_strcasecmp(pg_role, auth_user) == 0)
! return STATUS_OK;
! }
! else {
! if (strcmp(pg_role, auth_user) == 0)
! return STATUS_OK;
! }
! ereport(LOG,
! (errmsg("provided username (%s) and authenticated username (%s) don't match",
! auth_user, pg_role)));
! return STATUS_ERROR;
}
else
{
***************
*** 1235,1247 **** check_ident_usermap(const char *usermap_name,
forboth(line_cell, ident_lines, num_cell, ident_line_nums)
{
parse_ident_usermap(lfirst(line_cell), lfirst_int(num_cell),
! usermap_name, pg_role, ident_user,
&found_entry, &error);
if (found_entry || error)
break;
}
}
! return found_entry;
}


--- 1361,1380 ----
forboth(line_cell, ident_lines, num_cell, ident_line_nums)
{
parse_ident_usermap(lfirst(line_cell), lfirst_int(num_cell),
! usermap_name, pg_role, auth_user, case_insensitive,
&found_entry, &error);
if (found_entry || error)
break;
}
}
! if (!found_entry && !error)
! {
! ereport(LOG,
! (errmsg("no match in usermap for user '%s' authenticated as '%s'",
! pg_role, auth_user),
! errcontext("usermap '%s'", usermap_name)));
! }
! return found_entry?STATUS_OK:STATUS_ERROR;
}


*** a/src/backend/libpq/pg_hba.conf.sample
--- b/src/backend/libpq/pg_hba.conf.sample
***************
*** 9,18 ****
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
! # local DATABASE USER METHOD [OPTION]
! # host DATABASE USER CIDR-ADDRESS METHOD [OPTION]
! # hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
! # hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
#
# (The uppercase items must be replaced by actual values.)
#
--- 9,18 ----
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
! # local DATABASE USER METHOD [OPTIONS]
! # host DATABASE USER CIDR-ADDRESS METHOD [OPTIONS]
! # hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTIONS]
! # hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
***************
*** 38,44 ****
# "krb5", "ident", "pam" or "ldap". Note that "password" sends passwords
# in clear text; "md5" is preferred since it sends encrypted passwords.
#
! # OPTION is the ident map or the name of the PAM service, depending on METHOD.
#
# Database and user names containing spaces, commas, quotes and other special
# characters must be quoted. Quoting one of the keywords "all", "sameuser" or
--- 38,47 ----
# "krb5", "ident", "pam" or "ldap". Note that "password" sends passwords
# in clear text; "md5" is preferred since it sends encrypted passwords.
#
! # OPTIONS are a set of options for the authentication in the format
! # NAME=VALUE. The available options depend on the different authentication
! # methods - refer to the "Client Authentication" section in the documentation
! # for a list of which options are available for which authentication methods.
#
# Database and user names containing spaces, commas, quotes and other special
# characters must be quoted. Quoting one of the keywords "all", "sameuser" or
*** a/src/backend/libpq/pg_ident.conf.sample
--- b/src/backend/libpq/pg_ident.conf.sample
***************
*** 5,22 ****
# Authentication" for a complete description. A short synopsis
# follows.
#
! # This file controls PostgreSQL ident-based authentication. It maps
! # ident user names (typically Unix user names) to their corresponding
# PostgreSQL user names. Records are of the form:
#
! # MAPNAME IDENT-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
# MAPNAME is the (otherwise freely chosen) map name that was used in
! # pg_hba.conf. IDENT-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
! # existence of a record specifies that IDENT-USERNAME may connect as
# PG-USERNAME. Multiple maps may be specified in this file and used
# by pg_hba.conf.
#
--- 5,22 ----
# Authentication" for a complete description. A short synopsis
# follows.
#
! # This file controls PostgreSQL username mapping. It maps
! # external user names to their corresponding
# PostgreSQL user names. Records are of the form:
#
! # MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
# MAPNAME is the (otherwise freely chosen) map name that was used in
! # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
! # existence of a record specifies that SYSTEM-USERNAME may connect as
# PG-USERNAME. Multiple maps may be specified in this file and used
# by pg_hba.conf.
#
***************
*** 28,35 ****
# Put your actual configuration here
# ----------------------------------
#
! # No map names are defined in the default configuration. If all ident
# user names and PostgreSQL user names are the same, you don't need
# this file.

! # MAPNAME IDENT-USERNAME PG-USERNAME
--- 28,35 ----
# Put your actual configuration here
# ----------------------------------
#
! # No map names are defined in the default configuration. If all system
# user names and PostgreSQL user names are the same, you don't need
# this file.

! # MAPNAME SYSTEM-USERNAME PG-USERNAME
*** a/src/include/libpq/hba.h
--- b/src/include/libpq/hba.h
***************
*** 25,37 **** typedef enum UserAuth
uaCrypt,
uaMD5,
uaGSS,
! uaSSPI
! #ifdef USE_PAM
! ,uaPAM
! #endif /* USE_PAM */
! #ifdef USE_LDAP
! ,uaLDAP
! #endif
} UserAuth;

typedef enum ConnType
--- 25,33 ----
uaCrypt,
uaMD5,
uaGSS,
! uaSSPI,
! uaPAM,
! uaLDAP
} UserAuth;

typedef enum ConnType
***************
*** 51,58 **** typedef struct
struct sockaddr_storage addr;
struct sockaddr_storage mask;
UserAuth auth_method;
char *usermap;
! char *auth_arg;
} HbaLine;

typedef struct Port hbaPort;
--- 47,60 ----
struct sockaddr_storage addr;
struct sockaddr_storage mask;
UserAuth auth_method;
+
char *usermap;
! char *pamservice;
! bool ldaptls;
! char *ldapserver;
! int ldapport;
! char *ldapprefix;
! char *ldapsuffix;
} HbaLine;

typedef struct Port hbaPort;
***************
*** 64,71 **** extern void load_role(void);
extern int hba_getauthmethod(hbaPort *port);
extern bool read_pg_database_line(FILE *fp, char *dbname, Oid *dboid,
Oid *dbtablespace, TransactionId *dbfrozenxid);
! extern bool check_ident_usermap(const char *usermap_name,
! const char *pg_role, const char *ident_user);
extern bool pg_isblank(const char c);

#endif /* HBA_H */
--- 66,74 ----
extern int hba_getauthmethod(hbaPort *port);
extern bool read_pg_database_line(FILE *fp, char *dbname, Oid *dboid,
Oid *dbtablespace, TransactionId *dbfrozenxid);
! extern int check_usermap(const char *usermap_name,
! const char *pg_role, const char *auth_user,
! bool case_sensitive);
extern bool pg_isblank(const char c);

#endif /* HBA_H */
This patch changes the options field of pg_hba.conf to take name/value
pairs instead of a fixed string. This makes it a lot nicer to deal with
auth methods that need more than one parameter, such as LDAP.

While at it, it also adds map support to kerberos, gssapi and sspi and
not just ident - basically all methods where the username comes from an
outside source (lmk if I missed one).

Also in passing, changes the methods in auth.c to deal with "unsupported
auth method on this platform" errors the same way for all authentication
methods.

I intend to build on this patch to support setting some
Kerberos/GSSAPI/SSPI parameters on a per-connection base, but wanted to
get the basics in first.

Obviously, documentation still pending. I'm working on that in parallel.


So, comments? Both in general, and specifically on if we need to do
backwards compatible parsing of LDAP options (doing it of all the other
options would be trivial, but LDAP would be harder)


//Magnus

Re: [HACKERS] Block-level CRC checks

On Tue, Sep 30, 2008 at 2:02 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:
> A customer of ours has been having trouble with corrupted data for some
> time. Of course, we've almost always blamed hardware (and we've seen
> RAID controllers have their firmware upgraded, among other actions), but
> the useful thing to know is when corruption has happened, and where.

Agreed.

> So we've been tasked with adding CRCs to data files.

Awesome.

> The idea is that these CRCs are going to be checked just after reading
> files from disk, and calculated just before writing it. They are
> just a protection against the storage layer going mad; they are not
> intended to protect against faulty RAM, CPU or kernel.

This is the common case.

> This code would be run-time or compile-time configurable. I'm not
> absolutely sure which yet; the problem with run-time is what to do if
> the user restarts the server with the setting flipped. It would have
> almost no impact on users who don't enable it.

I've supported this forever!

> The implementation I'm envisioning requires the use of a new relation
> fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
> for each block. FlushBuffer would calculate the checksum and store it
> in the CRC fork; ReadBuffer_common would read the page, calculate the
> checksum, and compare it to the one stored in the CRC fork.
>
> A buffer's io_in_progress lock protects the buffer's CRC. We read and
> pin the CRC page before acquiring the lock, to avoid having two buffer
> IO operations in flight.

If the CRC gets written before the block, how is recovery going to
handle it? I'm not too familiar with the new forks stuff, but
recovery will pull the old block, compare it against the checksum, and
consider the block invalid, correct?

> I'd like to submit this for 8.4, but I want to ensure that -hackers at
> large approve of this feature before starting serious coding.

IMHO, this is a functionality that should be enabled by default (as it
is on most other RDBMS). It would've prevented severe corruption in
the 20 or so databases I've had to fix, and other than making it
optional, I don't see the reasoning for a separate relation fork
rather than storing it directly on the block (as everyone else does).
Similarly, I think Greg Stark was playing with a patch for it
(http://archives.postgresql.org/pgsql-hackers/2007-02/msg01850.php).

--
Jonah H. Harris, Senior DBA
myYearbook.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[pgsql-es-ayuda] Re: [pgsql-es-ayuda] Re: [pgsql-es-ayuda] Encuesta para análisis para el proyecto de creación de una certificación para PostgreSQL

----- Original Message -----
From: "Jaime Casanova" <jcasanov@systemguards.com.ec>
To: "Edwin Quijada" <listas_quijada@hotmail.com>
Cc: <hermeszambra@yahoo.com>; "POSTGRES" <pgsql-es-ayuda@postgresql.org>
Sent: Tuesday, September 30, 2008 12:50 PM
Subject: Re: [pgsql-es-ayuda] Re: [pgsql-es-ayuda] Encuesta para análisis
para el proyecto de creación de una certificación para PostgreSQL


2008/9/30 Edwin Quijada <listas_quijada@hotmail.com>:
>
> Bueno, a pesar de todo creo que este es un buen paso para la comunidad.
> Por lo pronto quiero esa certificacion y
> estoy dispuesto hasta viajar para hacerla.
>
>la idea es que no haya necesidad de viajar
>

Que bueno !!!

Desde hace tiempo la comunidad venía pidiéndolo. Tan pronto me enteré
realicé la encuenta.

Yo sé que tener una certificación no es garantía de que se sabe sobre algo,
pero por lo menos a los empresarios les da algo de confianza. No es camisa
de fuerza certificarse quine no le guste simplemente no lo hace.

Atentamente,

RAUL DUQUE
Bogotá, Colombia

>--
>Atentamente,
>Jaime Casanova
>Soporte y capacitación de PostgreSQL
>Asesoría y desarrollo de sistemas
>Guayaquil - Ecuador
>Cel. +59387171157
>--
>TIP 3: Si encontraste la respuesta a tu problema, publícala, otros te lo
>agradecerán

--
TIP 7: no olvides aumentar la configuración del "free space map"

Re: [GENERAL] Alias name from subquery

I think you should look at crosstab contrib module.

Regards,
Taras Kopets

On 9/30/08, Felix Homann <fexpop@onlinehome.de> wrote:
> Hi Sam,
>
> Sam Mason wrote:
>> I think you may be trying to solve the wrong problem, what are you
>> really trying to do?
>
> Here you go. I have some tables created like this:
>
> CREATE TABLE player(
> id INTEGER PRIMARY KEY,
> name varchar(20)
> );
>
> CREATE TABLE transfer(
> id SERIAL PRIMARY KEY,
> fromID INTEGER REFERENCES player(id),
> toID INTEGER REFERENCES player(id),
> amount numeric
> );
>
> Now, let's fill them with some data:
>
> INSERT INTO player VALUES ('1', 'Peter'), ('2','David'), ('3','Fritz');
> INSERT INTO transfer(fromID, toID, amount) VALUES ('1','2','3'), ('1',
> '3', '1'), ('2','1','60');
>
> I would now like to have something like a view that transforms the table
> "transfer" from this:
>
> test=# SELECT * from transfer;
> id | fromid | toid | amount
> ----+--------+------+--------
> 1 | 1 | 2 | 3
> 2 | 1 | 3 | 1
> 3 | 2 | 1 | 60
>
>
> into this:
>
> id | Peter | David | Fritz | ...even more Names from player table
> ---+-------+-------+-------+-------------------------------------
> 1 | -3 | 3 | 0 | 0
> 2 | -1 | 0 | 1 | 0
> 3 | 60 | -60 | 0 | 0
>
>
> In other words, I would like to have a named column for every Name in
> the player table. I _can_ create such a view manually if I know each
> player.name beforehand, but I don't know how to automate it for any
> given number of players. (Think of something like a sparse interaction
> matrix representation.)
>
> Maybe it's the "wrong" problem I'm trying to solve, but at least I would
> like to know whether it's possible or not.
>
> Kind regards,
>
> Felix
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

--
Sent from Gmail for mobile | mobile.google.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [pgsql-es-ayuda] Modificar WHERE en rule

Finalmente he conseguido solucionarlo con una funcion.

Gracias a todos los que me contestaron.

--
*****************************************
Oswaldo Hernández
oswaldo (@) soft-com (.) es
*****************************************
PD:
Antes de imprimir este mensaje, asegúrese de que es necesario.
El medio ambiente está en nuestra mano.
--
TIP 8: explain analyze es tu amigo

Re: [ADMIN] turning of pg_xlog

Jonny wrote:
> Hi,
>
> I have installes Postgres 8.0.15 on a embedded Linux and have only 130
> MB for Postgres.
> Is it possible to turn off the comlete (Wal) pg_xlog? Because this is
> the biggest part.
> I found an Entry how to minimize the count of stored xlogs.
> Is it possible to store it to /dev/null or something else?
> Something like ln -s /dev/null pg_xlog/ (I know this does not work
> :-) ....
You should also understand that if you set checkpoint_segments to a
small number, its still possible that PostgreSQL might use more than
that number for a large transaction. As such, be careful about limiting
the space available to your xlogs, lest it cause transactions to fail
due to insufficient space for the xlogs.

--
Chander Ganesan
Open Technology Group, Inc.
One Copley Parkway, Suite 210
Morrisville, NC 27560
919-463-0999/877-258-8987
http://www.otg-nc.com


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: [HACKERS] Common Table Expressions (WITH RECURSIVE) patch

Greg Stark <greg.stark@enterprisedb.com> writes:
> On 24 Sep 2008, at 02:45, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> The next big
>> thing seems to be to figure out exactly how to do multiple references
>> to CTE outputs, so that we can de-bogotify the planner.

> I've looked and don't seem to still have the source tree where I
> worked on this. I remember how I made the changes to tuplestore which
> was mostly mechanical. The trick I think will be in adding a special
> purpose executor method which passes the call site to the node below.
> This depends on the observation that if we always memoize results then
> each call site can only have one active call. That is, we don't need
> to maintain a full stack tree.

I looked at the tuplestore code a bit and decided that this actually
doesn't need to be hard at all. Tuplestore already has a notion of a
write position and an independent read position, and we don't need more
than one write position. So what I think we should do is add "get read
position" and "set read position" functions to tuplestore.c, and have
each of the reader nodes remember its own read position. That is,
each reader has to do

set_read_position(tupstore, &local_read_position);
tuple = tuplestore_gettuple(tupstore, ...);
get_read_position(tupstore, &local_read_position);

rather than just tuplestore_gettuple. The set/get functions will be
cheap enough that this is no big deal. (Or maybe we should just
provide a wrapper function that does this sequence?)

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Block-level CRC checks

A customer of ours has been having trouble with corrupted data for some
time. Of course, we've almost always blamed hardware (and we've seen
RAID controllers have their firmware upgraded, among other actions), but
the useful thing to know is when corruption has happened, and where.

So we've been tasked with adding CRCs to data files.

The idea is that these CRCs are going to be checked just after reading
files from disk, and calculated just before writing it. They are
just a protection against the storage layer going mad; they are not
intended to protect against faulty RAM, CPU or kernel.

This code would be run-time or compile-time configurable. I'm not
absolutely sure which yet; the problem with run-time is what to do if
the user restarts the server with the setting flipped. It would have
almost no impact on users who don't enable it.

The implementation I'm envisioning requires the use of a new relation
fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
for each block. FlushBuffer would calculate the checksum and store it
in the CRC fork; ReadBuffer_common would read the page, calculate the
checksum, and compare it to the one stored in the CRC fork.

A buffer's io_in_progress lock protects the buffer's CRC. We read and
pin the CRC page before acquiring the lock, to avoid having two buffer
IO operations in flight.

I'd like to submit this for 8.4, but I want to ensure that -hackers at
large approve of this feature before starting serious coding.

Opinions?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers