Re: [GENERAL] rounding problems

On May 14, 3:27 pm, s...@samason.me.uk (Sam Mason) wrote:
> On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
> > Sam Mason wrote:
> > >What doesfoxprouse for storing numbers? or is it just that you never
> > >pushed it hard enough for the abstractions to show through.
>
> > I know i pushed it. Foxpro for the most has only 4 basic data types
> > Numeric (similar to Posgresql numeric), Boolean, Date, Text aka
> > (string) Thefoxprotables supported far more data types but when every
> > it was dumped to variable it acted like one of the 4.
>
> I really meant how much did you check the results, or did you accept
> that they were correct?
>
> >Foxprodid not suffer floating point math errors. I normally used 8 to
> > 10 points precision. Foxprowas limited to 15 points of precision
> > period. No more and no less, once you hit that was it.
>
> 15 places seems very similar to what a 64bit IEEE floating point number
> will give you, i.e. a double in C/C++.
>
> > My problem is we calculate resistance of parts in aFoxproapp that we
> > want to move because we want to bring all the custom apps into one
> > framework and single database.
>
> > Take this calculation (0.05/30000* 1.0025) which is used to calculate
> > parts resistance and Tolerance. (its Ohms Law) The value returned from
> > C++ = .0000016708 which is wrong
> > it should be .00000167418. We just shrank the tolerance on the part we
> > make
>
> Why are you so sure about theFoxProresult? I've just checked a few
> calculators and get results consistent with your C++ version.
>
> Justin C: 0.0000016708
> JFoxPro: 0.00000167418
> My C: 0.000001670833
> bc[1]: 0.0000016708333333333333333333333333333332
> PG[2]: 0.0000016708333333333333336675
> Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)
>
> Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
> the math, and as they all agree I'm thinkingFoxProis incorrect! Next
> I tried doing it accurately (in Haskell if it makes any difference) and
> get an answer of 401/240000000 out, which would agree with everything
> butFoxPro. If I calculate the ratio back out forFoxProI get
> 401/239520242 which is a little way out.
>
> > The Documentation from MS says 15 points of precision but the result say
> > otherwise.
>
> The docs for what?FoxProor their C compiler?
>
> If you meanFoxPro, I think this is another case of MS screwing up.
>
> > I'm glad You and others are taking the time to explain to me
> > the odd results before i get into redoing that application.
>
> Welcome to the PG community, lots of people to get interested in lots of
> things!
>
> > Why oh Why did MS killFoxpro. :'( I understood it, knew its quirks
> > and it worked very well with Postgresql
>
> Are you sure you want to stay with it if its answers are wrong?
>
> Sam

*********************************************************************************
This is fun, at 0400 AM. I enjoy reading Experts having serious fun!

VFP 6.0, using my defaults
? (0.05/30000* 1.00250000000000000)
displays "0l.000001670833333000"

SET DECIMALS TO 15
? ((0.05/30000)* 1.0025)
displays "0.000001670833333"

and a frivolous example:
SET DECIMALS TO 18
? ((0.050000/30000.00000000)* 1.0025000000000000)
displays "0.000001670833333000"

Anybody tried to reckon this math
the way we used to do it with a Slide-Rule ???
(In VFP of course)

glene77is

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general