Re: [GENERAL] rounding problems

glene77is wrote:

On May 14, 3:27 pm, s...@samason.me.uk (Sam Mason) wrote:   

On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:     

Sam Mason wrote:       

What doesfoxprouse for storing numbers? or is it just that you never pushed it hard enough for the abstractions to show through.         

I know i pushed it.  Foxpro for the most has only  4 basic data types Numeric (similar to Posgresql numeric), Boolean, Date, Text aka (string)  Thefoxprotables supported far more data types but when every it was dumped to variable it acted like one of the 4.       

I really meant how much did you check the results, or did you accept that they were correct?      

Foxprodid not suffer floating point math errors.  I normally used 8 to 10 points precision.  Foxprowas limited to 15 points of precision period.   No more and no less, once you hit that was it.       

15 places seems very similar to what a 64bit IEEE floating point number will give you, i.e. a double in C/C++.      

My problem is we calculate resistance of parts in aFoxproapp that we want to move because we want to bring all the custom apps into one framework and single database.       

Take this calculation  (0.05/30000* 1.0025) which is used to calculate parts resistance and Tolerance. (its Ohms Law)  The value returned  from C++ = .0000016708 which is wrong it should be .00000167418.  We just shrank the tolerance on the part we make       

Why are you so sure about theFoxProresult?  I've just checked a few calculators and get results consistent with your C++ version.    Justin C: 0.0000016708   JFoxPro: 0.00000167418       My C: 0.000001670833      bc[1]: 0.0000016708333333333333333333333333333332      PG[2]: 0.0000016708333333333333336675  Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)  Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do the math, and as they all agree I'm thinkingFoxProis incorrect!  Next I tried doing it accurately (in Haskell if it makes any difference) and get an answer of 401/240000000 out, which would agree with everything butFoxPro.  If I calculate the ratio back out forFoxProI get 401/239520242 which is a little way out.      

The Documentation from MS says 15 points of precision but the result say otherwise.       

The docs for what?FoxProor their C compiler?  If you meanFoxPro, I think this is another case of MS screwing up.      

I'm glad You and others are taking the time to explain to me the odd results before i get into redoing that application.       

Welcome to the PG community, lots of people to get interested in lots of things!      

Why oh Why did MS killFoxpro. :'(   I understood it, knew its quirks and it worked very well with Postgresql       

Are you sure you want to stay with it if its answers are wrong?    Sam     

 ********************************************************************************* This is fun, at 0400 AM.  I enjoy reading  Experts having serious fun!  VFP 6.0, using my defaults ? (0.05/30000* 1.00250000000000000) displays  "0l.000001670833333000"  SET DECIMALS TO 15 ? ((0.05/30000)* 1.0025) displays "0.000001670833333"  and a frivolous example: SET DECIMALS TO 18 ? ((0.050000/30000.00000000)* 1.0025000000000000) displays "0.000001670833333000"   

Foxpro always stops at 15 decimals points,  Even though some of the documentation says 20  and 22 points of precision depending on the version.  I have versions 5 to 9

 Anybody tried to reckon this math the way we used to do it with a Slide-Rule ??? (In VFP of course)   

A slide what??.  I have never touched one or seen a slide rule in real life, just pretty pictures  :-)

 glene77is