Precision and CD's

Andreas Strotmann strotman at nu.cs.fsu.edu
Wed Jul 21 01:14:24 CEST 1999


Hmm... does this mean that the precision may be wrt. a different base than
the print base (say, binary precison of a decimal number)?

On Tue, 20 Jul 1999, Richard J. Fateman wrote:

> I suspect I send a note just to JHD re this.
> 
> Basically, unless you have the same radix on input and output and internal,
> you don't know the precision by counting digits.
> 
> the decimal float 0.125x10^0  converted to binary needs 1 bit of precision but
> the decimal float 0.1  x10^0  converted to binary needs 4 bits  13/128.
> 
> so decimal digits / .3010 is close, but only +-3 digits.
> (See Kahan's paper on this.)
> 
> 




More information about the Om mailing list