Precision and CD's
Andreas Strotmann
strotman at nu.cs.fsu.edu
Wed Jul 21 01:14:24 CEST 1999
Hmm... does this mean that the precision may be wrt. a different base than
the print base (say, binary precison of a decimal number)?
On Tue, 20 Jul 1999, Richard J. Fateman wrote:
> I suspect I send a note just to JHD re this.
>
> Basically, unless you have the same radix on input and output and internal,
> you don't know the precision by counting digits.
>
> the decimal float 0.125x10^0 converted to binary needs 1 bit of precision but
> the decimal float 0.1 x10^0 converted to binary needs 4 bits 13/128.
>
> so decimal digits / .3010 is close, but only +-3 digits.
> (See Kahan's paper on this.)
>
>
More information about the Om
mailing list