Precision and CD's

Richard J. Fateman fateman at cs.berkeley.edu
Wed Jul 21 01:04:15 CEST 1999


I suspect I send a note just to JHD re this.

Basically, unless you have the same radix on input and output and internal,
you don't know the precision by counting digits.

the decimal float 0.125x10^0  converted to binary needs 1 bit of precision but
the decimal float 0.1  x10^0  converted to binary needs 4 bits  13/128.

so decimal digits / .3010 is close, but only +-3 digits.
(See Kahan's paper on this.)




More information about the Om mailing list