Precision and CD's

Andreas Strotmann strotman at
Wed Jul 21 01:14:24 CEST 1999

Hmm... does this mean that the precision may be wrt. a different base than
the print base (say, binary precison of a decimal number)?

On Tue, 20 Jul 1999, Richard J. Fateman wrote:

> I suspect I send a note just to JHD re this.
> Basically, unless you have the same radix on input and output and internal,
> you don't know the precision by counting digits.
> the decimal float 0.125x10^0  converted to binary needs 1 bit of precision but
> the decimal float 0.1  x10^0  converted to binary needs 4 bits  13/128.
> so decimal digits / .3010 is close, but only +-3 digits.
> (See Kahan's paper on this.)

More information about the Om mailing list