17 Jul
2003
17 Jul
'03
9:31 a.m.
At 05:37 AM 7/17/2003, Henry Baker wrote:
The computation of the variance below involves the addition/subtraction of very large values, which can result in massive cancellations. Thus, though while the formulae work, you may have to carry very large precisions.
I did a test of this, and problems can occur if the standard deviation is very small. I tested with 1000 data points generated with a mean of 1 and a small standard deviation. For double precision, a SD of 10^-6 is OK, 10^-7 gives problems. For extended precision, a SD of 10^-8 is OK, 10^-9 gives problems. So the std. dev. of the data set has to be pretty small to cause problems.