Casio 330 - Manual (Page 934)

20060301
4 Number of Digits and Precision
I Number of Digits
Standard Mode
The following applies when the check box next to the “Decimal Calculation” item on the Basic
Format dialog box is not selected.
• Up to 611 digits are stored in memory for integer values.
• Decimal values up to 15 digits are converted to fraction format and saved in memory. When
a mathematical expression cannot be converted to fraction format, the result is displayed in
decimal format.
• Values stored in memory are displayed as-is, regardless of how [Number Format] settings
(Normal 1, Normal 2, Fix 0 – 9, Sci 0 – 9) are configured (except when a decimal value is
displayed).
Decimal Mode
The following applies when the check box next to the “Decimal Calculation” item on the Basic
Format dialog box is selected.
• Values stored in Ans memory and values assigned to variables have the same number of
digits as defined for Standard mode values.
• Values are displayed in accordance with how [Number Format] settings (Normal 1, Normal 2,
Fix 0 – 9, Sci 0 – 9) are configured.
• Displayed values are rounded to the appropriate number of decimal places.
• Some applications store values using a mantissa up to 15 digits long and a 3-digit
exponent.
I Precision
• Internal calculations are performed using 15 digits.
• The error for a single mathematical expression (Decimal mode calculation error) is
p1 at the
10th digit. In the case of exponential display, calculation error is p1 at the least significant
digit. Note that performing consecutive calculations causes error to be cumulative. Error
is also cumulative for internal consecutive calculations performed for: ^(
x
y
),
x
, x !, n P r ,
n C r , etc.
• Error is cumulative and tends to be larger in the vicinity of a function’s singular point(s) and
inflection point(s), and the vicinity of zero. With sinh( x ) and tanh( x ), for example, the
inflection point occurs when x = 0. In this vicinity, error is cumulative and precision is poor.
A -4-1
Number of Digits and Precision