Ophthalmology is a technologically advancing field with the constant production of new instrumentation.1 Instruments are usually released on the market with manufacturers claiming performance in various data formats, and this is typically followed by clinical evaluation studies by independent users.2–5 Ubiquitous issues in this area include, not least, the variety of methods, both practically and statistically, and the various terminology differences. It is becoming exceedingly difficult for clinicians and researchers to make sense of study results that have used nonstandard methodology in addition to confounding terminology. Terms such as consistency, precision, reliability, accuracy, repeatability, reproducibility, and agreement are just some examples of the many terms that frequently appear in the ophthalmic literature with inconsistent definitions and synonymous usage. The purpose of this editorial is to provide guidance to prospective authors conducting precision (repeatability and reproducibility) studies in terms of basic concepts, terminology, statistical methods, sample size considerations, study design, use of 1 eye or 2 eyes, and worked examples. A number of bodies in the scientific world have devised their own terminology and statistical recommendations, some of which are similar and others markedly different.6 An example is the International Organization for Standardization (ISO), an independent nongovernmental membership organization and the world’s largest developer of voluntary International Standards. ISO has published more than 20 500 International Standards covering almost every industry, including ophthalmology. Examples include standards relating to contact lenses, contact lens care products, intraocular lenses, intraocular implants, spectacle lenses, and ophthalmic instruments. The ISO standards might differ markedly even within ophthalmic instruments, an example being differing statistical methods for tonometry and topography.7–9 A full list is available on the ISO website (www.iso.org). Such ISO standards are developed by following a formal methodology involving a multi-stakeholder process. They are organic and dynamic, and developments might take years. This editorial is not set to replace any of the ISO standards or suggest that a single approach should be applied to every situation but rather provides a practical commentary on precision that draws but does not fully represent, replace, or supplant existing standards. Accuracy (trueness and precision) The accuracy of a measurement may be described by 2 terms; that is, trueness and precision. The trueness of a measurement refers to the closeness between the mean of a large number of results and the true or an accepted reference value. With regard to ophthalmology, it is unlikely that the trueness of a measurement will be possible because in general there are no true values or accepted reference values for any measurement relating to the eye. Exceptions are trueness of test objects such as surfaces cut on contact lens lathes. It is also not possible to use invasive or destructive methods for diagnostic purposes such as manometry in the measurement of intraocular pressure (IOP). Precision, on the other hand, refers to the closeness between repeated measurements. Precision arises as the result of variability found with repeated measurements on presumably identical subjects. Five factors may contribute to the variability found in repeated measurements: (1) observer, (2) instrument used (eg, model, serial number), (3) instrument calibration, (4) environment (temperature, humidity, etc), and (5) time interval between measurements. Precision consists of both repeatability and reproducibility. Repeatability refers to the variability in repeated measurements in which the above factors are considered constant. This is commonly referred to as the intraobserver repeatability;10 however, this is not wholly correct. Intraobserver is used to indicate that the observer is constant, but, as can be seen from the definition of repeatability, several other factors are also considered constant. Therefore, the term repeatability should be used instead of intraobserver repeatability to describe the situation in which all factors are assumed constant. Reproducibility, on the other hand, refers to variability when 1 or more of the 5 factors vary. The ISO Accuracy (Trueness and Precision) of Measurement Methods and Results document recommends strict methods to determine the maximum variability in measurement.11 Although a change in observer, instrument, calibration, environment, and time might provide an excellent estimate of instrument reproducibility, such a comprehensive comparison might not be particularly practical. Notably, they recommend that a minimum of 8 laboratories be used in the assessment of reproducibility.11 Hence, our recommendation consists of reporting reproducibility as per the factor or factors that have been varied. However, a better assessment of instrument reproducibility will be achieved when a greater number of factors are varied. Repeatability and reproducibility consequently express the 2 extremes of precision; the former (repeatability) describing the minimum and the latter (reproducibility) describing the maximum variability in results. A third term, accuracy, incorporates both faces of precision but can only be claimed when both the trueness and precision of measurements have been determined. Calculating precision (repeatability and reproducibility) Precision estimates can be expressed in terms of standard deviations. Repeatability Determining the repeatability of repeated measurements on a single instrument is a simple calculation. However, before the study, practical considerations are required. Under repeatability conditions, factors 1 to 5 are assumed constant. Hence, the time between repeated measurements should be the shortest possible; however, one must consider this carefully in the context of the study. For example, in the evaluation of the repeatability of a corneal topographer, repeated measurements could be taken one after the other without delay. Whereas in the evaluation of an applanation tonometer in the measurement of IOP, repeated measurements without delay are confounded by factors such as the effect that corneal applanation might have on IOP (potentially reducing IOP). If a number of hours are left between measures to account for such factors, other factors might take effect, such as the diurnal fluctuation in IOP. In such circumstances, available evidence and common sense must prevail in the study design. Assuming that one has 2 repeated measurements on a number of subjects and factors 1 to 5 are constant, a one-way analysis of variance (ANOVA) can be used to calculate the within-subject standard deviation (Sw). Most statistical software capable of performing a one-way ANOVA provides a table displaying the mean square. This equals the within-subject variance, and the square root of this value can be calculated, which will equal the Sw. The ANOVA method has the added advantage of being able to deal with more than 2 repeated measures. By way of example, we will consider central corneal thickness (CCT) data extracted from a precision study of the Pentacam HR (Oculus).5 Data from 20 subjects will be used for demonstration purposes but should not be considered an appropriate sample size. Two repeated measurements of CCT were obtained by 1 examiner (Table 1).Table 1: Two repeated measurements of central corneal thickness in a group of 20 subjects by one examiner.Data were entered into SPSS (version 20, IBM) as follows: In the first column (VAR00001), subject numbers were entered from 1 to 20 and again from 1 to 20 (i.e., stacked). In the second column (VAR00002), the first and second CCT measurements were stacked in a similar manner, corresponding to the relevant subject. Along the toolbar on the top, “Analyze” was clicked; then “Compare Means,” and then “One-Way ANOVA.” In the popup window, VAR00002 (i.e., the CCT measurement) was inserted into “Dependent List” and VAR00001 was inserted into “Factor” and “OK” was clicked. The resulting output table is shown in Table 2.Table 2: SPSS output table from a one-way analysis of variance from data in Table 1. Equating the square root of the mean square within groups provides the Sw. In our example, the mean square within groups is 36.425, hence √36.425 = 6.035. Therefore the Sw or repeatability (Sr) of the CCT in this example is 6.035 μm. Considering the 95% confidence intervals of Sr, the repeatability limit (r) is 1.96√2 × Sr, which provides an estimate of the limits within which 95% of measurements should be. In this example, the repeatability limit (r) is 1.96√2 × 6.035, which equals 16.728 μm. Therefore, 2 values can be reported for repeatability; that is: (1) repeatability (Sr) and (2) repeatability limit (r), which equals 1.96√2 × Sr. To manually calculate Sw, the following calculations and equations should be used: Calculate the mean of the 2 repeated measures for each subject: (2) Calculate the variance of the 2 repeated measures for each subject: where n is the number of subjects. (3) Calculate the sum of all the variances and then the mean. (4) Calculate the square root of the mean variance, which provides Sw. Reproducibility Reproducibility (SR) is calculated in the same way as repeatability (Sr). In the CCT example above, now consider that an additional examiner performs 1 measurement of CCT 1 week later on the same 20 subjects. One set of results from Examiner 1 and the 1 set of results from Examiner 2 are compared for reproducibility. We would perform a one-way ANOVA for the 2 measurements and perform the same calculation in the same way as described above. Similarly, the reproducibility limit (R) can be reported and is determined by the formula 1.96√2 × SR. Therefore, 2 values can be reported for reproducibility; that is, (1) reproducibility (SR) and (2) reproducibility limit (R), which equals 1.96√2 × SR. It is important to specify which factors have been varied in the assessment of instrument reproducibility. In the above example, the reproducibility was assessed by a change in observer and time. Sample size in precision studies A sample-size estimate for precision experiments can be calculated by use of the within-subject standard deviation (Sw). This estimation will depend on the number of subjects (n) and the number of repeated measurements (n′). One must decide to what level of confidence one wants to estimate the quantity of measurement. This is typically chosen to be within 10%. The sample size, n required depending on the basis of n′ repeated measurements, is given by the formula: Because Sw is on both sides of the formula it can be canceled, the sample size formula can therefore be simplified to: By way of example, if we were assessing the repeatability of a tonometer with 2 repeated measurements and we wanted to estimate IOP to within 10% (corresponding to 1.5 mm Hg at 15 mm Hg), we would require the following sample size: Hence, 192 subjects would be the estimated sample size for this study. If a higher degree of confidence such as 5%, were required, Equation 3 can be changed to: Figure 1 shows the relationship between the sample size and the degree of uncertainty (range within which the expectation is asserted to lie), using CCT estimates as an example. This model can be used for any other clinical measure to calculate the range of uncertainty for a given sample size.Figure 1: Relationship between sample size and the degree of uncertainty in estimates of central corneal thickness (CTT). The area within the 2 function curves represents the range of uncertainty in the measure. The values along the upper curve show the sample sizes (n) and the corresponding range of uncertainties in percentage. Small sample sizes are associated with more uncertainty, and large sample sizes are associated with less uncertainty.Table 3 shows an illustrative example of the impact of repeated measures and level of confidence on the sample size. As can be seen from Table 3, using a greater number of repeated measures requires a smaller sample size. However, increasing the number of repeated measurements to allow for a smaller sample size is cautioned. From a practical viewpoint, a small sample might restrict the range of values over which precision is tested and bias the result to the variation in the population because some individuals might have more variation than others. Therefore, a large sample is advocated in instrument precision studies to ensure that the instrument’s precision is adequately assessed over a greater range of measurement.Table 3: Sample size required for precision studies based on the number of repeated measures and the confidence in the estimate.Table 3 and Figure 1 clearly show that small samples provide little confidence in the result of a repeatability or reproducibility study. For a study design with 2 repeated measures and a sample of less than 50, there is at least 20% uncertainty in the repeatability or reproducibility result. Such studies have little value to the clinician and should not be published. Study design A number of ophthalmic devices provide automatic measurement with minimal examiner skill required.12–16 Hence, a change in examiner during the reproducibility assessment is unlikely to be the source of any significant precision error. On the contrary, if one is investigating the reproducibility of Goldmann tonometry with a change in observers, reproducibility is likely to become a much more significant factor. However, it might be more important to assess the reproducibility of automated devices, such as the Pentacam, by a change in the instrument used (i.e., 2 Pentacam devices), whereas it might be more appropriate to assess reproducibility of Goldmann tonometry by a change in observer. Other reproducibility factors are vital to consider during the design phase of a precision study. Instrument calibration procedures are usually recommended by manufacturers at set time intervals or after a specified number of uses. If, for example, a year-long study commences and 1 of the primary outcomes is IOP measurement with the Goldmann tonometer, instrument calibration is important during this time frame. The precision of the tonometer is likely to decrease beyond the time point when calibration is required. Knowledge of instrument decalibration and recalibration might be the key reproducibility factor affecting the study. Despite the majority of ophthalmic devices being used in standard clinical settings, it is prudent to consider environmental factors in certain circumstances. Examples include visual acuity testing or infrared autorefraction under different lighting conditions and excimer laser properties under different humidity or temperature conditions. With regard the reproducibility time interval, there are 2 main issues. First, there is the natural fluctuation in the measurement variable over short periods of time, such as the diurnal variation in IOP. Second, there is an alteration in the variable over longer periods, such as the reduction in the amplitude of accommodation, which would be important if one were assessing the precision of a device that measures objective accommodation.17 Precision studies should be as comprehensive as possible and assess as many of the output variables that an instrument produces rather than numerous smaller studies reporting the precision of specific measurements. For example, the Pentacam device produces a plethora of measurements across the anterior segment of the eye, yet many published precision studies have isolated their assessment to specific measurements such corneal thickness,18 corneal curvature,19 posterior corneal measurements,20 and anterior chamber volume,21 as opposed to the comprehensive study.22 This is a common trend in precision studies of many ophthalmic devices. Other statistics for precision measurements A number of statistics have been used in the past in the assessment of instrument precision, such as correlation coefficients (Pearson product moment and intraclass)23 and the Bland-Altman limits of agreement method.24 The Pearson correlation coefficient falls short on a number of issues and should be reserved for use in assessing whether a linear correlation exists between 2 variables. It is often used to incorrectly assess agreement between 2 tests or methods. There may be a strong positive correlation between 2 variables but poor agreement. The Pearson correlation coefficient assesses the linear relationship between 2 variables only, so there might a perfect correlation of 1, but the relationship between the 2 variables might be 2 to 1. The Bland-Altman limits of agreement method is related to the statistical precision methods described but should be reserved for determining the agreement between 2 clinical methods to ascertain whether they might be used interchangeably.25,26 Nonparametric statistics All data should be assessed for adherence to a normal distribution by plotting a histogram or by using statistical tests such as the Shapiro–Wilk or Anderson–Darling tests.27,28 If data follow a normal distribution, parametric statistics as described can be used. If data do not follow a normal distribution, data transformation methods such as a logarithmic transformation should be attempted in the first instance. If this is unsuccessful and data remain skewed, nonparametric statistics should be applied. For example, this commonly occurs with astigmatism, in which the median vector differences and the 95th percentile might be reported as an alternative.29–31 One eye or two A common statistical issue in precision studies is whether 1 or 2 eyes of patients should be used. Some studies have used only 1 eye,32–37 whereas other studies have used both eyes of each patient.38–40 The problem arises as a result of the greater correlation between the 2 eyes of an individual compared with the correlation between 2 individuals.41–43 Exceptions can occur in cases of asymmetric eye disease, such as keratoconus. Another consideration is that the ANOVA method requires variables to contain independent data. Therefore, when assessing the precision of devices that measure variables that are highly correlated between the 2 eyes of an individual, such as IOP or refractive error, there are 2 options. The first option is to use only 1 eye of the individual. The second option is to use both eyes but control for the non-independence of measurements. This control is usually possible with most statistical software packages. Further considerations and statistical tests that can be used in this situation are provided in the review article by Armstrong.44 Summary The growing choice of ophthalmic equipment has driven research into their precision. It is important that these studies are designed and apply appropriate statistical methods with the correct corresponding terminology. The ANOVA method provides the Sw as a descriptor of repeatability and reproducibility. The repeatability and reproducibility limits can be determined to provide an estimate of the likely limits within which 95% of measurements should be. One must also pay particular attention to an adequate sample size and factors such as the use of 1 eye or 2 eyes and non-normally distributed data.
Stored only when both papers are in the corpus — 245 citations exist upstream.