Evaluation of dependent variable, time effect, covariates, and covariation structure in concentration-QTc modeling: A simulation study.

Motivation: Certain studies use short-TE MRS to estimate metabolites such as gamma-aminobutyric acid. The findings of such studies hinge on the correlation of this technique to the gold-standard edited MRS technique. Goal(s): We investigate how the false negative rate and false positive rate of studies vary as a function of correlation between short-TE and MEGA-edited metabolite estimates. Approach: Simulated data was used to numerically estimate the false positive and false negative rates, using parameters obtained from the literature. Results: We show that these studies can have largely inflated false positive and false negative rates as compared to their nominally designed values. Impact: The false negative and false positive rate can be markedly inflated when using short-TE MRS to estimate metabolites such as gamma-aminobutyric acid. This calls for increased caution in interpreting results, and the accommodation of these effects when designing statistical analyses.

Ranaviruses have been identified as the etiologic agent in many amphibian die-offs across the globe. Polymerase chain reaction (PCR) is commonly used to detect ranavirus infection in amphibian hosts, but the test results may vary between tissue samples obtained by lethal and non-lethal procedures. Testing liver samples for infection is a common lethal sampling technique to estimate ranavirus prevalence because the pathogen often targets this organ and the liver is easy to identify and collect. However, tail clips or swabs may be more practicable for ranavirus surveillance programs compared with collecting and euthanizing animals, especially for uncommon species. Using PCR results from liver samples for comparison, we defined false-positive test results as occurrences when a non-lethal technique indicated positive but the liver sample was negative. Similarly, we defined false-negative test results as occurrences when a non-lethal technique was negative but the liver sample was positive. Using these decision rules, we estimated false-negative and false-positive rates for tail clips and swabs. Our study was conducted in a controlled facility using American bullfrog Lithobates catesbeianus tadpoles; false-positive and false-negative rates were estimated after different periods of time following exposure to ranavirus. False-negative and false-positive rates were 20 and 6%, respectively, for tail samples, and 22 and 12%, respectively, for swabs. False-negative rates were constant over time, but false-positive rates decreased with post-exposure duration. Our results suggest that non-lethal sampling techniques can be useful for ranavirus surveillance, although the prevalence of infection may be underestimated when compared to results obtained with liver samples.

The Effect of Testing Reliability on Visual Field Sensitivity in Normal Eyes: The Singapore Chinese Eye Study

Short‐term precipitation forecasts are critical to regional water management, particularly in the Western U.S. where atmospheric rivers can be predicted reliably days in advance. However, spatial error in these forecasts may reduce their utility when the costs of false positives and negatives differ greatly. Here we investigate whether deep learning methods can leverage spatial patterns in precipitation forecasts to (a) improve the skill of predicting the occurrence of precipitation events at lead times from 1 to 14 days, and (b) balance the tradeoff between the rate of false negatives and false positives by modifying the discrimination threshold of the classifiers. This approach is demonstrated for the Sacramento River Basin, California, using the Global Ensemble Forecast System (GEFS) v2 precipitation fields as input to convolutional neural network (CNN) and multi‐layer perceptron models. Results show that the deep learning models do not significantly improve the overall skill (F1 score) relative to the ensemble mean GEFS forecast with bias‐corrected threshold. However, additional analysis of the CNN models suggests they often correct missed predictions from GEFS by compensating for spatial error at longer lead times. Additionally, the deep learning models provide the ability to adjust the rate of false positives and negatives based on the ratio of costs. Finally, analysis of the network activations (saliency) indicates spatial patterns consistent with physical understanding of atmospheric river events in this region, lending additional confidence in the ability of the method to support water management applications.

Testing for the presence of a condition may give an incorrect result Sensitivity and specificity of a test depend upon the chosen cut-off value ‘No difference’ usually means ‘these data do not suggest a difference’ The power of a test needs to be known to conclude that there is no difference Power calculations should be considered if the null hypothesis is not rejected Calvin Coolidge was a taciturn president of the United States, nicknamed ‘Silent Cal’. When told that he had died, the acerbic writer Dorothy Parker remarked, ‘How could they tell?’ Telling if something has not happened is a perennial problem in science, particularly so in laboratory experiments. Why should this be? In the case of Cal Coolidge, they probably could tell. But in science, random events make it hard to tell, with certainty. There are also different ways of ‘telling’. Mortality is a good categorical measure, an unequivocal endpoint. When diagnostic tests are used to define categories, we meet the concepts of ‘false positive’ and ‘false negative’. Here, we use the expression ‘false’ to indicate that the conclusion we drew from the test would be incorrect. In the example we use below, we have a test for ‘alive’. If we know the true condition, we can classify test results as true or false. Classifying a dead animal as alive would be a false positive. True and false are words usually applied to a diagnosis or similar categorical events but also crop up in other aspects of statistical inference. Categorical analysis and categorical logic (it's either there or it isn’t) are not the same as the frequentist logic (the likelihood is small) that is used in common statistical analysis, and this leads to a great deal of argument and misunderstanding. Our previous comment that statistics has an impoverished vocabulary rings true: in this instance, the concepts of false positive and false negative conclusions are applied in different ways to different procedures, causing confusion. Considering categories, imagine that a pool that has become overheated in the sun: some of the frogs in it have died. Our observations suggest that the frogs on the surface of the pool are still alive, and the ones at the bottom of the pool are dead. Is this correct? A random sample produces the data in Fig. 1. Not all the frogs on the surface are alive. If we used ‘on the surface’ to indicate being alive, we would falsely attribute life to 10 frogs. Since 107 frogs are dead, this is a false positive rate of 10/107 (which we only know if we have an unequivocal means of determining death). The positive predictive power of the test for life being present, if we were to look at the next frog on the surface, would be 80/90. Unfortunately, expressions such as sensitivity, false positive, and so on can be used in two different senses. Here we assume we already have an exact means to determine death, much in the way that some statistics theory will start by defining a population. Thus we consider that the false positive rate of our test for ‘alive’ is the proportion of dead frogs on the surface, expressed as a proportion of the frogs that really ARE dead. An alternative view would be if we were to consider the proportion of frogs on the surface that are later found to be dead, we would then be considering the outcome of the test. The diagnostic false positive rate would be 10/90. This is a diagnostic probability, and this form is less commonly used. Many diagnostic tests don't rely on categories like this: a continuous variable is often used to distinguish categories. Suppose we measure the blood lactate concentration in frogs that are still alive, and relate these concentrations to subsequent survival, we might obtain the results shown in Fig. 2. Plasma lactate values in frogs after overheating, related to survival The groups are different (P < 0.001, Mann–Whitney U test). The median value for survivors is 1.29, and for non-survivors, 1.86 (mmol l−1). Although the median concentrations are different, some frogs with high lactate concentrations can survive. Is there a particular lactate concentration that will distinguish survival from non-survival? The capacity to predict outcome (sensitivity and specificity) alters as we choose different cut-off values. Using a series of cut-off values we can plot sensitivity and (1 – specificity) for each chosen value (Fig. 3). This is known as a Receiver-Operator Characteristic Curve, or ROC curve, because one of its first applications was to find how operators of radar receivers distinguish ‘real’ echoes from background noise. In this case, we are looking for survivors using a lactate concentration that is less than the cut-off value we have chosen. A Receiver Operator Characteristic curve for the capacity of a lactate concentration below a chosen value to predict survival An effective diagnostic test should have a large area below the curve, which would show a combination of good sensitivity and specificity. Since tests vary, so does opinion about the area below the curve that indicates a good test. It should be substantially greater that 50%, which would be no better than chance alone, and the closer to 100% the better. The large overlap of individual values between the two groups in Fig. 2 suggests that lactate concentration is a poor indicator of survival, as there is no value that combines useful specificity and sensitivity. Any cut-off values we could choose would have too many false positive or false negative conclusions. Despite this, there is a statistically significant difference according to the Mann–Whitney test, showing that a significant difference doesn't mean the measure is the basis of a good diagnostic test. In other forms of statistical inference, an expression such as ‘false positive’ is less appropriate and is often not well defined, hidden in the logic of these tests. For example, the t test does not provide black or white conclusions: statistical inference is restricted to judging how large a part chance could play in what we observe. Frequentist statistical tests such as the t tests are measures of uncertainty. The usual laboratory experiment question ‘do these data show an effect?’ has to be worded ‘are these samples statistically different?’ As we know, the question then is re-worded ‘How likely are these results if the samples came from the same population?’ Let's say our test result is that P= 0.01, which would usually be taken to indicate there is an ‘effect’. This is not so: this P value is the probability that we conclude there is an effect, and we still could be wrong. Although small, a chance still exists that we would be wrong, about 1 in 100 times. This is known as the alpha error, the possibility that we might classify our result as showing an effect, when in fact there is none. In the long run, if we were to repeatedly sample two populations that actually were the same, then results like the ones we found would be found 1 in 100 times. This is sometimes called the ‘false positive’ rate. However, it's based on the premise that there is NO difference and is perhaps better called the ‘false conclusion’ rate. Reasoning for ‘no effect’ is even less certain, and also often misconstrued. If the P value is 0.12, then we judge that the results are quite possible, assuming the data are from the same population. If we did the same experiment again and again, then we might obtain data like this in almost 1 out of 8 experiments. Is this unlikely? We usually choose to decide this is quite likely, and thus do not reject the null hypothesis: if we rejected it, the chance of a false conclusion is too great. This ‘chance of a false conclusion’ is often misleadingly called a ‘false negative rate’. The hypothesis the test uses, which is the NULL hypothesis, is resoundingly negative; there are no positive results lurking here. Positive results come if we are persuaded, by an estimate of probability (the P value), that the null hypothesis is not tenable. The context of the question ‘is there an effect?’ is relevant. The consequence of the answer ‘no effect’ may be unimportant. However, in some cases the consequence of concluding that there is no difference, because a difference is not apparent, could be devastating. How often do we hear the claim ‘there is no evidence that X is toxic?’ to assert that X is not toxic? Immediately, the sceptic will ask how toxic, and how much evidence? The statistician may add another qualification: how much variation is present? A reasonable example (not a perfect analogy) is the radar operator: there is a greater certainty about a blip on the screen if the signal is large (accept the answer only if it's very likely), if you can keep looking at the screen (improve the sample size), and if there is not a lot of background noise (Fig. 4). In each of these examples of signal averaging, there is an identical signal with added random noise. The signal is more easily seen if the noise is reduced (vertical arrow). Signal detection is improved by taking more samples to allow more averaging (horizontal arrow). The final feature of signal detection is how certain we wish to be (from ‘just discernible’ to ‘barn door’) that the event is not random (a concept analogous to the P value). In this example, a signal is always present, but we just can't be sure in all the cases. This is equally so when statistics are applied to the null hypothesis. We have to consider the beta error: the possibility that we might classify our result as showing no effect where in fact there could be a difference. To demonstrate, consider two populations that are known to be different, as shown in Fig. 5. A, random samples from two different populations, low and high, where the population means differ by 0.1 mmol l−1. A t test finds the P values shown. B, a valuable further analysis that should be applied before erroneously assuming ‘no difference’ is to inspect the 90% confidence limits of the difference between the means. This indicates the size of the possible differences. Here, a small sample of 20 per group, and a small true difference between the groups, does not reject the null hypothesis, with a P value of 0.05. In this particular case, there is indeed a true difference in the populations from which these samples have been taken, but the power of the test applied has been inadequate. The power of a test is the chance of finding a defined difference: a power of 90% would mean that there was only a 10% chance of failing to find a difference of the defined magnitude. Thus power is (1 −β). The 10% false negative conclusion rate or beta error, is also known as a type II error. Some call this a ‘false negative’ but this could be misleading since there is never a ‘true’ negative here. The frequentist definition is that if the experiment were repeated over and over, the conclusion would be incorrect occasionally, and the rate is the beta error. For the example we have chosen, the relationships between power, sample size and the difference we seek to detect are shown in Fig. 6. Using the samples in Fig. 4, the power of the test (the chance of finding a specified difference between samples, given that it exists) is shown, where n= 20 and n= 40, for two different P values. The populations that provided the example data in Fig. 4 were different, by 0.1 mmol l−1, indicated by the vertical dotted line. If P is accepted at 0.05 and n= 40, the power of the test is only about 40%. The retrospective use of power calculations has been often criticized, but this is usually because the authors were seeking to reject a null hypothesis (they would have liked to find a difference). If a null hypothesis cannot be rejected, this is not sufficient evidence that there is NO difference. Claiming ‘no difference’ can be a serious error. If we wish to convince ourselves, and others, that there is no difference, we must be sure that the test was capable of rejecting a specifically defined difference, if it were present. If a small difference is considered important, and variation in the population is substantial, then a small sample will be inadequate. Put simply, the study is underpowered. Although there are software packages to calculate power, a simple approach for a simple two group comparison is to plot the 90% confidence limits of the difference between the means (Fig. 5B). This will show the likely range of differences that could be detected. The logical error (concluding that no significant difference is equivalent to the same) is frequently made worse when test results are compared, for example: the effect of substance A causes a significant difference, but substance B does not; thus A has more effect than B. As we explain above, several factors, not explicitly stated, could affect this result. It could well be that B has a greater, but more varied, effect. Because most tests are looking for a positive, many scientists fail to consider negatives adequately. In the old song ‘Accentuate the positive’ the accomplished song writer Johnny Mercer also cautioned to ‘eliminate the negative, don't mess with Mr In-Between’.

Recent advances in microarray technology have opened new ways for functional annotation of previously uncharacterised genes on a genomic scale. This has been demonstrated by unsupervised clustering of co-expressed genes and, more importantly, by supervised learning algorithms. Using prior knowledge, these algorithms can assign functional annotations based on more complex expression signatures found in existing functional classes. Previously, support vector machines (SVMs) and other machine-learning methods have been applied to a limited number of functional classes for this purpose. Here we present, for the first time, the comprehensive application of supervised neural networks (SNNs) for functional annotation. Our study is novel in that we report systematic results for ~100 classes in the Munich Information Center for Protein Sequences (MIPS) functional catalog. We found that only ~10% of these are learnable (based on the rate of false negatives). A closer analysis reveals that false positives (and negatives) in a machine-learning context are not necessarily "false" in a biological sense. We show that the high degree of interconnections among functional classes confounds the signatures that ought to be learned for a unique class. We term this the "Borges effect" and introduce two new numerical indices for its quantification. Our analysis indicates that classification systems with a lower Borges effect are better suitable for machine learning. Furthermore, we introduce a learning procedure for combining false positives with the original class. We show that in a few iterations this process converges to a gene set that is learnable with considerably low rates of false positives and negatives and contains genes that are biologically related to the original class, allowing for a coarse reconstruction of the interactions between associated biological pathways. We exemplify this methodology using the well-studied tricarboxylic acid cycle.

"Test and Standard Precautions"-Is It Enough to Protect Us From False-Negative Severe Acute Respiratory Syndrome Coronavirus 2 Test Results?

To evaluate the false-negative and false-positive error rates both in a screening and a non-screening population. A total of 4192 prostatic biopsies were reported in a 6-year period by 15 consultant histopathologists, two of whom had an interest in uropathology and were deemed to be specialists (J.O. and C.S.). All biopsies were reviewed prior to the multidisciplinary team (MDT) meeting. The overall false-negative rate was 1.7% (screening 2.1%, non-screening 1.5%). The overall false-positive rate was 0.5% (screening 0.9%, non-screening 0.4%). These error rates varied among pathologists, with the false-negative rate ranging from 0% to 9.3%, and the false-positive rate ranging from 0% to 3.8%. The false-negative rate was three times greater than the false-positive rate, showing that detection of significant pathology is far greater in the negative biopsies. More errors occurred in the screening population than in the non-screening population. The consultants making the most errors were non-specialists, but the specialists also made false-negative errors, suggesting that just using specialist reporting alone would not have eradicated errors.

Food and Agriculture Organization/World Health Organization (FAO/WHO) recommended that IgE cross-reactivity between a transgenic protein and allergen be considered when there is >or= 35% identity over a sliding "window" of 80 amino acids. Our objective was to evaluate the false positive and negative rates observed using the FAO/WHO versus conventional FASTA analyses. Data used as queries against allergen databases and analyzed to assess false positive rates included: 1,102 hypothetical corn ORFs; 907 randomly selected proteins; 89 randomly selected corn proteins; and 97 corn seed proteins. To evaluate false negative rates of both methods: Bet v 1a along with several crossreacting fruit/vegetable allergens and a bean alpha-amylase inhibitor were used as queries. Both methods were also evaluated for their ability to detect a putative nonallergenic test protein containing a sequence derived from Ara h 1. FASTA versions 3.3t0 and 3.4t25 were utilized. Data indicate a conventional FASTA analysis produced fewer false positives and equivalent false negative rates. Conventional FASTA versus sliding window derived E scores were generally more significant. Results suggest a conventional FASTA search provides more relevant identity to the query protein and better reflects the functional similarities between proteins. It is recommended that the conventional FASTA analysis be conducted to compare identities of proteins to allergens.

The influence of fatigue and related factors on automated perimetric testing was evaluated in both eyes of sixteen normal observers and sixteen patients with early-to-moderate visual field loss using a Digilab 750 automated perimeter and a customized test procedure. False positive rate, false negative rate, and detection sensitivity at 5,10,15, and 20 degrees eccentricity were measured in 1.5-rain intervals throughout a 21-min visual field examination. Half of the normal observers and patients with visual field loss were given a briefrest(1.5-min) midway through each visual field exam to determine whether this would reduce fatigue effects. Our findings revealed that patients displayed considerably higher average false positive and false negative rates than normal observers. However, neither the patients nor normal observers demonstrated any consistent changes in false positive rate or false negative rate as a function of testing duration. In contrast, both normal observers and patients showed an average decrease in sensitivity as a function of increasing test duration with the magnitude of the time-dependent sensitivity loss becoming greater with increasing stimulus eccentricity. Patients demonstrated a larger time-dependent sensitivity loss than normal observers, averaging ~4dB at 20 degrees eccentricity. The introduction of a brief pause midway through the test procedure appeared to reduce the time-dependent sensitivity loss for the second half of the test procedure, especially for greater eccentricities.

Cost-sensitive Feature Selection for Support Vector Machines

BackgroundThe objective of this study was to investigate if application of United Kingdom National Osteoporosis Society (UK-NOS) triage approach, using calcaneal quantitative ultrasound (QUS), phalangeal radiographic absorptiometry (RA), or both methods in combination, for identification of women with osteoporosis, would reduce the percentage of women who need further assessment with Dual Energy X-ray Absorptiometry (DXA) among older women with a high prevalence of falls.MethodsWe assessed 286 women with DXA of hip and spine (Hologic Discovery) of whom 221 were assessed with calcaneal QUS (Achilles Lunar), 245 were assessed with phalangeal RA (Aleris Metriscan), and 202 were assessed with all three methods. Receiver operator characteristics (ROC) curve for QUS, RA, and both methods in combination predicting osteoporosis defined by central DXA were performed. We identified cutoffs at different sensitivity and specificity values and applied the triage approach recommended by UK-NOS. The percentage of women who would not need further examination with DXA was calculated.ResultsMedian age was 80 years (interquartile range [IQR]) [75–85], range 65–98. 66.8% reported at least one fall within the last 12 months. Prevalence of osteoporosis was 44.4%. Area under the ROC-curve (AUC) (95% confidence interval (CI)) was 0.808 (0.748-0.867) for QUS, 0.800 (0.738-0.863) for RA, and 0.848 (0.796-0.900) for RA and QUS in combination. At 90% certainty levels, UK-NOS triage approach would reduce the percentage of women who need further assessment with DXA by 60% for QUS, and 43% for RA. The false negative and false positive rates ranged from 4% to 5% for QUS and RA respectively. For the combined approach using 90% certainty level the proportion of DXAs saved was 22%, the false negative rate was 0% and false positive rate was 0.5%. Using 85% certainty level for the combined approach the proportion of DXAs saved increased to 41%, but false negative and false positive values remained low (0.5%, and 0.5% respectively).ConclusionsIn a two-step, triage approach calcaneal QUS and phalangeal RA perform well, reducing the number of women who would need assessment with central DXA. Combining RA and QUS reduces misclassifications whilst still reducing the need for DXAs.

Relationships between age and visual field (VF) reliability indices were investigated using a large real-world dataset (42,421 VF data points from 11,525 eyes of 5930 subjects). All VFs tested and stored at Shimane University Hospital between 1988 and 2019 were exported. Correlations between age, mean deviation (MD), pattern standard deviation (PSD), and reliability indices including fixation losses (FLs), false negatives (FNs), and false positives (FPs) were analyzed. The mean ± standard deviation age was 65.0 ± 15.1 years; MD—−6.9 ± 8.1 decibels (dB); PSD—6.3 ± 4.6 dB; FL—8.6 ± 11.7%; FN—5.3 ± 8.3%; and FP—2.6 ± 5.0%. Univariate analyses showed strong associations between age and FNs (correlation coefficient, ρ = 0.20, p < 0.0001) and MD (ρ = −0.21, p < 0.0001). All FLs, FNs, and FPs were lowest during the third decade (20–29 years) of life. FLs were elevated consistently after that decade, and FNs were elevated sharply after the seventh decade. FPs were relatively stable after the fourth decade (30–39 years). Mixed-effect regression analyses in subjects 40 years and older showed that older age was associated with worse FLs (p < 0.0001) and FNs (p < 0.0001) but not FPs (p = 0.4126). Aging affects FLs and FNs with different modes but had minimal effects on FPs. Decreased VF sensitivity, deteriorated macular function, and technical difficulties with testing may be mechanisms of age-related changes in FLs and FNs.

An Evaluation of Conventional Culture, invA PCR, and the Real-Time PCR iQ-Check Kit as Detection Tools for Salmonella in Naturally Contaminated Premarket and Retail Turkey

Several different types of detectors, software, and hardware designs are employed in instruments used to monitor radioactive content of freight shipments in order to detect illicit trafficking of nuclear materials. The instruments can be container monitors, hand-held radiation detectors, mobile analysis systems, or fixed radiation portal monitors. However, within the various groupings (e.g., portal monitors), all instruments are expected to solve the same problem, that is, to detect and identify any radioactive material present according to the prescribed investigative procedure or CONOPS. The best way to compare the performance of different instruments is with a numerical score or Figure of Merit (FOM). The FOM must quantify the performance of the instrument with respect to true positives (TP), false positives (FP), and false negatives (FN). The minimization of FN for certain radionuclides (e.g., uranium and plutonium or other Special Nuclear Material (SNM)) is more important than the minimization of FN for non-threat nuclides (e.g., low level NORM). In a similar way, the minimization of FP for SNM is more important than falsely reporting the common NORM nuclides which represent no threat. System performance depends on the measurement detail (e.g., transit speed through a portal, measurement time in general, amount and distribution of NORM), therefore the test conditions should also be included in the statement of the FOM. The FOM is expected to vary significantly with the above measurement details. A FOM has been developed based on the number of true positives (TP), the number of false important positives (FIP), the number of FP, the number of true positives for SNM (TPSNM), and the number of false positives for SNM (FPSNM). This formula rates the overall performance with extra weight given to FP and FN for SNM. Examples will be shown for testing of portal monitors.