For each of the N-values 2, 4, 8, 16, 32, 64, 128, 256, 512, and 1,024, 50,000 samples of size N were drawn from an L-shaped population, and for each sample the Z and t statistics were calculated. The resulting distributions of 50,000 Z or t values at each sample size were then used to study the robustness of left-tailed, right-tailed, and two-tailed Z and t tests at α levels of.05,.01, and.001 (and, for Z only,.0001). The actually obtained proportion, ρ, (...) of Type I errors was often far greater or far smaller than the nominal proportion, α. Furthermore, although α is the expected value of ρ at infinite N, no N-value below 512 ever brought the deviation of p from α to within 10% of α for any t tests or one-tailed Z tests. (shrink)

The robustness of the classical tests on means (Z, t, and F) and variances (chi square and F) was investigated by obtaining 30,000 (or, sometimes, 10,000 or 150,000) values of the test statistic under assumption-violating conditions and comparing the actual proportion of Type I errors with the proportion expected when all assumptions are met. The sampling and testing conditions investigated were: population shape (L-shape or bell-shape), relative population variance (1 or 4), sample size (8, 16, or 24), nominal significance level (...) (.05,.01, or.001), and location of rejection region (left-tailed, right-tailed, or two-tailed). All tests on means were nonrobust under most of the investigated conditions, and one was nonrobust under all of them. Both tests on variances were extremely nonrobust under virtually all conditions. The worst nonrobustness of the one- and two-independent-sample Z and t tests rivaled or exceeded that of the notoriously nonrobust chi-square and F tests on variances. (shrink)

The alleged robustness of Z, t, and F tests against nonnormality and, when sample sizes are equal, of t and F tests against heterogeneity as well was investigated in a large-scale sampling study under conditions realistic to experimentation and testing in the behavioral sciences. Factors varied were: population shape (L or bell), σ1/σ2 (1/2, 1, or 2), size N of smallest sample (2, 4, 8, 16, 32, 64, 128, 256, 512, or 1,024), N1/N2 (1/3,1/2,1, 2, or 3), α (.05,.01, or.001), (...) and test tailedness (left, right, or two). In about 25% of the situations investigated, the test failed to meet a very lax criterion for robustness at every examined N value less than 100, and in 8% at every value less than 1,000; no test met the criterion in all of the situations studied before N=512. Robustness was strongly influenced by all of the factors investigated, and interactions among the influencing factors were often strong and complex. (shrink)

L-shaped distributions are not rare and are probably far more prevalent than is generally realized. They are highly conducive to nonrobustness of normality-assuming statistical tests, and they strongly resist transformation to normality. The thinner the tail of the distribution, the more unlikely it is that its L-shapedness will be detected by inspecting a sample drawn from it. Yet, as the tail of an L-shaped distribution becomes increasingly shallow, its skewness and kurtosis depart increasingly from their “normal-distribution” values, and the distribution (...) becomes increasingly conducive to drastic nonrobustness. Worse, a fairly common type of experimental situation in psychological research produces shallow-tailed L-shaped distributions. (shrink)

A quarter-century ago, during a period when belief in the robustness of classical tests on means was practically a professional shibboleth, a series of large, carefully controlled, and well-validated experiments and sampling studies (supplemented and supported by extensive mathematical derivations) dramatically showed that highly publicized claims of robustness were insufficiently qualified and that extreme nonrobustness could occur under perfectly reasonable experimental and testing conditions. When these findings were published in technical reports, they tended either to be ignored or to be (...) so misrepresented and distorted by those who cited them as to make them appear to support, rather than question, the very claims of robustness they tended to refute. Attempts to publish these iconoclastic results in many of the most renowned professional journals were met with rejection based upon reviewer comments so illogical or fatuous as to be interpreted only as an indication of either contrived obstructionism or pathetic professional incompetence. Eventual acceptance by a few refereed journals could, in every case, be interpreted as a political-type fluke. (shrink)

The optimal-pessimal paradox (Bradley, 1975) has been criticized on bizarre grounds by Childs (1980). Assumptions that it never made were attributed to it and attacked. Empirical evidence for its existence (which occupied a large portion of the criticized article) was totally ignored, and it was treated as a mere theoretically based conjecture. Originally proposed solutions to the problems it presents were dismissed and replaced by ineffectual alternatives. In spite of Childs’ claim that “the paradox, although theoretically sound, is grounded upon (...) assumptions that are empirically untenable,” the paradox makes no such assumptions and is an empirical fact. (shrink)