In _Reliable Reasoning_, Gilbert Harman and Sanjeev Kulkarni -- a philosopher and an engineer -- argue that philosophy and cognitive science can benefit from statistical learning theory, the theory that lies behind recent advances in machine learning. The philosophical problem of induction, for example, is in part about the reliability of inductive reasoning, where the reliability of a method is measured by its statistically expected percentage of errors -- a central topic in SLT. After discussing (...) philosophical attempts to evade the problem of induction, Harman and Kulkarni provide an admirably clear account of the basic framework of SLT and its implications for inductive reasoning. They explain the Vapnik-Chervonenkis dimension of a set of hypotheses and distinguish two kinds of inductive reasoning. The authors discuss various topics in machine learning, including nearest-neighbor methods, neural networks, and support vector machines. Finally, they describe transductive reasoning and suggest possible new models of human reasoning suggested by developments in SLT. (shrink)

We compare Karl Popper’s ideas concerning the falsifiability of a theory with similar notions from the part of statistical learning theory known as VC-theory . Popper’s notion of the dimension of a theory is contrasted with the apparently very similar VC-dimension. Having located some divergences, we discuss how best to view Popper’s work from the perspective of statistical learning theory, either as a precursor or as aiming to capture a different (...) class='Hi'>learning activity. (shrink)

Statistical Learning Theory (e.g., Hastie et al., 2001; Vapnik, 1998, 2000, 2006) is the basic theory behind contemporary machine learning and data-mining. We suggest that the theory provides an excellent framework for philosophical thinking about inductive inference.

In this article, we provide a tutorial overview of some aspects of statistical learning theory, which also goes by other names such as statistical pattern recognition, nonparametric classification and estimation, and supervised learning. We focus on the problem of two-class pattern classification for various reasons. This problem is rich enough to capture many of the interesting aspects that are present in the cases of more than two classes and in the problem of estimation, and many (...) of the results can be extended to these cases. Focusing on two-class pattern classification simplifies our discussion, and yet it is directly applicable to a wide range of practical settings. (shrink)

Nelson Goodman's new riddle of induction forcefully illustrates a challenge that must be confronted by any adequate theory of inductive inference: provide some basis for choosing among alternative hypotheses that fit past data but make divergent predictions. One response to this challenge is to distinguish among alternatives by means of some epistemically significant characteristic beyond fit with the data. Statistical learning theory takes this approach by showing how a concept similar to Popper's notion of degrees of (...) testability is linked to minimizing expected predictive error. In contrast, formal learning theory appeals to Ockham's razor, which it justifies by reference to the goal of enhancing efficient convergence to the truth. In this essay, I show that, despite their differences, statistical and formal learning theory yield precisely the same result for a class of inductive problems that I call strongly VC ordered, of which Goodman's riddle is just one example. (shrink)

The idea that “simplicity is a sign of truth”, and the related “Occam’s razor” principle, stating that, all other things being equal, simpler models should be preferred to more complex ones, have been long discussed in philosophy and science. We explore these ideas in the context of supervised machine learning, namely the branch of artificial intelligence that studies algorithms which balance simplicity and accuracy in order to effectively learn about the features of the underlying domain. Focusing on statistical (...) learning theory, we show that situations exist for which a preference for simpler models provably slows down, instead of favoring, the supervised learning process. Our results shed new light on the relations between simplicity and truth approximation, which are briefly discussed in the context of both machine learning and the philosophy of science. (shrink)

Nelson Goodman’s new riddle of induction forcefully illustrates a challenge that must be confronted by any adequate theory of inductive inference: provide some basis for choosing among alternative hypotheses that fit past data but make divergent predictions. One response to this challenge is to distinguish among alternatives by means of some epistemically significant characteristic beyond fit with the data. Statistical learning theory takes this approach by showing how a concept similar to Popper’s notion of degrees of (...) testability is linked to minimizing expected predictive error. In contrast, formal learning theory appeals to Ockham’s razor, which it justifies by reference to the goal of enhancing efficient convergence to the truth. In this essay, I show that, despite their differences, statistical and formal learning theory yield precisely the same result for a class of inductive problems that I call strongly VC ordered , of which Goodman’s riddle is just one example. (shrink)

This essay demonstrates a previously unnoticed connection between formal and statistical learning theory with regard to Nelson Goodman’s new riddle of induction. Discussions of Goodman’s riddle in formal learning theory explain how conjecturing “all green” before “all grue” can enhance efficient convergence to the truth, where efficiency is understood in terms of minimizing the maximum number of retractions or “mind changes.” Vapnik-Chervonenkis (VC) dimension is a central concept in statistical learning theory and (...) is similar to Popper’s notion of degrees of testability. I show that for a class inductive problems of which Goodman’s riddle is one example, a reliable inductive method minimizes the maximum number of mind changes exactly if it always conjectures the hypothesis from the set with lowest VC dimension consistent with the data. I also discuss the relevance of these results to language invariance and curve fitting. (shrink)

At its strongest, Hume's problem of induction denies the existence of any well justified assumptionless inductive inference rule. At the weakest, it challenges our ability to articulate and apply good inductive inference rules. This paper examines an analysis that is closer to the latter camp. It reviews one answer to this problem drawn from the VC theorem in statistical learning theory and argues for its inadequacy. In particular, I show that it cannot be computed, in general, whether (...) we are in a situation where the VC theorem can be applied for the purpose we want it to. (shrink)

In order to extract the regularities underlying a continuous sensory input, the individual elements constituting the stream have to be encoded and their transitional probabilities (TPs) should be learned. This suggests that variance in statistical learning (SL) performance reflects efficiency in encoding representations as well as efficiency in detecting their statistical properties. These processes have been taken to be independent and temporally modular, where first, elements in the stream are encoded into internal representations, and then the co‐occurrences (...) between them are computed and registered. Here, we entertain a novel hypothesis that one unifying construct—the rate of information in the sensory input—explains learning performance. This theoretical approach merges processes related to encoding of events and those related to learning their regularities into a single computational principle. We present data from two large‐scale experiments with over 800 participants tested in support for this hypothesis, showing that rate of information in a visual stream clearly predicts SL performance, and that similar rate of information values leads to similar SL performance. We discuss the implications for SL theory and its relation to regularity learning. (shrink)

From a theoretical perspective, most discussions of statistical learning have focused on the possible “statistical” properties that are the object of learning. Much less attention has been given to defining what “learning” is in the context of “statistical learning.” One major difficulty is that SL research has been monitoring participants’ performance in laboratory settings with a strikingly narrow set of tasks, where learning is typically assessed offline, through a set of two-alternative-forced-choice questions, (...) which follow a brief visual or auditory familiarization stream. Is that all there is to characterizing SL abilities? Here we adopt a novel perspective for investigating the processing of regularities in the visual modality. By tracking online performance in a self-paced SL paradigm, we focus on the trajectory of learning. In a set of three experiments we show that this paradigm provides a reliable and valid signature of SL performance, and it offers important insights for understanding how statistical regularities are perceived and assimilated in the visual modality. This demonstrates the promise of integrating different operational measures to our theory of SL. (shrink)

A key motivation for Universal Grammar (UG) is developmental: UG can help children acquire the linguistic knowledge that they do as quickly as they do from the data that's available to them. Some of the most fruitful recent work in language acquisition has combined ideas about different hypothesis space building blocks with domain‐general statistical learning. Statistical learning can then provide a way to help navigate the hypothesis space in order to converge on the correct hypothesis. Reinforcement (...) learning is a principled way to update the probability of a categorical option that's in competition with other categorical options. There are two prominent options for linking theory theoretical representations: the Uniformity of Theta Assignment Hypothesis (UTAH) and the relativized form of that theory, relativized UTAH. The learning strategy of Pearl and Sprouse shares the intuition with subjacency that there's a local structural anomaly when syntactic islands occur. (shrink)

In this article, we consider infants’ acquisition of foundational aspects of language and emotion through the lens of statistical learning. By taking a comparative developmental approach, we highlight ways in which the learning problems presented by input from these two rich communicative domains are both similar and different. Our goal is to encourage other scholars to consider multiple domains of human experience when developing theories in developmental cognitive science.

There are two main approaches to how statistical patterns are extracted from sequences: The transitional probability approach proposes that statistical learning occurs through the computation of probabilities between items in a sequence. The chunking approach, including models such as PARSER and TRACX, proposes that units are extracted as chunks. Importantly, the chunking approach suggests that the extraction of full units weakens the processing of subunits while the transitional probability approach suggests that both units and subunits should strengthen. (...) Previous findings using sequentially organized, auditory stimuli or spatially organized, visual stimuli support the chunking approach. However, one limitation of prior studies is that most assessed learning with the two-alternative forced-choice task. In contrast, this pre-registered experiment examined the two theoretical approaches in sequentially organized, visual stimuli using an online self-paced task—arguably providing a more sensitive index of learning as it occurs—and a secondary offline familiarity judgment task. During the self-paced task, abstract shapes were covertly organized into eight triplets (ABC) where one in every eight was altered (BCA) from the canonical structure in a way that disrupted the full unit while preserving a subunit (BC). Results from the offline familiarity judgment task revealed that the altered triplets were perceived as highly familiar, suggesting the learned representations were relatively flexible. More importantly, results from the online self-paced task demonstrated that processing for subunits, but not unit-initial stimuli, was impeded in the altered triplet. The pattern of results is in line with the chunking approach to statistical learning and, more specifically, the TRACX model. (shrink)

We propose a new model for forming and revising beliefs about unknown probabilities. To go beyond what is known with certainty and represent the agent’s beliefs about probability, we consider a plausibility map, associating to each possible distribution a plausibility ranking. Beliefs are defined as in Belief Revision Theory, in terms of truth in the most plausible worlds. We consider two forms of conditioning or belief update, corresponding to the acquisition of two types of information: learning observable evidence (...) obtained by repeated sampling from the unknown distribution; and learning higher-order information about the distribution. The first changes only the plausibility map, but leaves the given set of possible distributions essentially unchanged; the second rules out some distributions, thus shrinking the set of possibilities, without changing their plausibility ordering.. We look at stability of beliefs under either of these types of learning, defining two related notions, as well as a measure of the verisimilitude of a given plausibility model. We prove a number of convergence results, showing how our agent’s beliefs track the true probability after repeated sampling, and how she eventually gains in a sense knowledge of that true probability. Finally, we sketch the contours of a dynamic doxastic logic for statistical learning. (shrink)

The present paper shows how statistical learning theory and machine learning models can be used to enhance understanding of AI-related epistemological issues regarding inductive reasoning and reliability of generalisations. Towards this aim, the paper proceeds as follows. First, it expounds Price’s dual image of representation in terms of the notions of e-representations and i-representations that constitute subject naturalism. For Price, this is not a strictly anti-representationalist position but rather a dualist one (e- and i-representations). Second, the (...) paper links this debate with machine learning in terms of statistical learning theory becoming more viable epistemological tool when it abandons the perspective of object naturalism. The paper then argues that machine learning grounds a form of knowing that can be understood in terms of e- and i-representation learning. Third, this synthesis shows a way of analysing inductive reasoning in terms of reliability of generalisations stemming from a structure of e- and i-representations. In the age of Artificial Intelligence, connecting Price’s dual view of representation with Deep Learning provides an epistemological way forward and even perhaps an approach to how knowing is possible. (shrink)

One important problem in the philosophy of science is whether there can be a normative theory of discovery, as opposed to a normative theory of justification. Although the possibility of developing a logic of scientific discovery has been often doubted by philosophers, it is particularly interesting to consider how the basic insights of a normative theory of discovery have been turned into an effective research program in computer science, namely the research field of machine learning. In (...) this paper, I introduce some current research on statistical models to a philosophical audience. In particular, I will stress those features of statistical models that make them plausible computational counterparts of scientific theories. After noticing how these models allow for the main kinds of inference that are a trademark of scientific theories, I will focus on the problem of learning statistical models from data. The analysis will show how machine learning is casting new light on traditional problems in the philosophy of science. More precisely, I will explore the implications of some results in statistical learning with respect to the role of simplicity in theory choice and to the role of scalability (as a formal property of induction) in making scientific discovery effective. (shrink)

This paper shows how to define probability distributions over linguistically realistic syntactic structures in a way that permits us to define language learning and language comprehension as statistical problems. We demonstrate our approach using lexical‐functional grammar (LFG), but our approach generalizes to virtually any linguistic theory. Our probabilistic models are maximum entropy models. In this paper we concentrate on statistical inference procedures for learning the parameters that define these probability distributions. We point out some of (...) the practical problems that make straightforward ways of estimating these distributions infeasible, and develop a “pseudo‐likelihood” estimation procedure that overcomes some of these problems. This method raises interesting questions concerning the nature of the data available to a language learner and the modularity of language learning and processing. (shrink)

This study discusses the mediating role of statistics anxiety and motivation in the relationship comprising academic dishonesty, personality traits, and previous academic achievements in three different learning environments. Self-determination theory provides a broad psychological framework for these phenomena. Data were collected from 649 bachelor-degree students in the Social Sciences in five Israeli academic institutions. Structural equation modelling was employed to investigate the research variables’ relationships. Findings indicate that statistics anxiety mediates the relationship between personality traits and academic dishonesty (...) in the POE and the ERT learning environments. Findings also indicate the relationship between students’ achievements and academic dishonesty, but only in the ERT learning environment. In contrast, motivation mediates the relationship between students’ achievements and statistics anxiety only in the POE learning environment. This study unveils that learning environments determine the mediating role of statistical anxiety. In digital learning environments, mediation has been found between students’ personality traits and academic dishonesty. No similar parallel mediation could be established in the physical learning environment, F2F. Thus, we conclude that online courses should be designed according to student-centred approaches. (shrink)

We analyze naturally occurring datasets from student use of educational technologies to explore a long-standing question of the scope of transfer of learning. We contrast a faculty theory of broad transfer with a component theory of more constrained transfer. To test these theories, we develop statistical models of them. These models use latent variables to represent mental functions that are changed while learning to cause a reduction in error rates for new tasks. Strong versions of (...) these models provide a common explanation for the variance in task difficulty and transfer. Weak versions decouple difficulty and transfer explanations by describing task difficulty with parameters for each unique task. We evaluate these models in terms of both their prediction accuracy on held-out data and their power in explaining task difficulty and learning transfer. In comparisons across eight datasets, we find that the component models provide both better predictions and better explanations than the faculty models. Weak model variations tend to improve generalization across students, but hurt generalization across items and make a sacrifice to explanatory power. More generally, the approach could be used to identify malleable components of cognitive functions, such as spatial reasoning or executive functions. (shrink)

Rational Rules argues that moral learning can be understood in terms of general-purpose rational learning procedures. Nichols provides statistical learning accounts of some fundamental aspects of moral development, combining aspects of traditional empiricist and rationalist approaches.

Natural languages contain many layers of sequential structure, from the distribution of phonemes within words to the distribution of phrases within utterances. However, most research modeling language acquisition using artificial languages has focused on only one type of distributional structure at a time. In two experiments, we investigated adult learning of an artificial language that contains dependencies between both adjacent and non-adjacent words. We found that learners rapidly acquired both types of regularities and that the strength of the adjacent (...) statistics influenced learning of both adjacent and non-adjacent dependencies. Additionally, though accuracy was similar for both types of structure, participants’ knowledge of the deterministic non-adjacent dependencies was more explicit than their knowledge of the probabilistic adjacent dependencies. The results are discussed in the context of current theories of statistical learning and language acquisition. (shrink)

We investigate the general properties of general Bayesian learning, where “general Bayesian learning” means inferring a state from another that is regarded as evidence, and where the inference is conditionalizing the evidence using the conditional expectation determined by a reference probability measure representing the background subjective degrees of belief of a Bayesian Agent performing the inference. States are linear functionals that encode probability measures by assigning expectation values to random variables via integrating them with respect to the probability (...) measure. If a state can be learned from another this way, then it is said to be Bayes accessible from the evidence. It is shown that the Bayes accessibility relation is reflexive, antisymmetric and non-transitive. If every state is Bayes accessible from some other defined on the same set of random variables, then the set of states is called weakly Bayes connected. It is shown that the set of states is not weakly Bayes connected if the probability space is standard. The set of states is called weakly Bayes connectable if, given any state, the probability space can be extended in such a way that the given state becomes Bayes accessible from some other state in the extended space. It is shown that probability spaces are weakly Bayes connectable. Since conditioning using the theory of conditional expectations includes both Bayes’ rule and Jeffrey conditionalization as special cases, the results presented generalize substantially some results obtained earlier for Jeffrey conditionalization. (shrink)

Issues of reliability are claiming center-stage in the epistemology of machine learning. This paper unifies different branches in the literature and points to promising research directions, whilst also providing an accessible introduction to key concepts in statistics and machine learning – as far as they are concerned with reliability.

Interpreting a seemingly simple function word like “or,” “behind,” or “more” can require logical, numerical, and relational reasoning. How are such words learned by children? Prior acquisition theories have often relied on positing a foundation of innate knowledge. Yet recent neural‐network‐based visual question answering models apparently can learn to use function words as part of answering questions about complex visual scenes. In this paper, we study what these models learn about function words, in the hope of better understanding how the (...) meanings of these words can be learned by both models and children. We show that recurrent models trained on visually grounded language learn gradient semantics for function words requiring spatial and numerical reasoning. Furthermore, we find that these models can learn the meanings of logical connectives and and or without any prior knowledge of logical reasoning as well as early evidence that they are sensitive to alternative expressions when interpreting language. Finally, we show that word learning difficulty is dependent on the frequency of models' input. Our findings offer proof‐of‐concept evidence that it is possible to learn the nuanced interpretations of function words in a visually grounded context by using non‐symbolic general statistical learning algorithms, without any prior knowledge of linguistic meaning. (shrink)

View of future time orientation is a cognitive construct about future time. This view has its unique work of motivation and effect on academic performance. Previous studies have only explored the influence that future time orientation brings to the learning process at a single time, and most of them focus on cross-sectional studies. To further explore the cross-lagged relationship for freshmen between future time orientation and learning engagement during different periods, AMOS 23.0 was performed for cross-lagged analysis in (...) this study to explore the influence and effect among variables of different periods. This research was based on the theory of self-determination to discover the relationship between future time and learning involvement for freshmen in enrollment and the first summer vacation. In this research, there were 1,000 valid samples in the first stage and 840 valid samples in the second stage for the conduction of descriptive statistics, pair t-test, and cross-lagged analysis. The results show that: for learning engagement, freshmen at the end of the first year have a higher average score than at the beginning of the first year. View of students of future time orientation can affect their learning engagement of the future through self-determination of students. At last, we provide some suggestions as references for institutional research and future research. (shrink)

While theories of rationality and decision making typically adopt either a single-powertool perspective or a bag-of-tricks mentality, the research program of ecological rationality bridges these with a theoretically-driven account of when different heuristic decision mechanisms will work well. Here we described two ways to study how heuristics match their ecological setting: The bottom-up approach starts with psychologically plausible building blocks that are combined to create simple heuristics that fit specific environments. The top-down approach starts from the statistical problem facing (...) the organism and a set of principles, such as the bias– variance tradeoff, that can explain when and why heuristics work in uncertain environments, and then shows how effective heuristics can be built by biasing and simplifying more complex models. We conclude with challenges these approaches face in developing a psychologically realistic perspective on human rationality. (shrink)

Machine learning (ML) refers to a class of computer-facilitated methods of statistical modelling. ML modelling techniques are now being widely adopted across the sciences. A number of outspoken representatives from the general public, computer science, various scientific fields, and philosophy of science alike seem to share in the belief that ML will radically disrupt scientific practice or the variety of epistemic outputs science is capable of producing. Such a belief is held, at least in part, because its adherents (...) take ML to exist on novel epistemic footing relative to classical mathematical or statistical modelling approaches utilised in science. Namely, they take modelling with ML to be a “theory-free” enterprise, in the sense of not resting essentially on input from human conceptual grasp on the target phenomenon and domain expertise. I take this view to arise from the further, and more deeply entrenched belief that data is worldly and objective; i.e., data is viewed as recapitulating or representing with perfect fidelity the structure or properties of the systems in nature it is sampled from. Yet most contemporary philosophers of science take on board, in one version or other, the thesis of theory-ladenness or theory-mediation of observation or measurement. From this, it follows that most philosophers of science (and, I will venture, many scientists) hold irreconcilable views on the nature of data, an internal tension which threatens the integrity of their appraisals of ML in science. Taking the thesis of theory-ladenness on board---and its implications for the nature of data seriously---it follows that there is no reason to believe that ML differs fundamentally in its epistemic footing from established mathematical modelling approaches in the sciences. I show that usage and interpretation can differ from “standard practice” in scientific projects which wield the tools of ML without accepting a difference in the epistemic or representational status of such tools. (shrink)

Probability and inverse inference; Neyman-Pearson theory; Fisherian significance testing; The fiducial argument: one parameter; The fiducial argument: several parameters; Ian hacking's theory; Henry Kyburg's theory; Relevance and experimental design.

This paper deals with meta-statistical questions concerning frequentist statistics. In Sections 2 to 4 I analyse the dispute between Fisher and Neyman on the so called logic of statistical inference, a polemic that has been concomitant of the development of mathematical statistics. My conclusion is that, whenever mathematical statistics makes it possible to draw inferences, it only uses deductive reasoning. Therefore I reject Fisher's inductive approach to the statistical estimation theory and adhere to Neyman's deductive one. (...) On the other hand, I assert that Neyman-Pearson's testing theory, as well as Fisher's tests of significance, properly belong to decision theory, not to logic, neither deductive nor inductive. I then also disagree with Costantini's view of Fisher's testing model as a theory of hypothetico-deductive inferences.In Section 5 I disapprove Hacking1's evidentialists criticisms of the Neyman-Pearson's theory of statistics (NPT), as well as Hacking2's interpretation of NPT as a theory of probable inference. In both cases Hacking misses the point. I conclude, by claiming that Mayo's conception of the Neyman-Pearson's testing theory, as a model of learning from experience, does not purport any advantages over Neyman's behavioristic model. (shrink)

We propose a formal framework for interpretable machine learning. Combining elements from statistical learning, causal interventionism, and decision theory, we design an idealised explanation game in which players collaborate to find the best explanation for a given algorithmic prediction. Through an iterative procedure of questions and answers, the players establish a three-dimensional Pareto frontier that describes the optimal trade-offs between explanatory accuracy, simplicity, and relevance. Multiple rounds are played at different levels of abstraction, allowing the players (...) to explore overlapping causal patterns of variable granularity and scope. We characterise the conditions under which such a game is almost surely guaranteed to converge on a optimal explanation surface in polynomial time, and highlight obstacles that will tend to prevent the players from advancing beyond certain explanatory thresholds. The game serves a descriptive and a normative function, establishing a conceptual space in which to analyse and compare existing proposals, as well as design new and improved solutions. (shrink)

Selectional preferences have a long history in both generative and computational linguistics. However, since the publication of Resnik's dissertation in 1993, a new approach has surfaced in the computational linguistics community. This new line of research combines knowledge represented in a pre‐defined semantic class hierarchy with statistical tools including information theory, statistical modeling, and Bayesian inference. These tools are used to learn selectional preferences from examples in a corpus. Instead of simple sets of semantic classes, selectional preferences (...) are viewed as probability distributions over various entities. We survey research that extends Resnik's initial work, discuss the strengths and weaknesses of each approach, and show how they together form a cohesive line of research. (shrink)

According to classical arguments, language learning is both facilitated and constrained by cognitive biases. These biases are reflected in linguistic typology—the distribution of linguistic patterns across the world's languages—and can be probed with artificial grammar experiments on child and adult learners. Beginning with a widely successful approach to typology (Optimality Theory), and adapting techniques from computational approaches to statistical learning, we develop a Bayesian model of cognitive biases and show that it accounts for the detailed pattern (...) of results of artificial grammar experiments on noun-phrase word order (Culbertson, Smolensky, & Legendre, 2012). Our proposal has several novel properties that distinguish it from prior work in the domains of linguistic theory, computational cognitive science, and machine learning. This study illustrates how ideas from these domains can be synthesized into a model of language learning in which biases range in strength from hard (absolute) to soft (statistical), and in which language-specific and domain-general biases combine to account for data from the macro-level scale of typological distribution to the micro-level scale of learning by individuals. (shrink)