There is a paradox in the standard model of cosmology. How can matter in the early universe have been in thermal equilibrium, indicating maximum entropy, but the initial state also have been low entropy (the “past hypothesis"), so as to underpin the second law of thermodynamics? The problem has been highly contested, with the only consensus being that gravity plays a role in the story, but with the exact mechanism undecided. In this paper, we construct a well-defined (...) mechanical model to study this paradox. We show how it reproduces the salient features of standard big-bang cosmology with surprising success, and we use it to produce novel results on the statistical mechanics of a gas in an expanding universe. We conclude with a discussion of potential uses of the model, including the explicit computation of the time-dependent coarse-grained entropies needed to investigate the past hypothesis. (shrink)

This paper shows how to define probability distributions over linguistically realistic syntactic structures in a way that permits us to define language learning and language comprehension as statistical problems. We demonstrate our approach using lexical‐functional grammar (LFG), but our approach generalizes to virtually any linguistic theory. Our probabilistic models are maximum entropy models. In this paper we concentrate on statistical inference procedures for learning the parameters that define these probability distributions. We point out some of the practical problems (...) that make straightforward ways of estimating these distributions infeasible, and develop a “pseudo‐likelihood” estimation procedure that overcomes some of these problems. This method raises interesting questions concerning the nature of the data available to a language learner and the modularity of language learning and processing. (shrink)

We investigate uncertain reasoning with quantified sentences of the predicate calculus treated as the limiting case of maximum entropy inference applied to finite domains.

Two open questions of inductive reasoning are solved: (1) does the principle of maximum entropy (pme) give a solution to the obverse Majerník problem; and (2) is Wagner correct when he claims that Jeffrey’s updating principle (jup) contradicts pme? Majerník shows that pme provides unique and plausible marginal probabilities, given conditional probabilities. The obverse problem posed here is whether pme also provides such conditional probabilities, given certain marginal probabilities. The theorem developed to solve the obverse Majerník problem demonstrates (...) that in the special case introduced by Wagner pme does not contradict jup, but elegantly generalizes it and offers a more integrated approach to probability updating. (shrink)

Through the analysis of facial feature extraction technology, this paper designs a lightweight convolutional neural network. The LW-CNN model adopts a separable convolution structure, which can propose more accurate features with fewer parameters and can extract 3D feature points of a human face. In order to enhance the accuracy of feature extraction, a face detection method based on the inverted triangle structure is used to detect the face frame of the images in the training set before the model extracts the (...) features. Aiming at the problem that the feature extraction algorithm based on the difference criterion cannot effectively extract the discriminative information, the Generalized Multiple Maximum Dispersion Difference Criterion and the corresponding feature extraction algorithm are proposed. The algorithm uses the difference criterion instead of the entropy criterion to avoid the “small sample” problem, and the use of QR decomposition can extract more effective discriminative features for facial recognition, while also reducing the computational complexity of feature extraction. Compared with traditional feature extraction methods, GMMSD avoids the problem of “small samples” and does not require preprocessing steps on the samples; it uses QR decomposition to extract features from the original samples and retains the distribution characteristics of the original samples. According to different change matrices, GMMSD can evolve into different feature extraction algorithms, which shows the generalized characteristics of GMMSD. Experiments show that GMMSD can effectively extract facial identification features and improve the accuracy of facial recognition. (shrink)

This paper is a sequel to an earlier result of the authors that in making inferences from certain probabilistic knowledge bases the maximum entropy inference process, ME, is the only inference process respecting “common sense.” This result was criticized on the grounds that the probabilistic knowledge bases considered are unnatural and that ignorance of dependence should not be identified with statistical independence. We argue against these criticisms and also against the more general criticism that ME is representation dependent. (...) In a final section, however, we provide a criticism of our own of ME, and of inference processes in general, namely that they fail to satisfy compactness. (shrink)

Jaynes's maximum entropy principle (MEP) is analyzed by considering in detail a recent controversy. Emphasis is placed on the inductive logical interpretation of “probability” and the concept of “total knowledge.” The relation of the MEP to relative frequencies is discussed, and a possible realm of its fruitful application is noted.

The principle of maximum entropy is a general method to assign values to probability distributions on the basis of partial information. This principle, introduced by Jaynes in 1957, forms an extension of the classical principle of insufficient reason. It has been further generalized, both in mathematical formulation and in intended scope, into the principle of maximum relative entropy or of minimum information. It has been claimed that these principles are singled out as unique methods of statistical (...) inference that agree with certain compelling consistency requirements. This paper reviews these consistency arguments and the surrounding controversy. It is shown that the uniqueness proofs are flawed, or rest on unreasonably strong assumptions. A more general class of inference rules, maximizing the so-called Re[acute ]nyi entropies, is exhibited which also fulfill the reasonable part of the consistency assumptions. (shrink)

This editorial explains the scope of the special issue and provides a thematic introduction to the contributed papers.

This paper concerns the question of how to draw inferences common sensically from uncertain knowledge. Since the early work of Shore and Johnson (1980), Paris and Vencovská (1990), and Csiszár (1989), it has been known that the Maximum Entropy Inference Process is the only inference process which obeys certain common sense principles of uncertain reasoning. In this paper we consider the present status of this result and argue that within the rather narrow context in which we work this (...) complete and consistent mode of uncertain reasoning is actually characterised by the observance of just a single common sense principle (or slogan). (shrink)

Sometimes we receive evidence in a form that standard conditioning (or Jeffrey conditioning) cannot accommodate. The principle of maximum entropy (MAXENT) provides a unique solution for the posterior probability distribution based on the intuition that the information gain consistent with assumptions and evidence should be minimal. Opponents of objective methods to determine these probabilities prominently cite van Fraassen’s Judy Benjamin case to undermine the generality of maxent. This article shows that an intuitive approach to Judy Benjamin’s case supports (...) maxent. This is surprising because based on independence assumptions the anticipated result is that it would support the opponents. It also demonstrates that opponents improperly apply independence assumptions to the problem. (shrink)

Objective Bayesian epistemology invokes three norms: the strengths of our beliefs should be probabilities, they should be calibrated to our evidence of physical probabilities, and they should otherwise equivocate sufficiently between the basic propositions that we can express. The three norms are sometimes explicated by appealing to the maximum entropy principle, which says that a belief function should be a probability function, from all those that are calibrated to evidence, that has maximum entropy. However, the three (...) norms of objective Bayesianism are usually justified in different ways. In this paper we show that the three norms can all be subsumed under a single justification in terms of minimising worst-case expected loss. This, in turn, is equivalent to maximising a generalised notion of entropy. We suggest that requiring language invariance, in addition to minimising worst-case expected loss, motivates maximisation of standard entropy as opposed to maximisation of other instances of generalised entropy. Our argument also provides a qualified justification for updating degrees of belief by Bayesian conditionalisation. However, conditional probabilities play a less central part in the objective Bayesian account than they do under the subjective view of Bayesianism, leading to a reduced role for Bayes’ Theorem. (shrink)

I begin with the definition of power, and find that it is finalistic inasmuch as work directs energy dissipation in the interests of some system. The maximum power principle of Lotka and Odum implies an optimal energy efficiency for any work; optima are also finalities. I advance a statement of the maximum entropy production principle, suggesting that most work of dissipative structures is carried out at rates entailing energy flows faster than those that would associate with (...) class='Hi'>maximum power. This is finalistic in the sense that the out-of-equilibrium universe, taken as an isolated system, entrains work in the interest of global thermodynamic equilibration. I posit an evolutionary scenario, with a development on Earth from abiotic times, when promoting convective energy flows could be viewed as the important function of dissipative structures, to biotic times when the preservation of living dissipative structures was added to the teleology. Dissipative structures are required by the equilibrating universe to enhance local energy gradient dissipation. (shrink)

The Maximum Entropy Production Principle (MEPP) stands out as an overarching principle that rules life phenomena in Nature. However, its explanatory power beyond heuristics remains controversial. On the one hand, the MEPP has been successfully applied principally to non-living systems far from thermodynamic equilibrium. On the other hand, the underlying assumptions to lay the MEPP’s theoretical foundations and range of applicability increase the possibilities of conflicting interpretations. More interestingly, from a metaphysical stance, the MEPP’s philosophical status is hotly (...) debated: does the MEPP passively translate physical information into macroscopic predictions or actively select the physical solution in multistable systems, granting the connection between scientific models and reality? This paper deals directly with this dilemma by discussing natural determination from three angles: (1) Heuristics help natural philosophers to build an ontology. (2) The MEPP’s ontological status may stem from its selection of new forms of causation beyond physicalism. (3) The MEPP’s ontology ultimately depends on the much-discussed question of the ontology of probabilities in an information-theoretic approach and the ontology of macrostates according to the Boltzmannian definition of entropy. (shrink)

Several authors have investigated the question of whether canonical logic-based accounts of belief revision, and especially the theory of AGM revision operators, are compatible with the dynamics of Bayesian conditioning. Here we show that Leitgeb's stability rule for acceptance, which has been offered as a possible solution to the Lottery paradox, allows to bridge AGM revision and Bayesian update: using the stability rule, we prove that AGM revision operators emerge from Bayesian conditioning by an application of the principle of (...) class='Hi'>maximum entropy. In situations of information loss, or whenever the agent relies on a qualitative description of her information state - such as a plausibility ranking over hypotheses, or a belief set - the dynamics of AGM belief revision are compatible with Bayesian conditioning; indeed, through the maximum entropy principle, conditioning naturally generated AGM revision operators. This mitigates an impossibility theorem of Lin and Kelly for tracking Bayesian conditioning with AGM revision, and suggests an approach to the compatibility problem that highlights the information loss incurred by acceptance rules in passing from probabilistic to qualitative representations of beliefs. (shrink)

The principle of insufficient reason assigns equal probabilities to each alternative of a random experiment whenever there is no reason to prefer one over the other. The maximum entropy principle generalizes PIR to the case where statistical information like expectations are given. It is known that both principles result in paradoxical probability updates for joint distributions of cause and effect. This is because constraints on the conditional P P\left result in changes of P P\left that assign higher probability (...) to those values of the cause that offer more options for the effect, suggesting “intentional behavior.” Earlier work therefore suggested sequentially maximizing entropy according to the causal order, but without further justification apart from plausibility on toy examples. We justify causal modifications of PIR and MaxEnt by separating constraints into restrictions for the cause and restrictions for the mechanism that generates the effect from the cause. We further sketch why causal PIR also entails “Information Geometric Causal Inference.” We briefly discuss problems of generalizing the causal version of MaxEnt to arbitrary causal DAGs. (shrink)

This paper proposes a feature extraction algorithm based on the maximum entropy phrase reordering model in statistical machine translation in language learning machines. The algorithm can extract more accurate phrase reordering information, especially the feature information of reversed phrases, which solves the problem of imbalance of feature data during maximum entropy training in the original algorithm, and improves the accuracy of phrase reordering in translation. In the experiment, they were combined with linguistic features such as parts (...) of speech, words, and syntactic features extracted by using the syntax analyzer, and the maximum entropy classifier was used to predict translation errors, and the experimental verification was performed on the Chinese-English translation data set and compared. The experimental results show that different word posterior probabilities have a significant impact on the classification error rate, and the combination of linguistic features based on the word posterior probability can significantly reduce the classification error rate and improve the translation error prediction performance. (shrink)

An important suggestion of objective Bayesians is that the maximum entropy principle can replace a principle which is known to get into paradoxical difficulties: the principle of indifference. No one has previously determined whether the maximum entropy principle is better able to solve Bertrand’s chord paradox than the principle of indifference. In this paper I show that it is not. Additionally, the course of the analysis brings to light a new paradox, a revenge paradox of the (...) chords, that is unique to the maximum entropy principle. (shrink)

The principle of maximum entropy is a method for assigning values to probability distributions on the basis of partial information. In usual formulations of this and related methods of inference one assumes that this partial information takes the form of a constraint on allowed probability distributions. In practical applications, however, the information consists of empirical data. A constraint rule is then employed to construct constraints on probability distributions out of these data. Usually one adopts the rule that equates (...) the expectation values of certain functions with their empirical averages. There are, however, various other ways in which one can construct constraints from empirical data, which makes the maximum entropy principle lead to very different probability assignments. This paper shows that an argument by Jaynes to justify the usual constraint rule is unsatisfactory and investigates several alternative choices. The choice of a constraint rule is also shown to be of crucial importance to the debate on the question whether there is a conflict between the methods of inference based on maximum entropy and Bayesian conditionalization. (shrink)

To identify relationships among entities in natural language texts, extraction of entity relationships technically provides a fundamental support for knowledge graph, intelligent information retrieval, and semantic analysis, promotes the construction of knowledge bases, and improves efficiency of searching and semantic analysis. Traditional methods of relationship extraction, either those proposed at the earlier times or those based on traditional machine learning and deep learning, have focused on keeping relationships and entities in their own silos: extracting relationships and entities are conducted in (...) steps before obtaining the mappings. To address this problem, a novel Chinese relationship extraction method is proposed in this paper. Firstly, the triple is treated as an entity relation chain and can identify the entity before the relationship and predict its corresponding relationship and the entity after the relationship. Secondly, the Joint Extraction of Entity Mentions and Relations model is based on the Bidirectional Long Short-Term Memory and Maximum Entropy Markov Model. Experimental results indicate that the proposed model can achieve a precision of 79.2% which is much higher than that of traditional models. (shrink)

Kristina Toutanova Christopher D. Manning Dept of Computer Science Depts of Computer Science and Linguistics Gates Bldg 4A, 353 Serra Mall Gates Bldg 4A, 353 Serra Mall Stanford, CA 94305–9040, USA Stanford, CA 94305–9040, USA [email protected] [email protected]..

In a previous paper, a statistical method of constructing quantum models of classical properties has been described. The present paper concludes the description by turning to classical mechanics. The quantum states that maximize entropy for given averages and variances of coordinates and momenta are called ME packets. They generalize the Gaussian wave packets. A non-trivial extension of the partition-function method of probability calculus to quantum mechanics is given. Non-commutativity of quantum variables limits its usefulness. Still, the general form of (...) the state operators of ME packets is obtained with its help. The diagonal representation of the operators is found. A general way of calculating averages that can replace the partition function method is described. Classical mechanics is reinterpreted as a statistical theory. Classical trajectories are replaced by classical ME packets. Quantum states approximate classical ones if the product of the coordinate and momentum variances is much larger than Planck constant. Thus, ME packets with large variances follow their classical counterparts better than Gaussian wave packets. (shrink)

We formulate an elementary statistical game which captures the essence of some fundamental quantum experiments such as photon polarization and spin measurement. We explore and compare the significance of the principle of maximum Shannon entropy and the principle of minimum Fisher information in solving such a game. The solution based on the principle of minimum Fisher information coincides with the solution based on an invariance principle, and provides an informational explanation of Malus' law for photon polarization. There is (...) no solution based on the principle of maximum Shannon entropy. The result demonstrates the merits of Fisher information, and the demerits of Shannon entropy, in treating some fundamental quantum problems. It also provides a quantitative example in support of a general philosophy: Nature intends to hide Fisher information, while obeying some simple rules. (shrink)

Entropy is one of the physical bases for the fractal dimension definition, and the generalized fractal dimension was defined by Renyi entropy. Using the fractal dimension, we can describe urban growth and form and characterize spatial complexity. A number of fractal models and measurements have been proposed for urban studies. However, the precondition for fractal dimension application is to find scaling relations in cities. In the absence of the scaling property, we can make use of the entropy (...) function and measurements. This paper is devoted to researching how to describe urban growth by using spatial entropy. By analogy with fractal dimension growth models of cities, a pair of entropy increase models can be derived, and a set of entropy-based measurements can be constructed to describe urban growing process and patterns. First, logistic function and Boltzmann equation are utilized to model the entropy increase curves of urban growth. Second, a series of indexes based on spatial entropy are used to characterize urban form. Furthermore, multifractal dimension spectra are generalized to spatial entropy spectra. Conclusions are drawn as follows. Entropy and fractal dimension have both intersection and different spheres of application to urban research. Thus, for a given spatial measurement scale, fractal dimension can often be replaced by spatial entropy for simplicity. The models and measurements presented in this work are significant for integrating entropy and fractal dimension into the same framework of urban spatial analysis and understanding spatial complexity of cities. (shrink)

The application of the maximum entropy principle to determine probabilities on finite domains is well-understood. Its application to infinite domains still lacks a well-studied comprehensive approach. There are two different strategies for applying the maximum entropy principle on first-order predicate languages: applying it to finite sublanguages and taking a limit; comparing finite entropies of probability functions defined on the language as a whole. The entropy-limit conjecture roughly says that these two strategies result in the same (...) probabilities. While the conjecture is known to hold for monadic languages as well as for premiss sentences containing only existential or only universal quantifiers, its status for premiss sentences of greater quantifier complexity is, in general, unknown. I here show that the first approach fails to provide a sensible answer for some \-premiss sentences. I discuss implications of this failure for the first strategy and consequences for the entropy-limit conjecture. (shrink)

The application of the maximum entropy principle to determine probabilities on finite domains is well-understood. Its application to infinite domains still lacks a well-studied comprehensive approach. There are two different strategies for applying the maximum entropy principle on first-order predicate languages: applying it to finite sublanguages and taking a limit; comparing finite entropies of probability functions defined on the language as a whole. The entropy-limit conjecture roughly says that these two strategies result in the same (...) probabilities. While the conjecture is known to hold for monadic languages as well as for premiss sentences containing only existential or only universal quantifiers, its status for premiss sentences of greater quantifier complexity is, in general, unknown. I here show that the first approach fails to provide a sensible answer for some \-premiss sentences. I discuss implications of this failure for the first strategy and consequences for the entropy-limit conjecture. (shrink)

One well-known objection to the principle of maximum entropy is the so-called Judy Benjamin problem, first introduced by van Fraassen. The problem turns on the apparently puzzling fact that, on the basis of information relating an event’s conditional probability, the maximum entropy distribution will almost always assign to the event conditionalized on a probability strictly less than that assigned to it by the uniform distribution. In this article, I present an analysis of the Judy Benjamin problem (...) that can help to make sense of this seemingly odd feature of maximum entropy inference. My analysis is based on the claim that, in applying the principle of maximum entropy, Judy Benjamin is not acting out of a concern to maximize uncertainty in the face of new evidence, but is rather exercising a certain brand of epistemic charity towards her informant. This epistemic charity takes the form of an assumption on the part of Judy Benjamin that her informant’s evidential report leaves out no relevant information. Such a reconceptualization of the motives underlying Judy Benjamin’s appeal to the principle of maximum entropy can help to further our understanding of the true epistemological grounds of this principle and correct a common misapprehension regarding its relationship to the principle of insufficient reason. 1Introduction2The Principle of Maximum Entropy3An Apologia for Judy Benjamin4Conclusion: Entropy and Insufficient Reason. (shrink)

The maximum entropy principle is widely used to determine non-committal probabilities on a finite domain, subject to a set of constraints, but its application to continuous domains is notoriously problematic. This paper concerns an intermediate case, where the domain is a first-order predicate language. Two strategies have been put forward for applying the maximum entropy principle on such a domain: applying it to finite sublanguages and taking the pointwise limit of the resulting probabilities as the size (...) n of the sublanguage increases; selecting a probability function on the language as a whole whose entropy on finite sublanguages of size n is not dominated by that of any other probability function for sufficiently large n. The entropy-limit conjecture says that, where these two approaches yield determinate probabilities, the two methods yield the same probabilities. If this conjecture is found to be true, it would provide a boost to the project of seeking a single canonical inductive logic—a project which faltered when Carnap's attempts in this direction succeeded only in determining a continuum of inductive methods. The truth of the conjecture would also boost the project of providing a canonical characterisation of normal or default models of first-order theories. Hitherto, the entropy-limit conjecture has been verified for languages which contain only unary predicate symbols and also for the case in which the constraints can be captured by a categorical statement of quantifier complexity. This paper shows that the entropy-limit conjecture also holds for categorical statements of complexity, for various non-categorical constraints, and in certain other general situations. (shrink)

It is shown that entropy increase in thermodynamic systems can plausibly be accounted for by the random action of vacuum radiation. A recent calculation by Rueda using stochastic electrodynamics (SED) shows that vacuum radiation causes a particle to undergo a rapid Brownian motion about its average dynamical trajectory. It is shown that the magnitude of spatial drift calculated by Rueda can also be predicted by assuming that the average magnitudes of random shifts in position and momentum of a particle (...) correspond to the lower limits of the uncertainty relation. The latter analysis yields a plausible expression for the shift in momentum caused by vacuum radiation. It is shown that when the latter shift in momentum is magnified in particle interactions, the fractional change in each momentum component is on the order of unity within a few collision times, for gases and (plausibly) for denser systems over a very broad range of physical conditions. So any system of particles in this broad range of conditions would move to maximum entropy, subject to its thermodynamic constraints, within a few collision times. It is shown that the spatial drift caused by vacuum radiation, as predicted by the above SED calculation, can be macroscopic in some circumstances, and an experimental test of this effect is proposed. Consistency of the above results with quantum mechanics is discussed, and it is shown that the diffusion constant associated with the above Brownian drift is the same as that used in stochastic interpretations of the Schrödinger equation. (shrink)

It is a fundamental cornerstone of thermodynamics that entropy (SU,V\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{U,V}$$\end{document}) increases in spontaneous processes in isolated systems (often called closed or thermally closed systems when the transfer of energy as work is considered to be negligible) and achieves a maximum when the system reaches equilibrium. But with a different sign convention entropy could just as well be said to decrease to a minimum in spontaneous constant U, V processes. (...) It would then change in the same direction as the thermodynamic potentials in spontaneous processes. This article discusses but does not advocate such a change. (shrink)

Michael Weisberg’s account of scientific models concentrates on the ways in which models are similar to their targets. He intends not merely to explain what similarity consists in, but also to capture similarity judgments made by scientists. In order to scrutinize whether his account fulfills this goal, I outline one common way in which scientists judge whether a model is similar enough to its target, namely maximum likelihood estimation method. Then I consider whether Weisberg’s account could capture the judgments (...) involved in this practice. I argue that his account fails for three reasons. First, his account is simply too abstract to capture what is going on in MLE. Second, it implies an atomistic conception of similarity, while MLE operates in a holistic manner. Third, Weisberg’s atomistic conception of similarity can be traced back to a problematic set-theoretic approach to the structure of models. Finally, I tentatively suggest how these problems might be solved by a holistic approach in which models and targets are compared in a non-set-theoretic fashion. (shrink)

Computational modeling is now ubiquitous in psychology, and researchers who are not modelers may find it increasingly difficult to follow the theoretical developments in their field. This book presents an integrated framework for the development and application of models in psychology and related disciplines. Researchers and students are given the knowledge and tools to interpret models published in their area, as well as to develop, fit, and test their own models. Both the development of models and key features of (...) any model are covered, as are the applications of models in a variety of domains across the behavioural sciences. A number of chapters are devoted to fitting models using maximum likelihood and Bayesian estimation, including fitting hierarchical and mixture models. Model comparison is described as a core philosophy of scientific inference, and the use of models to understand theories and advance scientific discourse is explained. (shrink)

It has been shown by several authors that in operations involving information a quantity appears which is the negative of the quantity usually defined as entropy in similar situations. This quantity ℜ = − KI has been termed “negentropy” and it has been shown that the negentropy of information and the physical entropy S are mirrorlike representations of the same train of events. In physical terminology the energy is degraded by an increase in entropy due to an (...) increased randomness in the positions or velocities of components, wave functions, complexions in phase space; in informational terminology some information about the same components has been lost or the negentropy has been decreased. In equilibrium the system has for a given energy content maximum randomness. One consequence of this dual aspect was the idea to apply the methods of statistical mechanics to problems of communication and Brillouin showed that Fermi-Dirac statistics or generalized Fermi statistics are applicable for example to a transmission of signals such as telegrams. (shrink)

In the previous papers, it was demonstrated that applying the principle of maximum information entropy by maximizing the conditional information entropy, subject to the constraint given by the Liouville equation averaged over the phase space, leads to a definition of the rate of entropy change for closed Hamiltonian systems without any additional assumptions. Here, we generalize this basic model and, with the introduction of the additional constraints which are equivalent to the hydrodynamic continuity equations, show that (...) the results obtained are consistent with the known results from the nonequilibrium statistical mechanics and thermodynamics of irreversible processes. In this way, as a part of the approach developed in this paper, the rate of entropy change and entropy production density for the classical Hamiltonian fluid are obtained. The results obtained suggest the general applicability of the foundational principles of predictive statistical mechanics and their importance for the theory of irreversibility. (shrink)