In this paper, building on these previous works, we propose to go deeper into the understanding of crowd behavior by proposing an approach which integrates ontologi- cal models of crowd behavior and dedicated computer vision algorithms, with the aim of recognizing some targeted complex events happening in the playground from the observation of the spectator crowd behavior. In order to do that, we first propose an ontology encoding available knowledge on spectator crowd behavior, built as a spe- cialization of (...) the DOLCE foundational ontology, which allows the representation of categories belonging both to the physical and to the social realms. We then propose a simplified and tractable version of such ontology in a new temporal extension of a description logic, which is used for temporally coupling events happening on the play- ground and spectator crowd behavior. At last, computer vision algorithms provide the input information concerning what is observed on the stands and ontological reasoning delivers the output necessary to perform complex event recognition. (shrink)

What is the most important reason for using Computer Vision methods in humanities research? In this article, I argue that the use of numerical representation and data analysis methods offers a new language for describing cultural artifacts, experiences and dynamics. The human languages such as English or Russian that developed rather recently in human evolution are not good at capturing analog properties of human sensorial and cultural experiences. These limitations become particularly worrying if we want to compare thousands, millions (...) or billions of artifacts—i.e. to study contemporary media and cultures at their new twenty-first century scale. When we instead use numerical measurements of image properties standard in Computer Vision, we can better capture details of a single artifact as well as visual differences between a number of artifacts–even if they are very small. The examples of visual dimensions that numbers can capture better then languages include color, shape, texture, contours, composition, and visual characteristics of represented faces, bodies and objects. The methods of finding structures and relationships in large numerical datasets developed in statistics and machine learning allow us to extend this analysis to very big datasets of cultural objects. Equally importantly, numerical image features used in Computer Vision also give us a new language to represent gradual and continuous temporal changes—something which natural languages are also bad at. This applies to both single artworks such as a film or a dance piece and also to changes in visual characteristics in millions of artifacts over decades or centuries. (shrink)

This article attempts to demystify computer vision for novice programmers through a survey of new applications in the arts, system design considerations, and contemporary tools. It introduces the concept and gives a brief history of computer vision within interactive art from Myron Kruger to the present. Basic techniques of computer vision such as detecting motion and object tracking are discussed in addition to various software applications created for exploring the topic. As an example, the results of a (...) one-week machine vision workshop are presented to show how designers are able to apply their skills toward creating novel uses of these technologies. The article concludes with a listing of code for basic computer vision techniques. (shrink)

Increasing computational power and improving deep learning methods have made computer vision technologies pervasively common in urban environments. Their applications in policing, traffic management, and documenting public spaces are increasingly common (Ridgeway 2018, Coifman et al. 1998, Sun et al. 2020). Despite the often-discussed biases in the algorithms' training and unequally borne benefits (Khosla et al. 2012), almost all applications similarly reduce urban experiences to simplistic, reductive, and mechanistic measures. There is a lack of context, depth, and specificity (...) in these practices that enables semantic knowledge or analysis within urban contexts, especially within the context of using and occupying urban space. This paper will critique existing uses of artificial intelligence and computer vision in urban practices to propose a new framework for understanding people, action, and public space. This paper revisits Geertz's (1973) use of thick descriptions in generating interpretive theories of culture and activity and uses this lens to establish a framework to approach evaluating the varied uses of computer vision technologies that weigh meaning. By discussing cases of implemented examples of urban computer vision—from LinkNYC and Numina's urban measurements to the Detroit Police's use of DataWorks Plus's facial recognition technology—it proposes a framework for evaluating the thickness of the algorithm's conclusions against the computational method's complexity required to produce that outcome. Further, we discuss how the framework's positioning may differ (and conflict) between different users of the technology, from engineer to urban planner and policymaker, to citizen. This paper also discusses how the current use and training of deep learning algorithms and how this process limits semantic learning and proposes three potential methodologies toward gaining a more contextually specific, urban-semantic, description of urban space relevant to urbanists. This paper contributes to the critical conversations regarding the proliferation of artificial intelligence by challenging the current applications of these technologies in the urban environment by highlighting their failures within this context while also proposing an evolution of these algorithms that may ultimately make them sensitive and useful within this spatial and cultural milieu. (shrink)

David Marr's theory of vision has been a rich source of inspiration, fascination and confusion. I will suggest that some of this confusion can be traced to discrepancies between the way Marr developed his theory in practice and the way he suggested such a theory ought to be developed in his explicit metatheoretical remarks. I will address claims that Marr's theory may be seen as an optimizing theory, along with the attendant suggestion that optimizing assumptions may be inappropriate for (...) cognitive mechanisms just as anti-adaptationists have argued they are inappropriate for other physiological mechanisms. I will discuss the nature of optimizing assumptions and theories. Considering various difficulties in identifying and assessing optimizing assumptions, I will suggest that Marr's theory is not purely an optimizing theory and that reaction to Marr on this issue prompts interesting considerations for the development of inter-disciplinary constraints in the cognitive and brain sciences. (shrink)

Imagine a world where machines can see and understand the world the way humans do. Rapid progress in artificial intelligence has led to smartphones that recognize faces, cars that detect pedestrians, and algorithms that suggest diagnoses from clinical images, among many other applications. The success of computer vision is founded on a deep understanding of the neural circuits in the brain responsible for visual processing. This book introduces the neuroscientific study of neuronal computations in visual cortex alongside of the (...) psychological understanding of visual cognition and the burgeoning field of biologically-inspired artificial intelligence. Topics include the neurophysiological investigation of visual cortex, visual illusions, visual disorders, deep convolutional neural networks, machine learning, and generative adversarial networks among others. It is an ideal resource for students and researchers looking to build bridges across different approaches to studying and developing visual systems. (shrink)

Computer vision is widely used in manufacturing, sports, medical diagnosis, and other fields. In this article, a multifeature fusion error action expression method based on silhouette and optical flow information is proposed to overcome the shortcomings in the effectiveness of a single error action expression method based on the fusion of features for human body error action recognition. We analyse and discuss the human error action recognition method based on the idea of template matching to analyse the key issues (...) that affect the overall expression of the error action sequences, and then, we propose a motion energy model based on the direct motion energy decomposition of the video clips of human error actions in the 3 Deron action sequence space through the filter group. The method can avoid preprocessing operations such as target localization and segmentation; then, we use MET features and combine with SVM to test the human body error database and compare the experimental results obtained by using different feature reduction and classification methods, and the results show that the method has the obvious comparative advantage in the recognition rate and is suitable for other dynamic scenes. (shrink)

For human interaction, it is important to understand what emotional state others are in. Especially the observation of faces aids us in putting behaviours into context and gives insight into emotions and mental states of others. Detecting whether someone is nervous, a form of state anxiety, is such an example as it reveals a person’s familiarity and contentment with the circumstances. With recent developments in computer vision we developed behavioural nervousness models to show which time-varying facial cues reveal whether (...) someone is nervous in an interview setting. The facial changes, reflecting a state of anxiety, led to more visual exposure and less chemosensory (taste and olfaction) exposure. However, experienced observers had difficulty picking up these changes and failed to detect nervousness levels accurately therewith. This study highlights humans’ limited capacity in determining complex emotional states but at the same time provides an automated model that can assist us in achieving fair assessments of so far unexplored emotional states. (shrink)

Focused on city-scale automation, and using self-driving cars (SDCs) as a case study, this article reflects on the role of AI—and in particular, computer vision systems used for mapping and navigation—as a catalyst for urban transformation. Urban research commonly presents AI and cities as having a one-way cause-and-effect relationship, giving undue weight to AI’s impact on cities and overlooking the role of cities in shaping AI. Working at the intersection of data science and social research, this paper aims to (...) counter this trend by exploring the reverse perspective: how do cities affect the development, and expose the present limits, of SDCs? The contribution of this paper is threefold. First, by comparing urban and nonurban environments and thoroughly examining the relationship between computer vision and city-specific sociality and form, it defines machine autonomy/automation as a function of the sociotechnical milieu in which an AI system operates. Second, and related, the paper problematizes the notion of SDCs as autonomous technologies and the role it plays in envisioning contending policy arrangements and technical solutions for achieving full driving automation. Finally, the article offers insight into a materialist and spatialized understanding of AI—namely, not as an abstract quality susceptible to replication within discrete machines, but rather as a distributed property emerging through embodied interactions among a multiplicity of agents (human, non-human, and technological) within/with their environments. (shrink)

Drawing on radical media archeology and assemblage theory, this article investigates assemblages of computer vision as they construct new spatiotemporal relations and new capacities for seeing and acting.

The project “Artificial and Artistic Vision. Computer Vision and Art History in Practical-Methodical Cooperation” is interdisciplinary by definition and also in its personnel composition and combines the humanities, engineering, and natural sciences. Together, prototypes and methodological approaches to an automatic vision that assists art history are being developed in the form of basic research.

During 2018, as part of a research project funded by the Deviant Practice Grant, artist Bruno Moreschi and digital media researcher Gabriel Pereira worked with the Van Abbemuseum collection (Eindhoven, NL), reading their artworks through commercial image-recognition (computer vision) artificial intelligences from leading tech companies. The main takeaways were: somewhat as expected, AI is constructed through a capitalist and product-focused reading of the world (values that are embedded in this sociotechnical system); and that this process of using AI is (...) an innovative way for doing institutional critique, as AI offers an untrained eye that reveals the inner workings of the art system through its glitches. This paper aims to regard these glitches as potentially revealing of the art system, and even poetic at times. We also look at them as a way of revealing the inherent fallibility of the commercial use of AI and machine learning to catalogue the world: it cannot comprehend other ways of knowing about the world, outside the logic of the algorithm. But, at the same time, due to their “glitchy” capacity to level and reimagine, these faulty readings can also serve as a new way of reading art; a new way for thinking critically about the art image in a moment when visual culture has changed form to hybrids of human–machine cognition and “machine-to-machine seeing”. (shrink)

Environmental landscaping is known to build, plan, and manage landscapes that consider the ecology of a site and produce gardens that benefit both people and the rest of the ecosystem. Landscaping and the environment are combined in landscape design planning to provide holistic answers to complex issues. Seeding native species and eradicating alien species are just a few ways humans influence the region’s ecosystem. Landscape architecture is the design of landscapes, urban areas, or gardens and their modification. It comprises the (...) construction of urban and rural landscapes via coordinating the creation and management of open spaces and economics, finding a job, and working within a confined project budget. There was a lot of discussion about global warming and water shortages. There is a lot of hope to be found even in the face of seemingly insurmountable obstacles. AI is becoming more significant in many urban landscape planning and design elements with the advent of web 4.0 and Human-Centred computing. It created a virtual reality-based landscape to create deep neural networks (DNNs) to make deep learning (DL) more user-friendly and efficient. Users may only manipulate physical items in this environment to manually construct neural networks. These setups are automatically converted into a model, and the real-time testing set is reported and aware of the DNN models that users are producing. This research presents a novel strategy for combining DL-DNN with landscape architecture, providing a long-term solution to the problem of environmental pollution. Carbon dioxide levels are constantly checked when green plants are in and around the house. Plants, on either hand, remove toxins from the air, making it easier to maintain a healthy environment. Human-centered Artificial Intelligence-based web 4.0 may be used to assess and evaluate the data model. The study findings can be sent back into the design process for further modification and optimization. (shrink)

Human motion gesture recognition is the most challenging research direction in the field of computer vision, and it is widely used in human-computer interaction, intelligent monitoring, virtual reality, human behaviour analysis, and other fields. This paper proposes a new type of deep convolutional generation confrontation network to recognize human motion pose. This method uses a deep convolutional stacked hourglass network to accurately extract the location of key joint points on the image. The generation and identification part of the network (...) is designed to encode the first hierarchy and the second hierarchy and show the spatial relationship of human body parts. The generator and the discriminator are designed as two parts in the network, and they are connected together in order to encode the possible relationship of appearance and, at the same time, the possibility of the existence of human body parts and the relationship between each part of the body and its parental part coding. In the image, the key nodes of the human body model and the general body posture can be identified more accurately. The method has been tested on different data sets. In most cases, the results obtained by the proposed method are better than those of other comparison methods. (shrink)

Conventional imagers and almost all vision processes use and rely on theories that are based on the principle of static image-frames. A frame is a 2D matrix that represents the spatial locations of intensities of a scene projected on the imager. The notion of a frame itself is so embedded in machine vision, that it is usually taken for granted that this is how biological systems store light information. This paper presents a biosinpired event-based image formation principle, which (...) output data rely on an asynchronous acquisition process. The generated information is stored in temporal volumes, which size and information depend only on the dynamic content of observed scenes. Practical analysis of such information will shows that the processing of visual information can only be based on a semiotic process. The paper also provides a general definition of the notion of visual features as the interpretation of signs according to different possible readings of the codified visual signal. (shrink)

Borrowing Timothy Morton’s notion of hyperobject, this article explores questions of network and scale in generative adversarial networks (GAN) images. In this context, the term network refers to the omnipresence of algorithmic images today and their significant impact on our lives. Such images are massively distributed in time and space beyond any sensible human-scale. Scale, in this context, denotes the relations between different operational layers of algorithmic images, such as the pictorial layer in contrast to the data layer. An algorithmic (...) image is simultaneously a visual image, a symbol, a data point and part of a mass visual milieu. Its meaning is thus polymorphic and can, arguably, never be exhausted. The article explores these terms through analysis of the website www.thispersondoesnotexist.com. (shrink)

One of the most lasting contributions of Marr's posthumous book is his articulation of the different “levels of analysis” that are needed to understand vision. Although a variety of work has examined how these different levels are related, there is comparatively little examination of the assumptions on which his proposed levels rest, or the plausibility of the approach Marr articulated given those assumptions. Marr placed particular significance on computational level theory, which specifies the “goal” of a computation, its (...) appropriateness for solving a particular problem, and the logic by which it can be carried out. The structure of computational level theory is inherently teleological: What the brain does is described in terms of its purpose. I argue that computational level theory, and the reverse-engineering approach it inspires, requires understanding the historical trajectory that gave rise to functional capacities that can be meaningfully attributed with some sense of purpose or goal, that is, a reconstruction of the fitness function on which natural selection acted in shaping our visual abilities. I argue that this reconstruction is required to distinguish abilities shaped by natural selection—“natural tasks” —from evolutionary “by-products”, rather than merely demonstrating that computational goals can be embedded in a Bayesian model that renders a particular behavior or process rational. (shrink)

Since the 1980s, several studies of visual perception have persuasively argued that important aspects of human vision are best accounted for not by recourse to inner mental representations but rather through socially observable actions and behaviors (e.g. Lynch 1985, Latour 1986, Lynch 1990, Goodwin 1994, Goodwin 1997, Sharrock & Coulter 1998). While there are clearly physiological mechanisms required for vision, psychological accounts of perception in terms of inner mental representations have been dislodged from their position as the basic (...) term in the interface between human beings and their environment and replaced with terms such as "social practice," and "vernacular intelligibility." The focus for these .. (shrink)

A great deal of philosophical work has addressed the question of whether Man’s computational theory of early vision is individualistic. Burge and Davies have argued that, according to Marr’s theory, visual states are individuated non-individualistically. Segal has denied that Marr’s theory has these non-individualistic implications. More recently, Shapiro has argued that the entire debate has been misguided. I argue that Shapiro is mistaken in a fairly deep way, attention to which allows us to raise and clarify several important (...) issues involved in discussions of individualism. Contrary to Burge and Davies, and by a route rather different from Segal’s, I defend the claim that Man’s theory offers no reason to think that visual states are individuated non-individualistically. (shrink)

The subject of this study is the problem of the failure of attempts by the scientific community to come to a common understanding of what exactly information can be as something encoded into material structures and moved along with them. At the same time, the following aspects of this problem are considered in detail: what is the immediate cause of the information problem; what are the objective and subjective prerequisites for its appearance; why the unresolved nature of this problem does (...) not interfere with the creation and development of communication systems, control and other "smart" devices; is there any general guideline for finding solutions to such problems, and what is it; what role did the philosophy of information play in turning the problem of information from a private scientific problem into a problem of the ideological level. The main results of the conducted research are as follows: the immediate cause of the information problem has been identified and a comprehensively substantiated solution to this problem has been given; it has been revealed that the incompleteness of the materialistic theory of cognition has become a fertile ground for the appearance of this problem; the circumstances that prevented its completion have been identified; the natural mechanism of control and cognition has been revealed, as a result of which the materialistic theory of cognition has been further developed the source of the ideal in the material world is revealed; it is shown that the mechanistic idea of information coding made the concept of information incompatible with the presence of any objective content; it is revealed what is actually hidden behind the words about information coding and its measurement; the nature of signals and signs is revealed; a kind of bias of the philosophy of information and a general error is revealed all known concepts of information. (shrink)

Even today, film restoration is a challenge, because it involves multidisciplinary competences: from analogue film inspection and conservation to digitisation and image enhancement. In this context, thanks to the high manageability of digital files, the film restoration workflow often follows a digitisation step, which presents many approximations and issues that are often ignored. In this work, we propose an alternative approach to the issues commonly encountered in film restoration aiming at restoring the original colour appearance, through models of human colour (...) perception. (shrink)

David Marr's theory of vision has been widely cited by philosophers and psychologists. I have three projects in this paper. First, I try to offer a perspicuous characterization of Marr's theory. Next, I consider the implications of Marr's work for some currently popular philosophies of psychology, specifically, the "hegemony of neurophysiology view", the theories of Jerry Fodor, Daniel Dennett, and Stephen Stich, and the view that perception is permeated by belief. In the last section, I consider what the phenomenon (...) of vision must be like for Marr's project to succeed. (shrink)

According to Marr's theory of vision, computational processes of early vision rely for their success on certain "natural constraints" in the physical environment. I examine the implications of this feature of Marr's theory for the question whether psychological states supervene on neural states. It is reasonable to hold that Marr's theory is nonindividualistic in that, given the role of natural constraints, distinct computational theories of the same neural processes may be justified in different environments. But to (...) avoid trivializing computational explanations, theories must respect methodological solipsism in the sense that within a theory there cannot be differences in content without a corresponding difference in neural states. (shrink)

Unsupervised statistical learning is the standard setting for the development of the only advanced visual system that is both highly sophisticated and versatile, and extensively studied: that of monkeys and humans. In this extended abstract, we invoke philosophical observations, computational arguments, behavioral data and neurobiological findings to explain why computer vision researchers should care about (1) unsupervised learning, (2) statistical inference, and (3) the visual brain. We then outline a neuromorphic approach to structural primitive learning motivated by these (...) considerations, survey a range of neurobiological findings and behavioral data consistent with it, and conclude by mentioning some of the more challenging directions for future research. (shrink)

Animate vision systems have gaze control mechanisms that can actively position the camera coordinate system in response to physical stimuli. Compared to passive systems, animate systems show that visual computation can be vastly less expensive when considered in the larger context of behavior. The most important visual behavior is the ability to control the direction of gaze. This allows the use of very low resolution imaging that has a high virtual resolution. Using such a system in a controlled way (...) provides additional constraints that dramatically simplify the computations of early vision. Another important behavior is the way the environment “behaves‘. Animate systems under real-time constraints can further reduce their computational burden by using environmental cues that are perspicuous in the local context. A third source of economy is introduced when behaviors are learned. Because errors are rarely fatal, systems using learning algorithms can amortize computational cost over extended periods. Further economies can be achieved when the learning system uses indexical reference, which is a form of dynamic variable binding. Animate vision is a natural way of implementing this dynamic binding. (shrink)

Although the study of visual perception has made more progress in the past 40 years than any other area of cognitive science, there remain major disagreements as to how closely vision is tied to general cognition. This paper sets out some of the arguments for both sides and defends the position that an important part of visual perception, which may be called early vision or just vision, is prohibited from accessing relevant expectations, knowledge and utilities - in (...) other words it is cognitively impenetrable. That part of vision is complex and articulated and provides a representation of the 3-D surfaces of objects sufficient to serve as an index into memory, with somewhat different outputs being made available to other systems such as those dealing with motor control. The paper also addresses certain conceptual and methodological issues, including the use of signal detection theory and event-related potentials to assess cognitive penetration of vision. A distinction is made among several stages in visual processing. These include, in addition to the inflexible early-vision stage, a pre-perceptual attention allocation stage and a post-perceptual evaluation, memory-accessing, and inference stage which provide several different highly constrained ways in which cognition can affect the outcome of visual perception. The paper discusses arguments that have been presented in both computer vision and psychology showing that vision is "intelligent" and involves elements of problem solving". It is suggested that these cases do not show cognitive penetration, but rather they show that certain natural constraints on interpretation, concerned primarily with optical and geometrical properties of the world, have been compiled into the visual system. The paper also examines a number of examples where instructions and "hints" are alleged to affect. (shrink)

Noam Chomsky and Frances Egan argue that David Marr.

Comics are complex multimodal documents that make for intriguing materials to analyze with computer vision and computational linguistics. This review summarizes the growing developments in computational modeling which have been progressing to analyze visual narratives across their various substructures.

Marr's computational theory of stereopsis is shown to imply that human vision employs a system of representation which has all the properties of a number system. Claims for an internal number system and for neural computation should be taken literally. I show how these ideas withstand various skeptical attacks, and analyze the requirements for describing neural operations as computations. Neural encoding of numerals is shown to be distinct from our ability to measure visual physiology. The constructs in Marr's (...) theory are neither propositional nor pictorial, and provide a counter example to many commonly held dichotomies concerning mental representation. (shrink)

What is the mind? How does it work? How does it influence behavior? Some psychologists hope to answer such questions in terms of concepts drawn from computer science and artificial intelligence. They test their theories by modeling mental processes in computers. This book shows how computer models are used to study many psychological phenomena--including vision, language, reasoning, and learning. It also shows that computer modeling involves differing theoretical approaches. Computational psychologists disagree about some basic questions. For instance, should (...) the mind be modeled by digital computers, or by parallel-processing systems more like brains? Do computer programs consist of meaningless patterns, or do they embody (and explain) genuine meaning? (shrink)

In the target article, Bowers et al. dispute deep artificial neural network (ANN) models as the currently leading models of human vision without producing alternatives. They eschew the use of public benchmarking platforms to compare vision models with the brain and behavior, and they advocate for a fragmented, phenomenon-specific modeling approach. These are unconstructive to scientific progress. We outline how the Brain-Score community is moving forward to add new model-to-human comparisons to its community-transparent suite of benchmarks.