On the Conception and Design of the PhilPapers Survey

by David Bourget and David Chalmers

Introduction

The PhilPapers Survey was a survey of professional philosophers and others on their philosophical views. (It was originally called "The Philosophical Survey", but we have retrospectively retitled it for reasons given below.) What follows are some thoughts on the conception and design of the survey, including responses to some feedback regarding the survey. We will discuss the results separately.

Why a survey of philosophers' philosophical views? We decided to do this in part because like many philosophers, we have an interest in the sociology of philosophy, and we were interested to see some hard data about this sociology. We are also interested in the experimental use of online tools as a method of philosophical communication. Using the PhilPapers technology to execute a survey of philosophical views plays into both of these interests.

One motivation for the Survey (and the Metasurvey) comes from the idea that philosophers may often have false sociological beliefs about the profession. Most of us have had the experience of reading philosophical papers that make sweeping sociological claims about the field that seem quite questionable. It is arguable that the "received wisdom" in a field (roughly, what people in a field take as a default view, one that can sometimes be presupposed) is not based on what most people in the field think, but rather is based on what most people think most people think. If the received wisdom is grounded in false sociological beliefs, that is worth knowing. It occurred to us that it would not be all that difficult to actually gather data about these matters, and that doing so might be an interesting and informative exercise.

We designed the Philosophical Survey at first through conversations with a number of other philosophers, and then refined it through three rounds of beta testing by many professional philosophers and graduate students. Of course we had to make many somewhat arbitrary choices along the way, in settling on a survey format, a set of questions, and so on.

Now that the Survey is completed, we have had the benefit of feedback by around 600 professional philosophers and others, across a wide variety of fields. (We encouraged respondents to leave feedback, and around 20% of them did so.) The feedback was roughly evenly divided between positive, negative, and neutral. Many philosophers liked the idea of a survey, with some saying they had been hoping for something like this for years, and many liked the design and the execution. At the same time, many thought that there is something odd about the whole idea of the survey, and many had criticisms of the design. In what follows we will offer some thoughts mainly on these critical remarks.

It should be noted up front that the Survey was never intended to be a definitive survey of all areas and traditions of philosophy, giving a definitive portrait of the views of the philosophical profession. Rather, we intended it as an initial study of a few aspects of the sociology of the profession, one that is likely to be refined by later work, and one that (like most surveys) is biased in various ways toward the interests of its designers. In retrospect, calling the Survey "The Philosophical Survey" may have encouraged some people to read pretensions of definitiveness into the survey. To help avoid that reading, we should have called it "The PhilPapers Survey" (or perhaps "A Philosophical Survey"). We will use that label from now on.


Criticisms

Critical remarks fell mainly into four categories, which we might summarize as follows: (i) The whole idea of a philosophical survey is unphilosophical, or philosophically irrelevant, or misrepresents philosophy, or is plain silly; (ii) The orientation of the survey is biased in various respects, and in particular neglects non-analytic traditions; (iii) The format of the questions and answers is problematic, and in particular the answer options are too brief; (iv) Various specific questions are problematic in various specific respects. The most common criticisms were of type (iv), followed by types (ii) and (iii) roughly equally, followed by type (i). We also received numerous emails from nonrespondents making comments of type (i), giving reason to think that this attitude is well-represented in the overall population. In what follows we offer some thoughts on criticisms of each type.

1. The idea of a philosophical survey is unphilosophical, or irrelevant to philosophy, or misrepresents philosophy, or is plain silly.

In response: The survey concerns philosophy, but it is not philosophy. If anything, it is an exercise in the sociology of philosophy. Sociology should not be confused with philosophy, but many philosophers have an interest in the sociology of philosophy all the same.

As for relevance: Even if the survey were wholly irrelevant to the practice of philosophy, we think the results would be interesting to many philosophers. And certainly, survey votes are likely to be a poor guide to philosophical truth. Still, beliefs about the sociology of philosophy are often given a role in the practice of philosophy, in various ways.

One notable role of sociological beliefs is in determining which views one can presuppose, attend to, or ignore. We do not especially condone this practice, but we think that insofar as sociological beliefs play this role, it is better for them to be accurate. For example: suppose that a philosopher accepts the analytic-synthetic distinction and thinks the arguments against it fail. Suppose that she is writing a paper in which she thinks that (sociology aside) an appeal to the distinction would strengthen the paper. Suppose that she nevertheless does not appeal to the distinction in a paper, solely on the grounds that she thinks a large majority of philosophers reject the distinction. Suppose that in fact, a large majority of philosophers accept the distinction. Then her decision will have been grounded in a false sociological belief, and the paper will be weaker by her own lights as a result. True sociological beliefs would put her in a position to write a better paper by her own lights.

The sociology of philosophical belief is also relevant to philosophy in other ways. For example, in arguing for a philosophical position, philosophers often make claims about the psychology of philosophical beliefs (for example, arguing that people only accept p because they accept q), and survey data may in principle be relevant to assessing such psychological claims. Finally, the history of philosophy is often directly concerned with sociological matters, and data about today's sociology may well be relevant to tomorrow's history.

As for misrepresentation: We certainly agree that a philosopher's views cannot be captured in a set of survey answers. The survey answers should not be seen as any sort of definitive representation of philosophers' views. At best, they capture a few dimensions or aspects of those views. Likewise, the survey should not be taken as any sort of definitive representation of the state of the philosophical profession. Like any survey, it just captures a few limited aspects of the state of the profession.

Finally, we recognize that there is something comical about the idea of doing philosophy by multiple choice. But we think that the results are of significant interest all the same, and multiple choice is by far the most feasible way to get the relevant data.

2. The survey is biased toward certain traditions and areas. In particular, it is biased toward analytic philosophy as opposed to non-analytic traditions, including continental traditions, non-Western traditions, and others.

In response: Yes, it is. We recognized this in the survey's information page. We considered incorporating questions drawn from non-analytic traditions, but it proved difficult to find questions that would be accessible enough to a predominantly analytic audience. So we decided to explicitly orient the survey toward analytic philosophy, which is the tradition within which our own expertise lies.

A number of respondents suggested that we should have incorporated questions drawn from continental traditions, Asian traditions, the pragmatist tradition, and from the history of philosophy, among others. There were relatively few concrete questions suggested here. We appreciated the concrete suggestions that were made (e.g. "Madness: historical or ahistorical", "Hegelian dialectic or Marxist dialectic", "Early and late Wittgenstein: complementary or contradictory?"), but they tended to reinforce our initial sense that such questions would not have worked in context.

It's also worth noting that we kept in a few questions that some analytic philosophers regarded as problematic -- e.g. the empiricism/rationalism question and the question about the nonliving philosophers with whom one most identified -- in part because we thought that questions of these sort would be accessible and engaging to philosophers from a variety of traditions, and that they would acknowledge the history of philosophy in a way that is otherwise hard to acknowledge in a survey like this.

Still, it is clear that the survey was somewhat alienating for some philosophers from non-analytic traditions. We could have done more to note the orientation explicitly up front. We may have other surveys in the future and input from philosophers from all traditions would be welcomed. We also encourage other philosophers oriented toward other traditions to design surveys of their own.

A few respondents suggested that the survey is biased toward certain areas within analytic philosophy. Comments included "Too much metaphysics and epistemology", "Too much ethics", "Too much epistemology and mind, not enough metaphysics", "Not enough aesthetics", "The survey is just analytic philosophy of mind", and so on.

As we said in the information page, we deliberately focused on the "core" areas of metaphysics, epistemology, mind, language, and ethics, because these are the most accessible to philosophers outside the area, though we did include questions in numerous other areas (aesthetics, decision theory, logic, metaphilosophy, philosophy of action, philosophy of science, political philosophy) as well. There ended up being a bit more epistemology, ethics, and mind (around five questions each) than language (three) or metaphysics (three to six, depending on whether one counts free will and personal identity), in part because the most natural further questions in the latter areas tended to be specialized.

There is also a slight bias toward philosophy of mind arising from the fact that the survey designers are both philosophers of mind. In particular, we allowed ourselves one "pet question" each that would not have made it onto the survey otherwise (too specialized). These were the questions about perceptual experience (Bourget) and zombies (Chalmers). That bias might be considered survey-designers' prerogative.

3. The answer options are too brief and should have been further explained.

In response: Again, this is a decision that we made explicitly up front. The trouble is that any longer explanation would inevitably be tendentious and would involve arbitrary choices. We tried designing some fuller questions, but it becomes a very difficult process and it is hard not to bias the questions in various ways. Furthermore, the "clarified" questions always call for further clarification in turn. In addition, survey results would inevitably be reported and discussed under simple labels (n% of respondents favored physicalism, atheism, etc), so we thought it made sense to have the answer options match those labels.

Of course there are downsides to this decision: especially ambiguities in various questions, and unfamiliarity with what various labels mean. We provided "other" options to help to handle these downsides. And we think that the survey results are clearly informative despite these problems. For future surveys, though, we will consider trying a different sort of question format, involving more fleshed-out questions.

The brevity and ambiguity of the labels means that caution must be used in interpreting the results. Of course one cannot simply assume that all respondents adopted the interpretation of "naturalism" or "empiricism" that one favors. But one can reasonably report that n% of the respondents reported accepting naturalism, or leaning toward empiricism, as long as one does not put forward a specific and contested interpretation of "naturalism" or "empiricism" in doing so.

4. The specific questions are problematic in respect X.

In response: The choice of questions is an imperfect art. We sought widely-known questions within analytic philosophy that have a relatively canonical set of associated views, each of which is associated with a reasonably simple label. We also aimed for questions that would be accessible to analytic philosophers in a wide range of areas. We came up with an initial list of twenty questions, and went through three rounds of beta testing in which about forty questions were trialed, before settling on the final list of thirty.

Of course no question is perfect, and some are more problematic than others. For most topics, the labels for views are at least somewhat ambiguous. For some topics, it is hard to find a short canonical list of views. For some of the topics, the list of views is far from exhaustive. For these reasons we provided "other" options (as well as the comment box) allowing respondents to indicate that they find the questions problematic in various respects, allowing them to endorse other alternatives, and so on.


Thoughts on specific questions

Some thoughts on some of the questions and on why we chose them follow.

Aesthetic value: objective or subjective?

We wanted to include a question on aesthetics, but it wasn't clear what to include. We consulted with a number of philosophers working in the area, and there was a consensus that a question along these lines would be best. Of course "objective" and "subjective" can be given various interpretations, but experts preferred this formulation to formulations in terms of aesthetic realism or relativism.

External world: idealism, skepticism, or non-skeptical realism?

We asked this one partly because of its centrality in the history of philosophy, and partly because we were especially interested in data about how many philosophers accept the "old, dead" positions that supposedly no-one accepts these days. Skepticism and idealism are often treated as gateways to reductio in contemporary discussion, for example, rather than as serious contenders for the truth. We would have liked to have an option for a view on which the external world is somehow mind-dependent without this being idealism (e.g. social constructivism), but we couldn't find a good accessible generic term here. Of course we expected a big majority for non-skeptical realism, but we were interested to see whether there would be a good number of skeptics and idealists out there.

Free will: compatibilism, libertarianism, or no free will?

These are roughly the standard options. We used "no free will" on the grounds that it is clearer for nonspecialists than "hard incompatibilism" and "free will skepticism", and "hard determinism" is overly narrow in a potentially nondeterministic world. Of course the three options aren't exclusive, but the answer options allow for that. We thought about asking two separate questions, one on compatibilism and one on whether there is free will, but the limited extra information that would result didn't seem worth the extra question.

Knowledge: empiricism or rationalism?

There is arguably no consensus interpretation of what these labels mean. "Rationalism" clearly isn't universally understood as the view that there is a priori knowledge, given the results of the two questions. Some take rationalism to be the claim that there is a priori knowledge of synthetic or substantial or contingent truths, but these interpretations aren't universal either. It is perhaps best to think of the question as capturing something about identification with two broad and ill-defined philosophical "camps", divided roughly over the role of a priori knowledge.

Logic: classical or non-classical?

Various respondents wondered how to interpret this questions, given that it's not obvious that there has to be a fact of the matter about whether classical or nonclassical logic is correct. Still, plenty of people think there is a fact of the matter, and those that don't (including ourselves) seem to have by and large chosen an "other" option.

Metaphilosophy: naturalism or non-naturalism?

As with empiricism and rationalism, there isn't a single standard interpretation of "naturalism." It has metaphysical readings (roughly involving rejection of non-natural entities) and epistemological readings (roughly involving the role of science in philosophical and other knowledge). The "metaphilosophy" tag tends to bias the interpretation toward the epistemological reading, but probably not universally. But again, "naturalism" seems most often used to label a broad camp or orientation, rather than a specific thesis, and data about identification with these camps is interesting.

Moral judgment: cognitivism or non-cognitivism?

This dichotomy is canonical but nevertheless somewhat problematic, given the recent prevalence of views that take moral judgments and sentences to be non-cognitive at root while still taking moral judgments and sentences to be truth-apt. Perhaps "expressivist" would have been slightly better than "non-cognitivist" here. Someone also suggested making the tag "Moral judgment (at base)".

Moral motivation: internalism or externalism?

Of course there are many meta-ethical internalisms, but the tag was supposed to focus attention on the connection between moral judgment and motivation. Even so there are various internalisms here, e.g. depending on whether the issue is whether moral judgments are always motivating, or whether they are always motivating for rational agents.

Normative ethics: deontology, consequentialism, or virtue ethics?

Yes, there are other options, as well as combined views. But clearly these are the canonical "big three".

Perceptual experience: disjunctivism, qualia theory, representationalism, or sense-datum theory?

Likewise, these are the canonical "big four" views in contemporary discussions. "Qualia theory" is perhaps ambiguous for nonspecialists, though is reasonably well-understood in the field to refer to a view on which experience consists at least in part in instantiating nonrelational nonintentional properties (the most common forms of adverbialism can be seen as qualia theories).

Personal identity: biological view, psychological view, or further-fact view?

It's not straightforward getting the names right here, especially for the first option. We considered various combinations of "brain view", "physical view", and "animal view". But after consulting with experts "biological view" seemed best. Of course the options here aren't exhaustive and may not be exclusive either.

Politics: communitarianism, egalitarianism, or libertarianism?

This was the hardest question to design. We wanted a question on political philosophy, but it is very unclear what that question should be. Perhaps the most central view in the field is liberalism, but after trialing questions involving this term, it became clear that one can't use it as it is ambiguous between a US political meaning, a European political meaning, a contemporary philosophical meaning, and more, sometimes labeling views that are diametrically opposed. There's also the question of what it should be opposed to. In much popular discourse liberalism is opposed to conservatism, but this doesn't straightforwardly reflect philosophical discourse. Left-wing vs. right-wing isn't really philosophy per se, either. A question about the relative importance of freedom and liberty seemed natural, but it's hard to phrase this question well. We tried "Egalitarianism vs libertarianism", but found relatively few taking the latter option (perhaps because it's often taken to stand for a relatively extreme view). A couple of people suggested including "communitarianism" too (thereby covering liberty, equality, fraternity!), and it polled well, so we decided that this was a good balance. The list is obviously imperfect and there are obviously other views that could have been included. But as far as we can tell, there is no really good canonical set of options here.

Proper names: Fregean vs Millian?

Obviously not exclusive, but still canonical. We tried "descriptivism vs direct reference", but that is even less exclusive, due to all the non-descriptive Fregeans. We also tried "Language: Fregean vs Russellian", but complications about Russell's own descriptivist views make this tricky. So this seemed best in the end.

Time: A-theory or B-theory?

Along with Newcomb's problem, this got the highest quota of "Insufficiently familiar with the issue" answers. We thought about fleshing out the labels -- e.g. "A-theory (passage is real) vs B-theory (block universe)" -- but any such labels are problematic. Still, in retrospect, we probably should have tried something like this to increase the response.

Trolley problem (five straight ahead, one on side track, turn requires switching, what ought one do?): switch or don't switch?

We started with "Trolley problem: straight or turn", but many complained that this was too ambiguous and needed details. So we went with this. The original trolley problem concerns permissibility rather than obligation, but still, asking about obligation not to switch gets at that reasonably well, and we found that when the question explicitly concerned permissibility the numbers were even more strongly tilted toward not switching. Of course the trolley case works best when conjoined with various other related cases, but we didn't have questions to spare for them this time around. Maybe in a future survey.

Truth: correspondence, deflationary, or epistemic?

Choosing the options here was tricky. We tried "deflationism vs nondeflationism", but it seemed a shame not to get data on correspondence too. But "correspondence vs deflationary" seemed too far from exclusive. Adding "epistemic" (our experts' consensus about the most popular cluster of nondeflationary views) helps a bit, though of course there are still missing options.

Zombies: inconceivable, conceivable but not metaphysically possible, or metaphysically possible?

Pet question. Of course "conceivable" is ambiguous, e.g. between prima facie and ideal conceivability. And one can allow that zombies are possible with holding that entire zombie worlds are possible, e.g. if one is a phenomenal externalists. Still, it's interesting to see the results.

For which nonliving philosophers X would you describe yourself or your work as X-ian or the equivalent?

Some philosophers (especially those working on contemporary analytic philsophy) complained about having to answer this question, while others said we should have asked about influence rather than identification. But the survey was about philosophical views, and the phrasing in terms of identification seems best to get at that. And having the question allows an element of engagement with the history of philosophy which is otherwise missing. The listed philosophers were largely based on Brian Leiter's polls concerning the "most important" philosophers in various historical eras. We included the top 21 from the all-time list (down to Berkeley) and the remainder of the top 17 from the last-200-years list (down to Husserl and Heidegger). Because the resulting list was all-male, we added Anscombe (the highest-ranked woman on the last-200-years list). Of course the choice was somewhat arbitrary. The highest-polling write-in choices were Dewey, Peirce, Sellars, James, and Merleau-Ponty, suggesting that if anything we might have gone further down the last-200-years list.

Consent questions:

We explored various configurations here (consulting with beta testers and the ANU human ethics board) before settling on basic consent to use results and consent to make them public. In the end, about 2% of those who otherwise completed the survey did not give consent to use the results (75 out of 3301 overall, and 16 out of 947 in the target group). We suspect that this is because at the consent point (and on the information page) we suggested that we might release potentially identifying institution-based statistics such "100% of individuals at institution X chose option Y". In fact, we intend to be cautious about releasing such statistics, but we thought that it would be best to overstate rather than understate the risks at the point of consent.


Conclusion

In light of the feedback from 3000+ survey respondents, we might have done a few things differently in retrospect. We might have used the alternative title from the start. To improve the reception of the survey among non-analytic philosophers, we could have more clearly labeled the analytic orientation up front. For similar reasons, we might have dropped some of the more specialized questions (e.g. perceptual experience, contextualism, one or two of the internalism/externalism questions) and some of the thought-experiments (e.g. zombies, trolley problems). We might also have changed the wording of a few questions (e.g. time, moral judgment, politics). We probably would have stayed with the dichotomy format and the brief labels, but for future surveys we may well try other formats.

That being said, we are happy with the survey overall, and we think that it produced interesting and informative results.


Back to contents