Q4 2024
Start of the project
The project has started with the following team: Bartosz Maćkiewicz, Ph.D (Principal Investigator), Katarzyna Kuś and Joanna Gęgotek (Co-Investigators).
Q2 2026
Research article on benefits of meta-analysis in experimental philosophy
Meta-analysis as a research tool has widely accepted benefits to a broad range of empirical research. It has been argued that it can be used to establish whether a given effect is present more reliably than standard, “vote-counting” systematic reviews, detect the existence of publication bias and p-hacking in a given field, estimate effect sizes using multiple studies even in presence of publication bias, and, by employing meta-regression models, identify critical study-level covariates. Although the general usefulness of meta-analysis for empirical research cannot be disputed, experimental philosophers might ask which benefits offered by this approach have the greatest potential to enhance their field and whether it has some deeper implications for experimental philosophy.
The primary theoretical question concerning meta-analysis in the context of experimental philosophy can be summarized in the following way: are meta-analyses primarily a tool for settling controversies within experimental philosophy (e.g., whether such and such effect really exists), or can their results have broader philosophical implications? One of the main lines of research in experimental philosophy is an examination of the universality, and the stability of intuitions. These two characteristics of philosophical intuitions are thought to be crucial for assessing their evidential value. This, in turn, is the greatest issue for intuition-driven philosophical methodology, in terms of which most of the developments in contemporary analytic philosophy can be described.
The putative hypothesis to be explored in the project is that a meta-analytical approach gives us unique tools for the investigation of the issues related to the stability and universality of intuitions. One topic that is seldom explored by experimental philosophers is the inter-study variation of observed effects. The discussions tend to focus on whether a given effect was replicated, and the differences in magnitudes of effect sizes are overlooked. Using meta-analysis, we can deepen this debate.
Q4 2026
Research article reporting the first meta-analysis: contextualism vs relativism
According to a number of theorists, an adequate semantics for various classes of expressions (e.g., epistemic terms or perspectival predicates of personal taste, epistemic modality or aesthetic and moral evaluation) calls for a technical notion of relativist truth or relativist content. There are two main questions in this debate, which several researchers are trying to answer experimentally: whether indeed context affects the content of natural language expressions and if so, what the context sensitivity of a particular type of expression consists in. Two major contenders defending the context-sensitivity are contextualism and relativism. In general, relativists claim that the truth/content of a statement like “Black pudding is tasty” or “Sara knows that the plane has a layover in Chicago” is context-sensitive. It depends on the speaker’s assessment in the case of black pudding, and on Sara and features of her situation (e.g., her practical stakes) in the case of knowledge ascription. On the other hand, according to the contextualists the truth value/content of a given sentence depends partly on the so-called “context of assessment”, i.e., standards and stakes of the subject assessing the truth of such a sentence.
Experimental studies on the subject appear to be non-conclusive. Their results seem to partly depend on the class of expressions studied, and also vary depending on the experimental task and formulation of the question that respondents were asked to answer. The magnitude of the impact of these factors is unclear, however. Existing studies also vary in terms of the languages studied, the scales on which respondents gave their answers, and the types of experimental stimuli. All of these are important study-level covariates whose impact can be evaluated using meta-analytical techniques.
Q4 2027
Research article reporting the second meta-analysis: influence of moral considerations and typicality on causation judgments
One of the most discussed topics from the beginning of experimental philosophy was how our moral judgment affects application of other concepts such as intentional action, knowledge, or cause. Knobe and Fraser (2008) investigated this problem using empirical methods. They found out that people presented with a scenario where two behaviors are jointly a necessary condition for the occurrence of a negative event are more likely to pick norm-violating behavior as a cause. This sparked an entire line of research attempting to examine whether this effect is robust and under which conditions it manifests itself. In the follow-up study, Roxborough and Cumby (2009) manipulated atypicality of the action and found results similar to the original findings. Systma, Livengood, Rose (2012) in the series of studies manipulated (a) whether the information on norm is present in the scenario, (b) experimental design (between-subjects vs. within-subjects), (c) prompt that was used (agent causation vs. agent’s action causation), (d) agent- and population- level typicality. Livengood, Sytsma, Rose (2017) expanded this line of research and replicated the effect with different scenarios, manipulating the experimental design and structure of norms involved. Research on this topic is vast, but similar effects were replicated in subsequent studies that employed different causal structures compared to the original scenario, and for omissions instead of actions. All these factors can be captured in a meta-analytical model and their impact on the effect size can be estimated.
Q4 2028
Research article reporting the third meta-analysis: Folk concept of pain
In general, the 20th century philosophical theory of the folk concept of pain is strongly connected to the mental (introspective) view, seeing pain as a paradigmatic Cartesian mental state and a conscious, private and subjective experience characteristic of sentient beings. Taking the mental view involves acknowledging the impossibility of pain hallucinations or illusions, i.e., that it makes no sense for subjects to doubt whether they are really in pain if they feel it.
The bodily (perceptual) view states that people treat pains as properties of bodily states and assign pains to bodily locations. It has been gaining popularity lately due to a number of experimental studies indicating that the folk reject the idea that pains display properties assumed by the mental view. These are privacy, subjectivity, and incorrigibility. However, opponents of the bodily view argue that most of these studies are characterized by a pretty small sample size that is nor representative of the general population, because the participants were students or people interested in philosophy. Moreover, in the overwhelming number of studies, participants were English speakers only, although there are a few exceptions. The impact of these factors related to the demographic structure of the sample can be investigated using meta-analytical tools. On the other hand, the recent empirical studies relied on by proponents of the polyeidic view are meant to challenge the bodily view.
Q4 2027
Research article on compliance of experimental philosophy with reporting standards
One of the most common problems that preclude the possibility of conducting meta-analyses on existing studies is the unavailability of the specific information required to use meta-analytical statistical techniques. Both in the past and in the present empirical papers lack details about the effect size and its variance. This would not be an issue if they included original, raw data or at least sufficiently detailed descriptive statistics that could enable researchers interested in conducting a meta-analysis to compute appropriate measures of effect sizes themselves. Unfortunately, empirical papers in many disciplines systematically miss even this mark. In addition to the quality of statistical reporting, it is valuable to check whether the essential methodological details on experimental design, method of recruitment, sample structure, and experimental procedure are adequately described in the paper. If this is not the case, key study-level covariates cannot be entered into the meta-analytical model. This may make some sources of variance difficult to account for.
For many fields closely related to experimental philosophy, meta-research consisting of systematic studies on compliance with best reporting practices is routinely conducted. In experimental philosophy, this is not the case. As a part of the project, we will conduct an empirical study on this matter, adapting existing meta-research methodology to specific characteristics of papers typical for experimental philosophy, taking into consideration standard methods used in this field of research. Doing that would enable us to get a better grip on what parts of reporting practices in experimental philosophy need to improve to make meta-analysis more accessible.
The following research questions about the quality of reporting in experimental philosophy will be answered:
1. To what extent do experimental philosophy papers consistently report statistics regarding the effect sizes from the empirical studies?
2. To what extent do experimental philosophy papers include all the descriptive statistics that are necessary to compute the size of the effect and its variance for meta-analysis?
3. To what extent experimental philosophy papers report the most common characteristics of the sample and sampling procedure (including but not limited to the method of recruitment, the demographic structure of the sample, the time in which the studies were conducted, and the language of the study)?
4. To what extent experimental philosophy papers report the necessary detail on the procedure (scales that were used, formulation of the prompts, etc.)?
5. How prevalent is making raw data publicly available in experimental philosophy?
6. Does the quality of reporting (overall and particular aspects specified above) improve in the experimental philosophy over time?
7. Are there systematic differences concerning the quality of reporting standards between academic journals that publish experimental philosophy papers?
Q4 2028
Guidelines for conducting meta-analyses in experimental philopsphy and statistical/methodological reporting of the results
The last main objective of the project is connected with improving the quality of reporting in the field of experimental philosophy and encouraging a meta-analytical approach to existing research. As a deliverable of the project, we will develop two sets of methodological guidelines.
The first one will concern best practices for describing the methodology and the results of the empirical studies. It will be tailored to typical research in experimental philosophy, taking into account its standard methodology and common experimental designs employed by researchers in this field. Building on the results of a planned meta-research study on quality of statistical and methodological reporting, it will also list mistakes and inaccuracies ubiquitous in the field. The overarching goal is to make the data in experimental philosophy more useful for conducting secondary research and to improve transparency in experimental philosophy research. Reporting standards should be tailored to a given discipline and to this date, no guidelines for experimental philosophy have been established.
The second set of guidelines will concern conducting meta-analyses in experimental philosophy. Based on theoretical analysis of the benefits of meta-analysis in experimental philosophy, acquired experience in conducting meta-analyses and the results of quantitative meta-research study, we will be able to identify:
a) kinds of specific questions relevant to experimental philosophy that could be answered using a meta-analytical approach;
b) typical ways in which experiments vary that could be sources of the variability of effect sizes and should be entered into the meta-analytical model;
c) statistical techniques most useful for experimental philosophers interested in conducting their own meta-analyses and how to interpret their results in the context of research questions typical for experimental philosophy.
These findings will then be used to formulate practical recommendations for researchers.