Aim: Understanding the effects of psychotherapy is a crucial concern for both research and clinical practice, especially when outcome tends to be negative. Yet, while outcome is predominantly evaluated by means of quantitative pre-post outcome questionnaires, it remains unclear what this actually means for patients in their daily lives. To explore this meaning, it is imperative to combine treatment evaluation with quantitative and qualitative outcome measures. This study investigates the phenomenon of non-improvement in psychotherapy, by complementing quantitative pre-post outcome scores (...) that indicate no reliable change in depression symptoms with a qualitative inquiry of patients’ perspectives. Methods: The study took place in the context of a Randomized Controlled Trial evaluating time-limited psychodynamic and cognitive behavioural therapy for major depression. A mixed methods study was conducted including patients’ pre-post outcome scores on the BDI-II-NL and post treatment Client Change Interviews. Nineteen patients whose data showed no reliable change in depression symptoms were selected. A grounded theory analysis was conducted on the transcripts of patients’ interviews. Findings: From the patients’ perspective, non-improvement can be understood as being stuck between knowing versus doing, resulting in a stalemate. Positive changes (mental stability, personal strength and insight) were stimulated by therapy offering moments of self-reflection and guidance, the benevolent therapist approach and the context as important motivations. Remaining issues (ambition to change but inability to do so) were attributed to the therapy hitting its limits, patients’ resistance and impossibility and the context as a source of distress. “No change” in outcome scores therefore seems to involve a “partial change” when considering the patients’ perspectives. Conclusion: The study shows the value of integrating qualitative first-person analyses into standard quantitative outcome evaluation and particularly for understanding the phenomenon of non-improvement. It argues for more multi-method and multi-perspective research to gain a better understanding of (negative) outcome and treatment effects. Implications for both research and practice are discussed. (shrink)

In psychotherapy research, treatment efficacy is commonly studied by means of self-report questionnaires to gain quantitative data on symptom development. The data serve as input for statistical analyses up to the level of evidence-based treatment. We analyzed how a patient in a psychotherapy study experienced the translation of her story into quantitative data by combining a phenomenological qualitative analysis of her therapeutic narrative and a visual analysis of her extensively annotated paper-andpencil questionnaires. Our findings provide a practical empirical illustration of (...) the nature and impact of known validity issues. Beyond, we found the act of questionnaire administration affected the primary complaint such that it became performative toward the symptoms under study. We discuss the epistemic consequences for the validity of data, and we argue that valid data collection requires qualitative scrutiny of meaning within data and a concept of validity that allows for consideration of performativity of data collection. (shrink)

Psychotherapy research is known for its pursuit of evidence-based treatment. Psychotherapeutic efficacy is assessed by calculation of aggregated differences between pre treatment- and post treatment symptom levels. As this ‘gold standard methodology’ is regarded as ‘procedurally objective’, the efficacy number that results from the procedure is taken as a valid indicator of treatment efficacy. However, I argue that the assumption of procedural objectivity is not justified, as the methodology is build upon a problematic numerical basis. I use an empirical case (...) study to show how measurement problems practically occur in the first step of data collection, i.e. in individual symptom measurement. These problems have been discussed and acknowledged for decades, but still measurement is regarded as the best epistemic means to gain evidence on psychotherapeutic efficacy. Therefore, I show how initial measurement problems are overlooked in the remainder of the methodological procedure, which harms the ‘evidence-base’ of psychotherapeutic EBT. Via this applied analysis, I exhibit concerns that are increasingly raised in the literature in an empirical way, to emphasize the need for a non-idealized consideration of the ‘gold standard methodology’ as a means towards its clinical end. (shrink)

With its focus on evidence, psychology has grown into a mature, professional, and scientifically supported practice over the last decades. In general, psychotherapy and psychological counselling have shown to be more efficacious than waiting it out and a staggering 350 specific treatments have been scientifically supported as effective. Although, evidencebased treatments seem to work equally well, not all people benefit from evidence-based treatments, and it often remains unclear why. This raised the field-wide concern of what works for whom and sparked (...) a wealth of research on specific ingredients, common factors, process factors, and intra- and... (shrink)

This dissertation is focused on the validity of “the data” that are collected in psychotherapy research for the purpose of evidencing treatment efficacy. In the ‘Evidence Based Treatment’ paradigm, researchers rely on the so-called ‘gold standard methodology’ to gather sound and trustworthy evidence, which increasingly influences the organization of mental health care worldwide. In the gold standard, data are collected by quantified self-report measures, to assess the presence and severity of symptoms before and after treatment. When the pre-post difference is (...) larger for a group of people that received the treatment of interest and a group of people who received no or an alternative treatment, the treatment of interest is called effective. In this methodological procedure, researchers tend to assume that when this gold standard methodology is conducted properly, “the data” will speak for themselves. However, when evidence is based on data, that evidence is principally limited by that data. In other words: output depends on input. That implies that when input is flawed, output will be flawed, even when the very best methods of analysis would be used. So what if “the data” yield validity issues despite being collected by validated measures? When “the data” do not straightforwardly evidence treatment effects, will the subsequent steps in the analysis of these data be enough to secure that the evidence in the end does evidence treatment efficacy? In this dissertation, the focus is turned to validity of “the data” that are concretely provided by patient-participants by scoring their own experienced symptoms in self-report questionnaires, in function of evidencing treatment efficacy. A series of empirical case studies were conducted to scrutinize how patient-participants in a randomized controlled psychotherapy study experienced the process of data collection, and how these experiences affected the data that they provided. Each of the studied patient-participants experienced a substantial effect of the questionnaire administration on their complaints. This impacted the level of complaints, but also changed the way in which the complaints were understood at all, and how patient-participants perceived themselves. Thus, rather than neutrally measuring symptoms, questionnaire administration changed the experienced complaints ‘performatively’, which turned measurement into a clinical intervention of its own. Consequently, what is measured cannot straightforwardly be called ‘treatment efficacy’, as it may be entangled with, enabled by or even obstructed by effects of measurement and research. Therefore, the act of measurement itself can pose a vital threat to the validity of “the data”. This way, the empirical case studies exhibited that data can yield validity problems despite the use of validated measures. Consequently, the validity of a measure as such is no guarantee for the validity of data. Nonetheless, in gold standard research, the data are straightforwardly taken as input for analyses of general treatment efficacy. The question is what happens with those validity issues on the level of individual data, when they pursue their journey towards becoming evidence. In this dissertation, it was argued that the validity issues are not sufficiently solved in the methodological steps after data collection, so when data are invalid in the beginning, these validity issues will simply become part of the data set that forms the input for analysis of the final evidence. This urges that validity issues should be solved on the level of individually provided data, as the validity issues will otherwise become inherent to “the data”. Put formally: valid data is a precondition for evidence in EBT. In conclusion, it is crucial for a sound evidence-base to scrutinize the validity of data in function of the overall goal and utility of the research. For this, it is important not to take “the data” as speaking for themselves, but to regard them as clinical narratives, which are framed in a specific format to be communicated between researcher and respondent in a research context. This emphasizes that the choice for a certain format determines what can be evidenced, so it is vital that these choices indeed allow for obtaining evidence that is useful and valid to serve the clinical goal of EBT. (shrink)