Considerable attention has focused on studying reviewer agreement via inter-rater reliability (IRR) as a way to assess the quality of the peer review process. Inspired by a recent study that reported an IRR of zero in the mock peer review of top-quality grant proposals, we use real data from a complete range of submissions to the National Institutes of Health and to the American Institute of Biological Sciences to bring awareness to two important issues with using IRR (...) for assessing peer review quality. First, we demonstrate that estimating local IRR from subsets of restricted-quality proposals will likely result in zero estimates under many scenarios. In both data sets, we find that zero local IRR estimates are more likely when subsets of top-quality proposals rather than bottom-quality proposals are considered. However, zero estimates from range-restricted data should not be interpreted as indicating arbitrariness in peer review. On the contrary, despite different scoring scales used by the two agencies, when complete ranges of proposals are considered, IRR estimates are above 0.6 which indicates good reviewer agreement. Furthermore, we demonstrate that, with a small number of reviewers per proposal, zero estimates of IRR are possible even when the true value is not zero. (shrink)

Playtime in urban cities has become an indoor activity for children due to limited access to natural outdoor environments. This product of urbanization makes the case for the introduction of biophilic design. However, playrooms are often neglected as a possibility in designing a natural space indoors. Interior designers and other specialists lack a reliable tool to identify and incorporate biophilic features into the design of these indoor environments in urban settings. The Biophilic Interior Design Matrix (BID-M) developed by McGee and (...) Marshall-Baker quantifies 52 of Kellert’s biophilic design attributes to assess their presence and absence within interior spaces. We expanded its use by testing the matrix in a new type of space, urban playrooms, and coded images of 45 children’s playrooms within Manhattan residential buildings in New York City, including assessing a larger sample and reliability rate compared to McGee and Marshall-Baker’s research. Inter-rater reliability of the overall design matrix and individual matrix items was measured with percent agreement and free-marginal multirater kappa. Reliability testing showed overall good reliability of the overall design matrix. Several matrix items had low reliability between raters. Our findings show that the BID-M needs to be modified to better assess urban interior spaces for children. (shrink)

Background. The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq's Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the US Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate (...) the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products. Method. Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments. Results. Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability. Conclusions. Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards. (shrink)

The Fibromyalgia Survey Questionnaire was self-administered by a sample of 207 Italian individuals with obesity to screen for fibromyalgia. We aimed to investigate the inter-rater reliability and the agreement in the detection of FM symptomatology between the self-administered FSQ and the clinical interview conducted by a rheumatologist. All the patients were divided randomly into two groups : a rheumatologist first interviewed patients of group A and after 48 h, the patients completed the self-report FSQ. Patients of group (...) B first completed the FSQ and 48 h later were interviewed by a rheumatologist. The agreement between the measurements was good with the Bland-Altman analysis showing low bias scores for the two subscales of the FSQ. Results showed that 33% of the sample satisfied the criteria for a diagnosis of fibromyalgia. The FSQ is a self-reporting measure that showed substantial reliability providing fast screening for FM symptomatology. (shrink)

This study assessed the International Critical Thinking Essay Test (ICTET-A) for inter-rater reliability, internal reliability, and criterion validity. A self-selecting sample of participants (N = 100) completed the ICTET-A and a comparison test online. We found the ICTET-A items to have moderate to good levels of inter-rater reliability, and overall excellent inter-rater consistency for total test scores. The test had good internal reliability. There was a strong correlation between scores on (...) the ICTET-A and the comparison test. Factor analysis showed that scores were best explained with one factor, suggesting the test measures a single construct. The ICTET-A can therefore be considered a valid measure of critical thinking. Additionally, we propose a short form of the test. (shrink)

Conditional constructions (if–then) enable us to express our thoughts about possible states of the world, and they form an important ingredient for our reasoning and argumentative capabilities. Different types and argumentative uses have been distinguished in the literature, but their applicability to actual language use is rarely evaluated. This paper focuses on the reliability of applying classifications of connections between antecedents and consequents of conditionals to discourse, and three issues are identified. First, different accounts produce incompatible results when applied (...) to language data. Second, a discrepancy between theory and data was observed in previous studies, which sometimes discard existing classifications for being detached from actual language use. Finally, language users construct various cognitive relations between clauses of conditionals without being able to rely on overt linguistic features, which poses problems for the annotation of conditionals in argumentation and discourse. This paper addresses these issues by means of comparing theoretical types and actual uses of conditionals, by inspecting the dispersion of types in natural-language corpora, and by conducting an experiment in which the inter-rater reliability of classifications was assessed. The results show that the reliability of classifications of conditionals when applied to language data is low. With respect to the aforementioned issues, different classifications produced incompatible results, a discrepancy between theory and data was indeed observed, and low reliability scores indicated a largely interpretative nature of types of conditionals. Given these results, suggestions for the enhancement of reliability in corpus studies of conditionals and beyond are provided to enhance future classification design. (shrink)

There are many theories surrounding infant imitation; however, there is no research to our knowledge evaluating the reliability of listener perception of vocal imitation in prelinguistic infants. This paper evaluates intra- and inter-rater judgments on the degree of “imitativeness” in utterances of infants below 12 months of age. 18 listeners were presented audio segments selected from naturalistic recordings to represent in each case a parent vocal model followed by an infant utterance ranging from low to high degrees (...) of imitativeness. The naturalistic data suggested vocal imitation occurred rarely across the first year, but strong intra- and inter-rater correlations were found for judgments of imitativeness. Our results suggest salience of the infant’s vocal imitation despite its rare occurrence as well as active perception by listeners of the imitative signal. We discuss infant vocal imitation as a potential signal of well-being and as an indicator of language learning and vocal communication skills as perceived by caregivers. (shrink)

One of the major roadblocks in conducting Environmental Corporate Social Responsibility (ECSR) research is operationalization of the construct. Existing ECSR measurement tools either require primary data gathering or special subscriptions to proprietary databases that have limited replicability. We address this deficiency by developing a transparent ECSR measure, with an explicit coding scheme, that strictly relies on publicly available data. Our ECSR measure tests favorably for internal consistency and inter-rater reliability, as well as convergent and discriminant validity.

In 2002, the Qualifications and Curriculum Authority (QCA) published the report of an independent panel of experts into maintaining standards at Advanced Level (A-Level). One of its recommendations was for: 'limited experimental double marking of scripts in subjects such as English to determine whether the strategy would significantly reduce errors of measurement' (p. 24). This recommendation provided the impetus for this paper which reviews the all but forgotten literature on double marking and considers its relevance now.

The Ethics Consultation Quality Assessment Tool (ECQAT) establishes standards by which the quality of ethics consultation records (ECRs) can be assessed. These standards relate to the ethics question, consultation-specific information, ethical analysis, and recommendations and/or conclusions, and result in a score associated with one of four levels of ethics consultation quality. For the ECQAT to be useful in assessing and improving the quality of healthcare ethics consultations, individuals who rate the quality of ECRs need to be able to reliably use (...) the tool.We developed a short course to train ethics consultants in using the ECQAT, and evaluated whether the participants (1) achieved an acceptable level of calibration in matching expert-established quality scores for a set of ethics consultations, and (2) were satisfied with the course. We recruited 28 ethics consultants to participate in a virtual, six-session course. At each session participants and faculty reviewed, rated, and discussed one to two ECRs. The participants’ calibration in matching expert-established quality scores improved with repeated exposure at all levels of ethics consultation quality. Participants were generally more accurate when assessing consultation quality at the dichotomous level of “acceptable” (scores of three or four) versus “unacceptable” (scores of one or two) than they were with a more specific score. Participants had higher rates of accuracy with the extreme ratings of “strong” (level four) or “poor” (level one). Although participants were highly satisfied with the course, only a minority of participants achieved the prespecified acceptable level of calibration (that is, 80 percent or greater accuracy between their score and expert-established scores). These results suggest that ECQAT training may require more sessions or need modification in the protocol to achieve higher reliability in scoring. Such trainings are an important next step in ensuring that the ECQAT is a tool that can be used to promote improvement in ethics consultation quality. (shrink)

Critical discourse analysts are being pulled in two directions. On one side, in the age of validity, inter-rater reliability and evidence-based research, it can seem subversive when researchers ‘tell stories’ (rather than ‘write reports’, ‘produce findings’ or ‘demonstrate effectiveness’). On the other side, public relations departments encourage researchers to use ‘storytelling’ techniques to engage public audiences. In this paper, I draw on social and cultural theory to assume that critical discourse analyses are always already narrative. I propose (...) that we embrace (rather than hide) the storytelling qualities of our research. Using similar strategies for all audiences, we undermine traditional power hierarchies between researchers and ‘the public’. Critical discourse analysis then performs its own critical approach. The paper also explores the potential of adopting a ‘carrier bag theory’ of storytelling (Le Guin) rather than telling ‘hero’ stories and reflects on the epistemological, relational and political work done by storytellers in critical discourse studies. (shrink)

Critical discourse analysts are being pulled in two directions. On one side, in the age of validity, inter-rater reliability and evidence-based research, it can seem subversive when researchers ‘tell stories’ (rather than ‘write reports’, ‘produce findings’ or ‘demonstrate effectiveness’). On the other side, public relations departments encourage researchers to use ‘storytelling’ techniques to engage public audiences. In this paper, I draw on social and cultural theory to assume that critical discourse analyses are always already narrative. I propose (...) that we embrace (rather than hide) the storytelling qualities of our research. Using similar strategies for all audiences, we undermine traditional power hierarchies between researchers and ‘the public’. Critical discourse analysis then performs its own critical approach. The paper also explores the potential of adopting a ‘carrier bag theory’ of storytelling (Le Guin) rather than telling ‘hero’ stories and reflects on the epistemological, relational and political work done by storytellers in critical discourse studies. (shrink)

Contemporary psychiatry finds itself in the midst of a crisis of classification. The developments begun in the 1980s—with the third edition of the Diagnostic and Statistical Manual of Mental Disorders —successfully increased inter-rater reliability. However, these developments have done little to increase the predictive validity of our categories of disorder. A diagnosis based on DSM categories and criteria often fails to accurately anticipate course of illness or treatment response. In addition, there is little evidence that the DSM (...) categories link up with genetic findings, and even less evidence that they... (shrink)

Over the last several decades there has been an increase in the amount of research conducted concerning academically dishonest behaviors at the undergraduate level. However, this research and subsequent interventions are based on the assumptions that there exists a clear understanding of what constitutes academic dishonesty. In an attempt to address this gap in the current literature, a concept synthesis of students’ perceptions of academic behavior was completed. The end result was 18 categories of potentially dishonest academic behaviors. Definitions and (...) examples are explored. Inter-rater reliability was completed and Kappa values are reported as well as implications for teaching and future research. (shrink)

BackgroundNarrative reviews are the commonest type of articles in the medical literature. However, unlike systematic reviews and randomized controlled trials (RCT) articles, for which formal instruments exist to evaluate quality, there is currently no instrument available to assess the quality of narrative reviews. In response to this gap, we developed SANRA, the Scale for the Assessment of Narrative Review Articles.MethodsA team of three experienced journal editors modified or deleted items in an earlier SANRA version based on face validity, item-total correlations, (...) and reliability scores from previous tests. We deleted an item which addressed a manuscript’s writing and accessibility due to poor inter-rater reliability. The six items which form the revised scale are rated from 0 (low standard) to 2 (high standard) and cover the following topics: explanation of (1) the importance and (2) the aims of the review, (3) literature search and (4) referencing and presentation of (5) evidence level and (6) relevant endpoint data. For all items, we developed anchor definitions and examples to guide users in filling out the form. The revised scale was tested by the same editors (blinded to each other’s ratings) in a group of 30 consecutive non-systematic review manuscripts submitted to a general medical journal.ResultsRaters confirmed that completing the scale is feasible in everyday editorial work. The mean sum score across all 30 manuscripts was 6.0 out of 12 possible points (SD 2.6, range 1–12). Corrected item-total correlations ranged from 0.33 (item 3) to 0.58 (item 6), and Cronbach’s alpha was 0.68 (internal consistency). The intra-class correlation coefficient (average measure) was 0.77 [95% CI 0.57, 0.88] (inter-rater reliability). Raters often disagreed on items 1 and 4.ConclusionsSANRA’s feasibility, inter-rater reliability, homogeneity of items, and internal consistency are sufficient for a scale of six items. Further field testing, particularly of validity, is desirable. We recommend rater training based on the “explanations and instructions” document provided with SANRA. In editorial decision-making, SANRA may complement journal-specific evaluation of manuscripts—pertaining to, e.g., audience, originality or difficulty—and may contribute to improving the standard of non-systematic reviews. (shrink)

Medical scientists employ ‘quality assessment tools’ (QATs) to measure the quality of evidence from clinical studies, especially randomized controlled trials (RCTs). These tools are designed to take into account various methodological details of clinical studies, including randomization, blinding, and other features of studies deemed relevant to minimizing bias and error. There are now dozens available. The various QATs on offer differ widely from each other, and second-order empirical studies show that QATs have low inter-rater reliability and low (...) inter-tool reliability. This is an instance of a more general problem I call the underdetermination of evidential significance. Disagreements about the strength of a particular piece of evidence can be due to different—but in principle equally good—weightings of the fine-grained methodological features which constitute QATs. (shrink)

Rheumatoid arthritis is a progressive systemic autoimmune disease characterized by inflammation of the joints and surrounding tissues, which seriously affects the life of patients. The Sharp/van der Heijde method has been widely used in clinical evaluation for the RA disease. However, this manual method is time-consuming and laborious. Even if two radiologists evaluate a specific location, their subjective evaluation may lead to low inter-rater reliability. Here, we developed an efficient model powered by deep convolutional neural networks to (...) solve these problems and automated the overall scoring on hand X-rays. The depthwise separable convolution technique is used based on ResNet-50 due to the high resolution of hand X-rays. An inverted residual block is introduced to devise a ResNet-Dwise50 model to enhance the efficiency of the model. The model was trained and tested using bilateral posteroanterior images of 3818 patients. The experiment results show the ResNet-Dwise50 model achieved an MAE of 14.90 and RMSE of 22.01 while ensuring high efficiency. There was no statistical difference between the average scores given by two experienced radiologists and predicted scores from our model. (shrink)

Much research has focused on intergenerational learning. However, its patterns and processes have rarely been explored. Therefore, this study aimed to develop a coding scheme for intergenerational learning, and to explore the patterns of collaborative communication emerging in the context of intergenerational learning. A total of 16 individuals participated in the study. Participants were invited to design digital games with their intergenerational team members. Of all the interactions, two sets of collaborative interactions were coded and analyzed. The findings revealed that (...) the coding scheme for intergenerational learning was developed with high inter-rater reliability for three sub-systems: power process, communication skills, and responses to bids. Additionally, although the patterns of collaborative communication showed a balance of power during the task involving the collection of game elements, younger learners dominated during the task of completing the game prototype. (shrink)

Background Analysis of aphasic narratives can be a challenge for clinicians. Previous studies have mainly employed measures that categorized speech samples at the word level. They included quantification of the use and misuse of different word classes, presence and absence of narrative contents and errors, paraphasias, and perseverations, as well as morphological structures and errors within a narrative. In other words, a great amount of research has been conducted in the aphasiology literature focusing on micro-linguistic structures of oral narratives. Aspects (...) of macro- linguistic structures, such as the analysis of content information by a speaker, consistency of using cohesive devices to present information within a narrative, and order of presenting information necessary to form a coherent discourse, have not been extensively investigated. The current investigation proposes a clinical analytic system to target three aspects of macro-linguistic structures in narratives among speakers with aphasia. Specifically, (1) the presence of search events (i.e., the mentioning of key events that allow the listener to understand; Capilouto, Wright, &Wagovich, 2006) within a narrative, (2) the sequence of the mentioned events, and (3) the informativeness (i.e., the fulfillment of lexical items that allow the user to understand what the event is detailing) of the event contents, were focused in the proposed framework. Method Ten controls transcripts from were selected from the AphasiaBank (MacWhinney, Fromm, Forbes, & Holland, 2011). Three narrative tasks, including sequential picture description of ‘Refused Umbrella’, procedural narrative of making a ‘Peanut Butter and Jelly Sandwich’, and telling of ‘Cinderella’ story, were used to establish normative data for the basis of analysis. Specifically, the Search Events (e) and Informative Words (i) used by at least 70% of the speakers were listed for each genre. The sequential order of mentioning the Search Events, i.e., common order of events in 90% of the speakers, (s.total) was determined. Twelve speakers with aphasia (nine fluent and three non-fluent) were recruited and administered the Western Aphasia Battery and Object and Action Naming Battery. Their performance in the above three discourse tasks were orthographically transcribed and analyzed using the following measures: e.total, e.matched with norm, e.missed, e.irrlevant, e.extra, s.total, i.pb&j, i.umbr, i.cind, and i.total. Three samples were randomly chosen to be re-analyzed for inter- and intra-rater reliability. Results Results of an independent t-test suggested significant differences between the fluent and non-fluent subjects in sequential order of Search Events for all genres (s.total) and i.total, suggesting sensitivity of this framework to differentiate between the two fluency groups. Results of Pearson product-moment correlations revealed significant relation between Informative Words and WAB spontaneous speech scores, AQ, as well as OAB scores, for both aphasic groups. The mean value of coefficients for intra-rater reliability (0.992), with significant correlations on e.match, e.miss, and e.extra, was higher than that of inter-rater reliability (0.897). Discussion Based on our preliminary results, this objective framework allowed clinical evaluation of impairment in macro-linguistic structures present in aphasic discourse. Further extension should involve more subjects encompassing a wider range of severity with different types of aphasia. (shrink)

IntroductionDespite the need for trained physician ethicists, fellowships in clinical ethics are limited and primarily offered to thosewho have completed a graduate degree. The standardization of credentialing for clinical ethics consultants (CECs) and the restructuring of undergraduate medical education allow innovative models to train CECs that can provide an expanded opportunity for formal ethics training at an earlier stage.MethodsAt the University of Michigan Medical School we developed, implemented, and evaluated a pre-doctoral clinical ethics fellowship program from 2017 to 2019 for (...) senior medical students, designed to meet established competencies for CECs. Consultation notes from the beginning and end of the fellowship program were randomly selected for each fellow and independently rated by two faculty ethicists who used the validated Ethics Consultation Quality Assessment Tool (ECQAT).1 Inter rater reliability and trends in scores over time were calculated.ResultsThree students successfully completed the fellowship at the University of Michigan Medical School, conducted at least 50 formal ethics consultations, and spent approximately 40 hours a week on service for a minimum of six months. All students finished the fellowship with teaching and policy development experience, first-authored peer-reviewed manuscripts, and national presentations. The fellows demonstrated improvement in their ECQAT overall holistic rating score; all ended their fellowship with a holistic score of adequate or above. There was 92 percent agreement between raters withrespect to the acceptability of the fellows’ ethics consult notes. Two fellows have taken and passed the Healthcare Ethics Consultant-Certification (HEC-C) exam offered by the American Society for Bioethics and Humanities (ASBH).ConclusionsIntegration of a pre-doctoral fellowship model that meets professional certification standards for clinical ethics consultation creates an opportunity for medical students to become trained CECs early in their career. (shrink)

Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, (...) it will remain unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. (shrink)

Integrating large language models (LLMs) like GPT-4 into medical ethics is a novel concept, and understanding the effectiveness of these models in aiding ethicists with decision-making can have significant implications for the healthcare sector. Thus, the objective of this study was to evaluate the performance of GPT-4 in responding to complex medical ethical vignettes and to gauge its utility and limitations for aiding medical ethicists. Using a mixed-methods, cross-sectional survey approach, a panel of six ethicists assessed LLM-generated responses to eight (...) ethical vignettes.The main outcomes measured were relevance, reasoning, depth, technical and non-technical clarity, as well as acceptability of GPT-4’s responses. The readability of the responses was also assessed. Of the six metrics evaluating the effectiveness of GPT-4’s responses, the overall mean score was 4.1/5. GPT-4 was rated highest in providing technical (4.7/5) and non-technical clarity (4.4/5), whereas the lowest rated metrics were depth (3.8/5) and acceptability (3.8/5). There was poor-to-moderate inter-rater reliability characterised by an intraclass coefficient of 0.54 (95% CI: 0.30 to 0.71). Based on panellist feedback, GPT-4 was able to identify and articulate key ethical issues but struggled to appreciate the nuanced aspects of ethical dilemmas and misapplied certain moral principles.This study reveals limitations in the ability of GPT-4 to appreciate the depth and nuanced acceptability of real-world ethical dilemmas, particularly those that require a thorough understanding of relational complexities and context-specific values. Ongoing evaluation of LLM capabilities within medical ethics remains paramount, and further refinement is needed before it can be used effectively in clinical settings. (shrink)

An important challenge when creating automatically processable laws concerns open-textured terms. The ability to measure open-texture can assist in determining the feasibility of encoding regulation and where additional legal information is required to properly assess a legal issue or dispute. In this article, we propose a novel conceptualisation of open-texture with the aim of determining the extent of open-textured terms in legal documents. We conceptualise open-texture as a lever whose state is impacted by three types of forces: internal forces (the (...) words within the text themselves), external forces (the resources brought to challenge the definition of words), and lateral forces (the merit of such challenges). We tested part of this conceptualisation with 26 participants by investigating agreement in paired annotators. Five key findings emerged. First, agreement on which words are open-texture within a legal text is possible and statistically significant. Second, agreement is even high at an average inter-rater reliability of 0.7 (Cohen’s kappa). Third, when there is agreement on the words, agreement on the Open-Texture Value is high. Fourth, there is a dependence between the Open-Texture Value and reasons invoked behind open-texture. Fifth, involving only four annotators can yield similar results compared to involving twenty more when it comes to only flagging clauses containing open-texture. We conclude the article by discussing limitations of our experiment and which remaining questions in real life cases are still outstanding. (shrink)

Introduction Financial conflicts of interest (fCOI) present well documented risks to the integrity of biomedical research. However, few studies differentiate among fCOI types in their analyses, and those that do tend to use preexisting taxonomies for fCOI identification. Research on fCOI would benefit from an empirically-derived taxonomy of self-reported fCOI and data on fCOI type and payor prevalence.Methods We conducted a content analysis of 6,165 individual self-reported relationships from COI statements distributed across 378 articles indexed with PubMed. Two coders used (...) an iterative coding process to identify and classify individual fCOI types and payors. Inter-rater reliability was κ = 0.935 for fCOI type and κ = 0.884 for payor identification.Results Our analysis identified 21 fCOI types, 9 of which occurred at prevalences greater than 1%. These included research funding (24.8%), speaking fees (20.8%), consulting fees (18.8%), advisory relationships (11%), industry employment (7.6%), unspecified fees (4.8%), travel fees (3.2%), stock holdings (3.1%), and patent ownership (1%). Reported fCOI were held with 1,077 unique payors, 22 of which were present in more than 1% of financial relationships. The ten most common payors included Pfizer (4%), Novartis (3.9%), MSD (3.8%), Bristol Myers Squibb (3.2%), AstraZeneca (3.1%), GSK (3%), Boehringer Ingelheim (2.9%), Roche (2.8%), Eli LIlly (2.5%), and AbbVie (2.4%).Conclusions These results provide novel multi-domain prevalence data on self-reported fCOI and payors in biomedical research. As such, they have the potential to catalyze future research that can assess the differential effects of various types of fCOI. Specifically, the data suggest that comparative analyses of the effects of different fCOI types are needed and that special attention should be paid to the diversity of payor types for research relationships. (shrink)

Direct to consumer offerings of unproven stem cell interventions is a pressing scientific and policy issue. According to media reports, providers of SCIs have emerged in Canada. This study provides the first systematic scan of Canadian providers and associated trends and claims. The study sample consisted of 15 websites retrieved from a Google™ keyword search. The websites were assessed by a rater using a peer-reviewed coding frame that queried treatment location, stem cell offerings, treatment claims, supporting evidence, and legal (...) and regulatory compliance. A second rater reviewed a subset of the websites for purposes of inter-rater reliability. Disagreements between raters were resolved by consensus. Data collected by the raters was analyzed in SPSS. Physicians are the dominant treatment providers in Canada. Providers operate in urban and semi-urban areas in the most populous provinces. SCIs provided are mainly autologous adult stem cells for multiple conditions including musculoskeletal disorders, spinal cord injury and diabetes. Efficacy and benefits of treatment are prominently and positively portrayed, while risks are not mentioned or portrayed as trivial. Regulatory concerns are not discussed. The involvement of physicians in promoting and providing unproven and unapproved SCIs raises significant ethical, legal and regulatory concerns. Treatment claims and trends appear to contravene applicable professional standards, statutory obligations, and consumer protection laws. While the number of providers observed is still marginal, urgent and proactive regulatory response is needed to prevent proliferation of a potentially exploitative and harmful market for unproven SCIs in Canada. (shrink)

Teaching is, to date, one of the most prone jobs to experiencing occupational stress and burnout. Owing to burnout’s negative personal, social, organizational and economic impacts, researchers, practitioners and education policy leaders are interested in developing practices and interventions aimed at preventing/reducing its prevalence. With teachers’ main professional demands to be of a social and emotional nature, interventions designed with a view to promote teachers’ social and emotional competence appears to be particularly promising, positively impacting teachers’ well-being and personal accomplishment (...) and contributing to a decrease in their psychological distress, namely emotional exhaustion. However, theoretical and empirically grounded interventions with ecological validity and specifically targeting teachers are still scarce. Thus, to bridge the previously identified gaps, the present study aimed to evaluate the efficacy and the quality of the intervention’s implementation of the A+, an online social and emotional learning intervention for elementary-school teachers. A quasi-experimental study was conducted with a total of 81 participants from three different school contexts. School clusters were similar in size, organizational structure and socioeconomic level, and as regards previous attendance at social and emotional learning interventions; however, they differed with regards to perceived organizational climate. Data on the efficacy of the A+ was collected across four waves using a set of self-report questionnaires that assessed proximal variables and distal variables, and analyzed through Robust Linear Mixed-Effects Models. Coefficient omegas suggested adequate reliability of the measures. Additionally, two trained observers completed an observation grid to evaluate the quality of the A+ implementation, with excellent inter-rater reliability. Results suggested that, over time, the A+ had positive impacts across proximal and distal variables favoring the experimental group. However, results differed across the school contexts. These findings were accompanied by good implementation quality indicators, namely high fidelity in the delivery of the A+ contents and high participants’ responsiveness. Despite its limitations, this study contributes to a growing body of research which reinforces the importance of investing in social and emotional learning interventions to prevent teachers’ burnout and improve their occupational health. Furthermore, it highlights the importance of implementation quality research as a component of program planning with a view to enhancing programs’ efficacy, as well as the need to adapt and consider context variables in research and practice. (shrink)

Research into expertise is relatively common in cognitive science concerning expertise existing across many domains. However, much less research has examined how experts within the same domain assess the performance of their peer experts. We report the results of a modified think-aloud study conducted with 18 pilots . Pairs of same-ranked pilots were asked to rate the performance of a captain flying in a critical pre-recorded simulator scenario. Findings reveal considerable variance within performance categories, differences in the process used as (...) evidence in support of a performance rating, different numbers and types of facts identified, and differences in how specific performance events affect choice of performance category and gravity of performance assessment. Such variance is consistent with low inter-rater reliability. Because raters exhibited good, albeit imprecise, reasons and facts, a fuzzy mathematical model of performance rating was developed. The model provides good agreement with observed variations. (shrink)

This article reports on Clean Language Interviewing (CLI), a rigorous, recently developed 'content-empty' (non-leading) approach to second-person interviewing in the science of consciousness. Also presented is a new systematic third-person method of validation that evaluates the questions and other verbal interventions by the interviewer to produce an adherence-to-method or 'cleanness' rating. A review of 19 interviews from five research studies provides a benchmark for interviewers seeking to minimize leading questions. The inter-rater reliability analysis demonstrates substantial agreement among (...) raters with an average intraclass correlation coefficient of 0.72 (95% CI). We propose that this method of validation is applicable not only to CLI but to second-person interviews more generally. (shrink)

This study is unique in that it presents the first empirically developed framework for use as a tool for measuring historical empathy. The Historical Empathy Measurement Tool (HEMT) was developed using a design-based research method with three iterative macrocycles of design, experiment, and retrospective analysis. The study involved question responses from 276 students in grade 8 studying WWI trench warfare. The research resulted in a framework of seven levels. (0) Non-response, (1) Facts, (2) Assumptions and Deficits, (3) General Comparison of (...) the Past and Present, (4) Emotional Comparison of the Past and Present, (5) Understanding Motives, Behaviors, Thoughts, & Emotions in the Past, and (6) Acknowledgement of Differences from those in the Past. The inter-rater reliability at the end of macrocycle three was 95%. (shrink)

Introduction The bioethics literature reflects significant interest in and concern with the use of genetic and genomic information in various settings. Because psychiatric treatment and research raises unique ethical, legal, and social issues, we conducted a scoping review of the biomedical, bioethics, and psychology literature regarding the application of genetic and genomic tools to psychiatric disorders (as listed in the DSM-5) and two associated behaviors or symptoms to provide a more detailed overview of the state of the field. Objectives The (...) primary objective was to examine the available bioethics, biomedical, and psychology literature on applying genetic and genomic tools to psychiatric disorders (other than neurodevelopmental disorders) and two behaviors or symptoms sometimes associated with them (aggression or violence and suicidality) to identify the disorders to which these tools have been applied, the contexts in or purposes for which they have been applied, the ethical, legal, or social concerns associated with those uses, and proposed recommendations for mitigating those concerns. Methods We used Arksey and O’Malley’s scoping review framework: (1) identify the research question; (2) identify relevant studies; (3) select studies; (4) chart the data; and (5) collate, summarize, and report results (2005). We relied on Levac et al. to inform our application of the framework (2010). The PRISMA extension for scoping reviews checklist informed our reporting (2018). We searched three electronic databases MEDLINE (PubMed), Embase, and PsycInfo (EbscoHost) for peer-reviewed journal articles in English to identify relevant literature. One author screened the initial results and additional screening was done in consultation with other authors. A data extraction form using DSM-5 diagnostic categories (excluding neurodevelopmental disorders) was developed and two authors independently each reviewed approximately half of the articles. Inter-rater reliability was ensured by double-coding approximately 10% of the papers. An additional author independently coded 10% of the articles to audit the data. Results In 365 coded publications, we identified 15 DSM-5 diagnostic categories in addition to the two pre-selected behaviors or symptoms (aggression or violence and suicidality) to which genetic or genomic tools have been applied. We identified 11 settings in or purposes for which these tools were applied. Twenty-two types of ethical, legal, or social concerns associated with the application of genetic or genomic tools to these disorders or behaviors/symptoms were identified along with 13 practices or policies that could mitigate these concerns. Conclusion Genetic and genomic tools have been applied to a wide range of psychiatric disorders. These raise a range of ethical, legal, and social concerns. Additional research is warranted to better understand the concerns and effective ways to address them. Advancing the literature to identify relevant ethical, legal, or social concerns and solutions to those problems likely requires greater attention to specific applications of genetic or genomic tools to particular psychiatric disorders and associated behaviors/symptoms as well as broad stakeholder engagement. (shrink)

The exit-poll survey of May 21, 1995, the day of the national election, contained after the question of the actual voting behaviour for the Chamber, an open-ended question asking the motives of this voting behaviour. 3547 voters were questioned in the Flemish part of Belgium. Because the exit-poll questioning was done on election day, the voters could not be influenced by the election results and the explanations given by experts in the media. Categorisation of the answers was done by content (...) analysis. The categorisation scheme was developed on the basis of the existing theories of voting behaviour. On this basis an overview is given of the cognitive and affective spontaneous motives the voters are given for their voting behaviour. The inter rater reliability is very high. With a HlCLASS analysis a motive profile for every party is developed. The results indicate that party identification in a broad sense is still one of the most important explanations ofvoting behaviour. But they also make clear that the stability in voting behaviour is very unsure for an important group ofvoters. The two sources of instability are short term ad hoc motives and a-politic or anti-politie motives among a still growing group of voters. (shrink)

Freezing of gait is a common symptom in the late stages of Parkinson’s disease and related disorders. Videos are the gold standard method to conduct FOG scoring; however, the measurement accuracy of FOG scoring based on videos has not been formally assessed, despite its use in previous studies. This study aimed to calculate the measurement accuracy of video-based FOG scoring. Three evaluators scored the FOG based on 157 video data points collected from 21 patients using an annotation tool. One evaluator (...) measured the intra-rater reliability of the retest. The total duration of observed FOG, percentage of the time spent with FOG during the walking task, and FOG phenotypes were evaluated. Intraclass correlation coefficients were used to determine the intra- and inter-rater reliabilities. The duration of FOG and %FOG showed good measurement accuracy for both intra-rater and inter-rater reliabilities. However, the FOG phenotypes showed poor measurement accuracy in inter-rater reliability. These results indicate that the temporal characteristics of FOG can be scored with a high degree of measurement accuracy, even with different evaluators; conversely, the FOG phenotypes need to be scored by several evaluators. (shrink)

Background: There are few standardized measures available to assess executive function in a naturalistic setting for children. The Children’s Cooking Task is a complex test that has been specifically developed to assess EF in a standardized open-ended environment. The aim of the present study was to evaluate the internal consistency, inter-rater reliability, sensitivity and specificity, and also convergent and divergent validity of the Norwegian version of CCT among children with pediatric Acquired Brain Injury and healthy controls.Methods: The (...) present study has a cross-sectional design, based on baseline data derived from a multicenter RCT. Seventy-five children with pABI from two university hospitals with parent-reported executive dysfunction and minimum of 12 months since injury/completed cancer therapy, as well as 59 HCs aged 10–17 years, were assessed with CCT using total errors as the main outcome measure. The pABI group completed tests assessing EF on the impairment level within the ICF framework, and on the participation level. In addition, they completed tests of intellectual ability, processing speed, attention, learning, and memory. Finally, overall functional outcome was evaluated for the children with pABI.Results: Acceptable internal consistency and good inter-rater reliability were found for the CCT. Children with pABI performed significantly worse on the CCT than the HCs. The CCT identified group membership, but the sensitivity and specificity were overall classified as poor. Convergent validity was demonstrated by associations between the CCT and performance-based tests assessing inhibition, cognitive flexibility, and working memory, as well as teacher-reported executive dysfunction. Divergent validity was supported by the lack of association with performance-based measures of learning and memory, attention, and verbal intellectual ability. However, there was a moderate association between the CCT and performance-based tests of processing speed. Lastly, better performance on the CCT was associated with a better functional outcome.Conclusion: Our study with a relatively large sample of children with pABI and HC’s demonstrated good psychometric properties of the CCT. CCT performance was associated with the overall level of disability and function, suggesting that CCT is related to the level of activity in everyday life and participation in society. Hence, our study suggests that the CCT has the potential to advance the assessment of EF by providing a valid analysis of real-world performance. Nevertheless, further research is needed on larger samples, focusing on predictors of task performance, and evaluating the ability of CCT to detect improvement in EF over time. The patterns of error and problem-solving strategies evaluated by the CCT could be used to inform neuropsychological rehabilitation treatmentand represent a more valid outcome measure of rehabilitation interventions. (shrink)

The analysis of penalty kick has played an important role in performance analysis. The study aims are to get formal feedback on the relevance of variables for penalty kick analysis, to design and validate an observational system; and to assess experts’ opinion on the optimum video footage in penalty kick analysis. A structured development process was adopted for content validity, reliability and agreement on video usage. All observational variables included in OSPAF showed Aiken’s V values above the cut-off. Cohen’s (...) Kappa resulted in mean intra- and inter-rater reliability values of 0.90 and 0.86, respectively. It is recommended to combine at least three different viewing angles with standardization of video quality. Changing the viewing angles may influence the observer perception. The aerial and pitch-level viewing angle behind the penalty taker and pitch-level viewing angle behind the goalkeeper were indicated as most appropriate for observational analysis. The OSPAF met all requirements of instrument validation. It may be recommended as basis of future observational systems on penalty kicks. (shrink)

BackgroundThis study aimed at standardizing and providing preliminary evidence on the clinical usability of the Italian telephone-based Verbal Fluency Battery, which includes phonemic, semantic and alternate verbal fluency tasks.MethodsThree-hundred and thirty-five Italian healthy participants and 27 individuals with neurodegenerative or cerebrovascular diseases were administered the t-VFB. Switch number and cluster size were computed via latent semantic analyses. HPs underwent the telephone-based Mental State Examination and Backward Digit Span. Construct validity, factorial structure, internal consistency, test-retest and inter-rater reliability (...) and equivalence with the in-person Verbal Fluency tasks were assessed. Norms were derived via Equivalent Scores. Diagnostic accuracy against clinical populations was assessed.ResultsThe majority of t-VFB scores correlated among each other and with the BDS, but not with the MMSE. Switch number correlated with t-PVF, t-SVF, t-AVF scores, whilst cluster size with the t-SVF and t-AVF scores only. The t-VFB was underpinned by a mono-component structure and was internally consistent. Test-retest and inter-rater reliability were optimal. Each t-VFB test was statistically equivalent to its in-person version. Education predicted all t-VFB scores, whereas age t-SVF and t-AVF scores and sex only some t-SVF scores. Diagnostic accuracy against clinical samples was optimal.DiscussionThe t-VFB is a valid, reliable and normed telephone-based assessment tool for language and executive functioning, equivalent to the in-person version; results show promising evidence of its diagnostic accuracy in neurological populations. (shrink)

Recent conceptualization of built‐in versus bolted‐on corporate social responsibility (CSR) initiatives has offered a much‐needed distinguishing framework to sophisticate our understanding of why different CSR initiatives yield varying corporate social performance (CSP) and associated recognition from stakeholders. One of the major roadblocks in conducting research on these two types of CSR initiatives is the absence of a valid and reliable measure. We address this void by developing a measure for bolted‐on versus built‐in CSR that relies on coding publicly available content. (...) Our measure of these two types of CSR initiatives tests favorably for both convergent and discriminant validity and inter‐rater reliability. Advancement of such a measurement instrument will positively contribute to CSR research, enabling scholars to better theorize the linkage between CSR and reputation, and other aspects of firm performance. (shrink)

The present study sought to determine whether Barrett's counterintuitiveness coding and quantifying scheme could be applied to cultural materials with sufficient intercoder reliability, provide evidence concerning just how counterintuitive is too counterintuitive for a concept to be a recurrent cultural idea, and test whether counterintuitive intentional agent concepts are more common in folktales than other classes of counterintuitive concepts. Seventy-three folktales from around the world were sampled from larger collections. Using Barrett's CI-Scheme, two independent coders identified 116 counterintuitive objects (...) and scored them for degree of counterintuitiveness with very high inter-rater concordance. Of folktales, 79% had one or two counterintuitive objects. Of the counterintuitive objects 93% had a counterintuitiveness score of only one. Of counterintuitive objects, 98% were agents. Results suggest the CI-Scheme may have utility for analyzing cultural materials, that the cognitive optimum for cultural transmission falls around one counterintuitive feature, and that counterintuitive agents are more common than other types of counterintuitive objects in folktales. (shrink)

ObjectiveTwo subregions of the dorsolateral prefrontal cortex have been identified as effective repetitive transcranial magnetic stimulation targets for the “anxiosomatic” and “dysphoric” symptoms, respectively. We aimed to develop a convenient approach to locate these targets on the scalp.Materials and methodsIn a discovery experiment, the two personalized targets were precisely identified on 24 subjects using a neuronavigation system. Then, a localized approach was developed based on individual scalp landmarks. This “landmark-based approach” was replicated and validated in an independent cohort. Reliability (...) of the approach was tested by calculating the correlation of both the inter-rater and intra-rater results. Validity was tested by comparing the mean distance between the personalized and landmark-based targets to the TMS spatial resolution. We further conducted a total of 24 sham rTMS sessions to estimate the misplacement between the coil center and target during a 10-min stimulation without neuronavigation.ResultsThe parameters of the “landmark-based approach” in the discovery experiment were replicated well in an independent cohort. Using discovery parameters, we successfully identified the symptom-specific targets in the independent cohort. Specifically, the mean distance between the personalized and landmark-based targets on the cortex was not significantly larger than 5 mm. However, the personalized and landmark-based targets distance exceeded 5 mm in more than 50% of subjects. During the 10-min sham rTMS session, the average coil misplacement was significantly larger than 5 mm.ConclusionThe “landmark-based approach” can conveniently and reliably locate the two symptom-specific targets at group level. However, the accuracy was highly varied at individual level and further improvement is needed. (shrink)

Adults with chronic pain interpret ambiguous information in a pain and illness related fashion. However, limitations have been highlighted with traditional experimental paradigms used to measure interpretation biases. Whilst ambiguous scenarios have been developed to measure interpretation biases in adolescents with pain, no scenario sets exist for use with adults. Therefore, the present study: sought to validate a range of ambiguous scenarios suitable for measuring interpretation biases in adults, whilst also allowing for two response formats ; and investigate paradigm efficacy, (...) by assessing the effects of recent pain experiences on task responding. A novel ambiguous scenarios task was administered to adults. Participants were presented with 62 ambiguous scenarios comprising 42 that could be interpreted in a pain/pain-illness or non-pain/non-pain illness manner: and 20 control scenarios. Participants generated their own solutions to each scenario, then rated how likely they would be to use two researcher-generated solutions to complete each scenario. Participants also rated their subjective experiences of pain in the last 3 months. Tests of reliability, including inter-rater agreement and internal consistency, produced two ambiguous scenario stimulus sets containing 18 and 20 scenarios, respectively. Further analyses revealed adults who reported more recent pain experiences were more likely to endorse the pain/pain-illness solutions in the Likelihood Ratings Task. This study provides two new stimulus sets for use with adults in pain research and/or interventions. Results also provide evidence for a negative endorsement bias in adults. (shrink)

ObjectiveParenting sensitivity and mutual parent-child attunement are key features of environments that support children’s learning and development. To-date, observational measures of these constructs have focused on children aged 2–6 years and are less relevant to the more sophisticated developmental skills of children aged 7–8 years, despite parenting being equally important at these ages. We undertook a rigorous process to adapt an existing observational measure for 7–8-year-old children and their parents. This paper aimed to: describe a protocol for adapting an existing (...) framework for rating parent-child interactions, determine variations in parents’ sensitive responding and parent-child mutual attunement by family demographics, and evaluate the psychometric properties of the newly developed measure.MethodParent-child dyads completed one home visit, including a free-play observation and parent questionnaire. Dyads were provided with three toy sets: LEGO® Classic Box, Classic Jenga®, and animal cards. The Coding of Attachment-Related Parenting was adapted for use with 7–8-year-old children, and rating procedures were streamlined for reliable use by non-clinician/student raters, producing the SCARP:7–8 Years. Trained staff rated video-recorded observations on 11 behaviors across two domains.ResultsData were available for 596 dyads. Consistently strong inter-rater agreement on the 11 observed behaviors was achieved across the 10-week rating period. Average ICCs were 0.77 for sensitive responding and 0.84 for positive mutuality. These domains were found to be related but distinct constructs. For both domains, average ratings were strongly associated with the main toy used during the observation. Adjusted multivariate linear regression models revealed that less sensitive responding was associated with younger parent, male parent, non-English speaking background, and greater neighborhood disadvantage. Construct validity was demonstrated using six parent-reported psychosocial and parenting measures.ConclusionThe SCARP: 7–8 Years shows promise as a reliable and valid measure of parent-child interaction in the early school years. Toy selection for direct observation should be considered carefully in research and practice settings. (shrink)

Laurence BonJour has recently proposed a novel and interesting approach to the problem of induction. He grants that it is contingent, and so not a priori, that our patterns of inductive inference are reliable. Nevertheless, he claims, it is necessary and a priori that those patterns are highly likely to be reliable, and that is enough to ground an a priori justification induction. This paper examines an important defect in BonJour's proposal. Once we make sense of the claim that inductive (...) inference is "necessarily highly likely" to be reliable, we find that it is not knowable a priori after all. (shrink)

Investigating the bases of inter-individual differences in risk-taking is necessary to refine our cognitive and neural models of decision-making and to ultimately counter risky behaviours in real-life policy settings. However, recent evidence suggests that behavioural tasks fare poorly compared to standard questionnaires to measure individual differences in risk-taking. Crucially, using model-based measures of risk taking does not seem to improve reliability. Here we put forward two possible - not mutually exclusive - explanations for these results and suggest future (...) avenues of research to improve the assessment of inter-individual differences in risk-taking by combining repeated online testing and mechanistic computational models. (shrink)

This study aimed to develop a valid and reliable performance analysis template for quantifying team action variables in elite men’s wheelchair basketball. First action variables and operational definitions were identified by the authors and verified by an expert panel of wheelchair basketball coaching staff in order to establish expert validity. A total of 109 action variable were then placed into 17 agreed Categorical Predictor Variable categories. The action variables were then used to develop a computerized performance analysis template for post-event (...) analysis. Each possession (n = 200) from an international men’s wheelchair basketball game was analyzed by the first author on two occasions for assessment of intra-observer reliability and by a coach and a performance analyst for inter-observer reliability. Percentage error and Weighted Kappa coefficients were calculated to compare the levels of error and agreement for each action variable. Intra-observer reliability demonstrated perfect or almost perfect agreement (. (shrink)