Knowing what we’re talking about: a Mastodon thread

mastodon
Author

Gjalt-Jorn Peters

Published

December 2, 2022

This is a Mastodon thread. The original thread is available here:


Intro

🚨 Preprint alert! 🚨

#Psychology #OpenScience #Measurement #Validity #RStats @rstats@gup.pe

Ever had trouble understanding exactly how to interpret a construct’s definition?

Ever wondered why different articles seem to have different definitions of the same construct?

Ever wanted a solution for this?

We address this issue and present a set of #OpenSource tools to deal with it!

🧵 1/15

A GIF of a woman’s face with lots of equations

Replication, measurement, and theory crises

The replication crisis (or credibility revolution) revealed an underlying “measurement crisis” and a “theory crisis”. One symptom of this theory crisis is the “jingle jangle jungle”, both at the conceptual level and in measurement 🤔

On the one hand, sometimes on closer inspection, constructs or measurement instruments that are purportedly different turn out to be practically identical, preventing accumulation of knowledge and hindering evidence syntheses.

🧵 2/15

A GIF of two spidermen pointing at each other

On the other hand, sometimes measurement instruments supposedly measuring the same thing exhibit substantial differences in content, or in qualitative studies about the same construct, codebooks vary substantially. This means the findings from such studies cannot be compared or aggregated.

🧵 3/15

A GIF of a woman saying “those are two very different things”

Construct Definitions

One cause of this is the exceptionally brief construct definitions used in psychology. Most construct definitions appear to be about 20 words or less - exceptionally concise for things that cannot be directly observed.

As a consequence, researchers using such concepts must first elaborate those definitions for themselves. Such elaborations, however, often remain hidden.

🧵 4/15

A screenshot of a paragraph from the preprint

This heterogeneity in “study-level elaborated construct definitions” is in itself not bad. Epistemic diversity has a number of benefits, and studying multiple variations of construct definitions seems sensible.

However, it is a problem that this heterogeneity is hidden.

So: we want to avoid curation of construct definitions by a central authority, but simultaneously eliminate this hidden heterogeneity.

🧵 5/15

A GIF of a girl saying “Why don’t we have both?”

Decentralized Construct Taxonomies

Enter: Decentralized Construct Taxonomies 💫

The idea is simple; a way to openly share:

  • 1️⃣ A unique construct identifier

  • 2️⃣ A human-readable label

  • 3️⃣ A comprehensive construct definition

  • 4️⃣ Instructions for developing measurement instruments

  • 5️⃣ Instructions for identifying/coding measurement instruments

  • 6️⃣ Instructions for qualitative research

These DCT specifications can easily be shared, re-used, or adapted.

🧵 6/15

A screenshot from the Constructor Shiny App

By combining the comprehensive construct definitions with corresponding instructions for using the construct in quantitative and/or qualitative research, there will hopefully be less need for individual researchers to “fill in the blanks” when they design a study.

In addition, DCT specifications can easily be adapted. This way everybody can redefine constructs if they want to - and through the unique identifiers it’s always clear exactly which construct definition is used.

🧵 7/15

A GIF of a woman saying “We speak the same language”

Producing comprehensive construct definitions

We explain a number of ways to arrive at such comprehensive construct definitions, such as expert consensus, using the Delphi approach, using scoping reviews, and using qualitative and quantitative primary research.

One method is basically documenting the comprehensive, elaborated definitions your research team uses. No extra work, but less ambiguity and more transparency and openness!

🧵 8/15

A GIF of a cat opening a door

The point is that to decrease the hidden heterogeneity, any effort to explicate construct definitions is helpful.

For example, in your research team, you could try the following.

Get together, take a construct you’re studying, and independently write down comprehensive definitions.

You’ll likely end up with different definitions. In some cases, the differences will be small; in others, larger.

🧵 9/15

A GIF from The Big Lebowski with the main character saying “That’s just, like… your opinion, man.

Once you did this, you can start discussing these differences in “implicit elaborated construct definitions”. This will allow you to make sure you’re all on the same page regarding what exactly you’re studying.

Or, alternatively, you could design studies based on where you differ, which can then feed into further theory development.

In any case, this will help you to be more transparent: more Open Science!

🧵 10/15


Five tools for better construct definitions

We also discuss how to work with decentralized construct taxonomies in practice. First, we introduce five tools we developed:

1️⃣ A YAML standard for DCTs

2️⃣ The {psyverse} #rstats 📦

3️⃣ Unique Construct Identifiers

4️⃣ The Constructor Shiny App 🏗️

5️⃣ The open source Psychological Construct Repository, PsyCoRe.one

🧵 11/15

A screenshot of the PsyCoRe.one repository

The Constructor Shiny app, for example (at https://psycore.one/constructor) allows you to:

  • 1️⃣ Type in a DCT specification (a comprehensive construct definition and the corresponding instructions for using it);

  • 2️⃣ Or import it from a spreadsheet (or a Google Sheets URL);

  • 3️⃣ And then creates a Unique Construct Identifier;

  • 4️⃣ Lets you download the .YAML file;

  • 5️⃣ And/or lets you submit it to PsyCoRe.one.

🧵 12/15

A screenshot from the Constructor Shiny App

How to use DCT specifications

Then, we discuss how to use DCT specifications in primary empirical research, both quantitative research and qualitative research; how to use DCT specifications in literature reviews; how to use them in theory development; how to use them in practice; and how to use them in large-scale collaborations.

🧵 13/15

A screenshot from the preprint

What I really like about DCTs is that they’re so super-simple, almost trivial.

Mostly, they’re a concrete, tangible tool for better conceptual clarification - but a tool that also helps to share, re-use, and adapt.

They can help a lab to do more consistent research; in teaching, repositories can help students; and they can help knowledge translation towards practitioners.

🧵 14/15

A GIF of the Negan character from the Walking Dead saying “Easy peasy lemon squeezy”

Review along at Meta-Psychology

Last toot: we submitted this to Meta-Psychology. That means that everybody’s invited to peer review along!

Head over to https://doi.org/jnjp and you can annotate using https://hypothes.is 🤩

🧵 15/15

A GIF from the Simpsons showing Mr. Burns saying “Thanks for the feedback” and shows disposing of shredded paper