Do IMCs and Time Measure Inattention?

class: inverse,middle,center
<style type="text/css">
.purpleb {
  font-weight: bold;
  color: #4F2683;
  font-size: 1.25em;
}

.purplebL {
  font-weight: bold;
  color: #4F2683;
  font-size: 1.5em;
}

.footnote {
  position: absolute;
  bottom: 60px;
  padding-right: 4em;
  font-size: .75em;
}

.large {
  font-size:1.5rem;
}

.Small {
  font-size:.9rem;
}

.small {
  font-size:.75rem;
}
.tiny {
  font-size:.25rem;
}
.shift { 
  position:relative; 
  top: -40px;
  }

.plot-callout {
  height: 225px;
  width: 450px;
  bottom: 5%;
  right: 5%;
  position: absolute;
  padding: 0px;
  z-index: 100;
}
.plot-callout img {
  width: 100%;
  border: 4px solid
  #  23373B;
}

.remark-slide table tr:nth-child(even) {
  background: none !important;
}
.remark-slide table thead {
  background: none !important;
}
.remark-slide table thead th {
  border-bottom: 1px solid #666;
}

.pull-leftL {
  float: left;
  width: 57%;
}
.pull-rightS {
  float: right;
  width: 37%;
}

.footer {
    font-size: .75rem;
    position: fixed;
    bottom: -8px; 
    left: 0;
    width: 100%;
    text-align: center;
    padding: 1rem 0;
    color: #4F2683;
}

.footer a {
    margin: 0 3rem;
    text-decoration: none;
    color: #4F2683;
    font-weight: 500;
}

.footer a:visited {
    color: #4F2683;
}

.footer a strong,
.footer a b {
    color: #4F2683;
    font-weight: 700;
}

.footer a:hover {
    text-decoration: underline;
}

</style>

# Do IMCs and Time Measure Inattention?

## Testing Established Measures Against Physiological Markers of Inattentive Responding

### William Poirier & Amanda Friesen

2026-05-06

<img src="images/social-science/PNG/SSC_Horiz_Rev.png" width="30%" style="display: block; margin: auto;" />

---

## Plan of presentation

1. Puzzle
2. Concept
3. Measures
4. Design
5. Preliminary Results

---
layout: true

.footer[[**Puzzle**](#puzzle) [Concept](#concept) [Measures](#measures) [Design](#design) [Results](#results) [Conclusion](#conclusion) ]

---
name: puzzle

## Puzzle

--

1. Poor fit between the different measures of inattention (IMC, response time, response pattern) <a name=cite-dunn2018intra></a><a name=cite-desimone2018dirty></a>([Dunn, Heggestad, Shanock, and Theilgard, 2018](#bib-dunn2018intra); [DeSimone and Harms, 2018](#bib-desimone2018dirty));

--

2. Validation of measures is done via nomological validity, testing against established measures <a name=cite-adcock2001measurement></a>([Adcock and Collier, 2001](#bib-adcock2001measurement));

--

3. Conceptual confusion among foundational studies <a name=cite-oppenheimer2009instructional></a><a name=cite-meade2012identifying></a><a name=cite-huang2012detecting></a>([Oppenheimer, Meyvis, and Davidenko, 2009](#bib-oppenheimer2009instructional); [Meade and Craig, 2012](#bib-meade2012identifying); [Huang, Curran, Keeney, Poposki, and DeShon, 2012](#bib-huang2012detecting));

--

4. Advice is to carpet bomb the survey <a name=cite-curran2016methods></a><a name=cite-huang2015detecting></a>([Curran, 2016](#bib-curran2016methods); [Huang, Bowling, Liu, and Li, 2014](#bib-huang2015detecting); [Meade and Craig, 2012](#bib-meade2012identifying));

--

5. At the end of the day, we still can't say whether the poor fit is due to the measures capturing different concepts, or because they are inappropriate.

--

.purplebL[What is the validity of current measures of inattention in survey research?] **... and what would be an appropriate test?**

---
layout: true

.footer[[Puzzle](#puzzle) [**Concept**](#concept) [Measures](#measures) [Design](#design) [Results](#results) [Conclusion](#conclusion) ]

---
name: concept

## Concept

Theory of optimal answering <a name=cite-tourangeau1988cognitive></a>([Tourangeau and Rasinski, 1988](#bib-tourangeau1988cognitive)):

---

## Concept

Theory of optimal answering ([Tourangeau and Rasinski, 1988](#bib-tourangeau1988cognitive)):

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-3-1.png" width="100%" style="display: block; margin: auto;" />

---

## Concept

Theory of optimal answering ([Tourangeau and Rasinski, 1988](#bib-tourangeau1988cognitive)):

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-4-1.png" width="100%" style="display: block; margin: auto;" />

---

## Concept

Theory of optimal answering ([Tourangeau and Rasinski, 1988](#bib-tourangeau1988cognitive)):

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-5-1.png" width="100%" style="display: block; margin: auto;" />

---

## Concept

Theory of optimal answering ([Tourangeau and Rasinski, 1988](#bib-tourangeau1988cognitive)):

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-6-1.png" width="100%" style="display: block; margin: auto;" />

---

## Concept

Theory of optimal answering ([Tourangeau and Rasinski, 1988](#bib-tourangeau1988cognitive)):

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-7-1.png" width="100%" style="display: block; margin: auto;" />

(Not unlike <a name=cite-zaller1992nature></a>[Zaller (1992)](#bib-zaller1992nature)'s RAS model.)

---

## Concept

.large[ `\(\boldsymbol\mapsto\)` We define inattentive responding (IR) as a failing to follow one or multiple of
these steps due to a lack of **motivation** or **ability**.]

Formally:

`$$IR_i = M_i\times A_i= 
\left\lbrace\begin{align}
&1 \quad \mathrm{if}\; M_i=0\lor A_i=0 \\ 
&0 \quad \mathrm{if}\; M_i=1\land A_i=1
\end{align}\right.$$`

.footnote[<a name=cite-anduiza2017answering></a><a name=cite-berinsky2024measuring></a>([Anduiza and Galais, 2017](#bib-anduiza2017answering); [Berinsky, Frydman, Margolis, Sances, and Valerio, 2024](#bib-berinsky2024measuring))]

---

layout: true

.footer[[Puzzle](#puzzle) [Concept](#concept) [**Measures**](#measures) [Design](#design) [Results](#results) [Conclusion](#conclusion) ]

---

name: measures

## Measures — The classics

.panelset[
  .panel[.panel-name[Direct]
  - **Provide snapshot assessments!**
  - Intructional Manipulation Checks (IMC):
    - "[...] Forget previous instructions and choose 'Strongly Agree'."
    - Can be more or less overt.
  - Bogus/Infrequency items:
    - "I am paid biweekly by leprechauns."
    - Covert
  - Factual Manipulation Checks (FMC)/Mock Vignette Checks (MVC):
    - For experimental designs, to check treatment compliance.
    
  .small[<a name=cite-kane2019no></a><a name=cite-mancosu2019short></a><a name=cite-kane2023analyze></a>([Oppenheimer, Meyvis, and Davidenko, 2009](#bib-oppenheimer2009instructional); [Meade and Craig, 2012](#bib-meade2012identifying); [Kane and Barabas, 2019](#bib-kane2019no); [Mancosu, Ladini, and Vezzoni, 2019](#bib-mancosu2019short); [Kane, Velez, and Barabas, 2023](#bib-kane2023analyze))]
  ]
  .panel[.panel-name[Response time]
  - **Offers global assessment!**
  - Page time indices:
    - Ad hoc threshold (2 seconds per question);
    - Concerned with respondents going too fast.
  - Response Time Attentiveness Clustering (RTAC):
    - 2 steps:  1) PCA; 2) EM clustering.
    - Gives probability of being in each of three clusters:
      1. Fast inattentive;
      2. Attentive;
      3. Slow inattentive.
      
   .small[<a name=cite-bowling2021quick></a><a name=cite-read2022racing></a>([Huang, Curran, Keeney et al., 2012](#bib-huang2012detecting); [Bowling, Huang, Brower, and Bragg, 2021](#bib-bowling2021quick); [Read, Wolters, and Berinsky, 2022](#bib-read2022racing))]
  ]
  .panel[.panel-name[Response pattern]
  - **Offers global assessment!**
  - Longest streak
  - Outlier detection
  - Individual consistency
  
`\(\mapsto\)` Not recommended as offer poor consistency with other measures and are overreliant on strong assumptions.
  
  .small[<a name=cite-converse2006nature></a>([Curran, 2016](#bib-curran2016methods); [DeSimone and Harms, 2018](#bib-desimone2018dirty); [Converse, 2006](#bib-converse2006nature))]
  ]
]

---

## Measures — Physiology

.panelset[
  .panel[.panel-name[Eye-tracking]
  - Infrared sensors used to track gaze time in area of interest.
  - Assumption that where you look is where attention is.
  - Used to measure task engagement <a name=cite-fortenbaugh2017recent></a><a name=cite-baumgartner2018misresponse></a>([Fortenbaugh, DeGutis, and Esterman, 2017](#bib-fortenbaugh2017recent); [Baumgartner, Weijters, and Pieters, 2018](#bib-baumgartner2018misresponse)).
  - Used to investigate answer ordering effects, and conjoint experiment assumptions <a name=cite-galesic2008eye></a><a name=cite-jenke2021using></a>([Galesic, Tourangeau, Couper, and Conrad, 2008](#bib-galesic2008eye); [Jenke, Bansak, Hainmueller, and Hangartner, 2021](#bib-jenke2021using)).
  - <a name=cite-babakhani2023instructional></a>[Babakhani, Paas, and Dolnicar (2023)](#bib-babakhani2023instructional) provides the only study that directly tests the validity of IMCs on physiological manifestations of attention (n=21). 
    - Find that task miscomprehension is also to blame for IMC failure.
<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-8-1.png" width="60%" style="display: block; margin: auto;" />
  ]
  .panel[.panel-name[EDA]
  - Electrodermal activity (skin conductance) varies as a function of sweat production.
  - Eccrine sweat gland activation part of sympathetic nervous system (the fight-or-flight response).
  - Associated with a wide array of states: arousal, stress, motivation, vigilance, etc. <a name=cite-andreassi2000activity></a>([Andreassi, 2000](#bib-andreassi2000activity)).
  - If treatment produces stress response, might be difficult to parse out what exactly made EDA increase.
  - Successfully used to measure vigilance <a name=cite-eason1965performance></a><a name=cite-krupski1971physiological></a><a name=cite-tao2019systematic></a><a name=cite-brishtel2020mind></a>([Eason, Beardshall, and Jaffee, 1965](#bib-eason1965performance); [Krupski, Raskin, and Bakan, 1971](#bib-krupski1971physiological); [Tao, Tan, Wang, Zhang, Qu, and Zhang, 2019](#bib-tao2019systematic); [Brishtel, Khan, Schmidt, Dingler, Ishimaru, and Dengel, 2020](#bib-brishtel2020mind)).
<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-9-1.png" width="60%" style="display: block; margin: auto;" />
  ]
  .panel[.panel-name[HRV]
  - Heart rate variation corresponds to the variation in the interval between heart beats, is it consistent or not.
  - Associated with global cognitive performance, memory, language, **attention**, executive functions, visuospatial skills, and processing speed <a name=cite-forte2019heart></a>([Forte, Favieri, and Casagrande, 2019](#bib-forte2019heart)).
  - Lower HRV during attention phases <a name=cite-richards1991heart></a><a name=cite-chen2010detecting></a><a name=cite-williams2016resting></a><a name=cite-griffiths2017sustained></a>([Richards and Casey, 1991](#bib-richards1991heart); [Chen, Wang, Chen, Wu, Yang, Wang, and Chung, 2010](#bib-chen2010detecting); [Williams, Thayer, and Koenig, 2016](#bib-williams2016resting); [Griffiths, Quintana, Hermens, Spooner, Tsang, Clarke, and Kohn, 2017](#bib-griffiths2017sustained)).
<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-10-1.png" width="60%" style="display: block; margin: auto;" />
  ]
]

---

## Measures — Physiology

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-11-1.png" width="60%" style="display: block; margin: auto;" />

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-12-1.png" width="90%" style="display: block; margin: auto;" />

---

## Measures — Physiology

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-13-1.png" width="60%" style="display: block; margin: auto;" />

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-14-1.png" width="90%" style="display: block; margin: auto;" />

---

## Measures — Physiology

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-15-1.png" width="60%" style="display: block; margin: auto;" />

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-16-1.png" width="90%" style="display: block; margin: auto;" />

---
layout: true

.footer[[Puzzle](#puzzle) [Concept](#concept) [Measures](#measures) [**Design**](#design) [Results](#results) [Conclusion](#conclusion) ]

---
name: design

## Design — Ideal Scenario

Recall our definition of IR:

`$$IR_i = M_i\times A_i= 
\left\lbrace\begin{align}
&1 \quad \mathrm{if}\; M_i=0\lor A_i=0 \\ 
&0 \quad \mathrm{if}\; M_i=1\land A_i=1
\end{align}\right.$$`

--

Let's describe a model of attention as a function of an external stimuli:

`$$IR_{mi}=\beta_m D_{i}+\mathbf{X}\boldsymbol{\gamma_{m}}+\varepsilon_{mi}$$`

---

## Design — Ideal Scenario

Recall our definition of IR:

`$$IR_i = M_i\times A_i= 
\left\lbrace\begin{align}
&1 \quad \mathrm{if}\; M_i=0\lor A_i=0 \\ 
&0 \quad \mathrm{if}\; M_i=1\land A_i=1
\end{align}\right.$$`

Let's describe a model of attention as a function of an external stimuli:

`$$\color{#FF0000}{IR_{mi}}=\beta_m D_{i}+\mathbf{X}\boldsymbol{\gamma_{m}}+\varepsilon_{mi}$$`
- `\(\color{#FF0000}{m}\)`:
  1. HRV;
  2. Eye-tracking;
  3. EDA;
  4. IMC index (overt, covert, bogus item);
  5. Response time index;
  6. RTAC.

---

## Design — Ideal Scenario

Recall our definition of IR:

`$$IR_i = M_i\times A_i= 
\left\lbrace\begin{align}
&1 \quad \mathrm{if}\; M_i=0\lor A_i=0 \\ 
&0 \quad \mathrm{if}\; M_i=1\land A_i=1
\end{align}\right.$$`

Let's describe a model of attention as a function of an external stimuli:

`$$IR_{mi}=\color{#FF0000}{\beta_m D_{i}}+\mathbf{X}\boldsymbol{\gamma_{m}}+\varepsilon_{mi}$$`
- `\(\color{#FF0000}{D_{i}}\)`: some stimuli that would instigate inattention.

---

## Design — Experiment

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-17-1.png" width="100%" style="display: block; margin: auto;" />

---

## Design — Experiment

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-18-1.png" width="100%" style="display: block; margin: auto;" />

---

## Design — Experiment

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-19-1.png" width="100%" style="display: block; margin: auto;" />

---

## Design — Experiment

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-20-1.png" width="100%" style="display: block; margin: auto;" />

---

## Design — Experiment — Treatment

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-21-1.png" width="50%" style="display: block; margin: auto;" />
<img src="figs/treat_pc_1.png" width="70%" style="display: block; margin: auto;" />

---

## Design — Experiment — Treatment

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-23-1.png" width="50%" style="display: block; margin: auto;" />
<img src="figs/treat_pc_2.png" width="70%" style="display: block; margin: auto;" />

---

## Design — Experiment — Control

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-25-1.png" width="50%" style="display: block; margin: auto;" />
<img src="figs/control_pc.png" width="70%" style="display: block; margin: auto;" />

---

## Design — Experiment

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-27-1.png" width="100%" style="display: block; margin: auto;" />

---

## Design — Experiment

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-28-1.png" width="100%" style="display: block; margin: auto;" />

- You probably have a favourite colour. But we are more interested in making sure you're doing the survey carefully, so please just select the colour brown here. (CES, 2021)
- In getting what you want, it is sometimes necessary to use force against other groups (ignore the instructions and choose "agree"). (Adapted from Pratto et al., 1994)
- I am paid biweekly by leprechauns. (Meade and Craig, 2012)

---

## Design — Experiment

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-29-1.png" width="100%" style="display: block; margin: auto;" />

- Taking a survey can be pretty boring, what types of things did you do to make it go faster?

---

## Design — Hardware & Software

.pull-left[
- Webcam based eye-tracking:
  - WebGazer.js integrated into Qualtrics.

]
.pull-right[
<img src="images/Calib.png" width="100%" style="display: block; margin: auto;" />
]

---

## Design — Hardware & Software

.pull-left[
- Webcam based eye-tracking:
  - WebGazer.js integrated into Qualtrics.

]
.pull-right[
<img src="images/matrix.png" width="100%" style="display: block; margin: auto;" />
]

---

## Design — Hardware & Software

.pull-left[
- Webcam based eye-tracking:
  - WebGazer.js integrated into Qualtrics.
  - Advertised `\(\approx4°\)` accuracy, translated into average error of 122px in practice.
  - Measure by Hutt et al. (2024):
      1. Global: How many points, how many areas of the screen visited, average distance of gazes from centroid.
      2. Local: How many gazes on AOIs of interest (question label, choices and progress bar).
  - Brought together via PCA -> GMM pipeline.
]
.pull-right[
<img src="images/matrix_error.png" width="100%" style="display: block; margin: auto;" />
]

---

## Design — Hardware & Software

.pull-left[
- Webcam based eye-tracking:
  - WebGazer.js integrated into Qualtrics.
  - Advertised `\(\approx4°\)` accuracy, translated into average error of 122px in practice.
- EDA & HRV via the Research ring from Biopac.
]
.pull-right[
<img src="images/sensor_combined.png" width="100%" style="display: block; margin: auto;" />
]

---

## Design — Hardware & Software

.pull-left[
- Webcam based eye-tracking:
  - WebGazer.js integrated into Qualtrics.
  - Advertised `\(\approx4°\)` accuracy, translated into average error of 122px in practice.
- EDA & HRV via the Research ring from Biopac.
  - Configuration with electrode above right clavicle.
  - Allows for cleaner HRV signal.
]
.pull-right[
<img src="images/ringPos.png" width="100%" style="display: block; margin: auto;" />
]
---

## Design — Hardware & Software

.pull-left[
- Webcam based eye-tracking:
  - WebGazer.js integrated into Qualtrics.
  - Advertised `\(\approx4°\)` accuracy, translated into average error of 122px in practice.
  - Measure by Hutt et al. (2024):
    1. Global: How many points, how many areas of the screen visited, average distance of gazes from centroid.
    2. Local: How many gazes on AOIs of interest (question label, choices and progress bar).
  - Brought together via PCA -> GMM pipeline.
- EDA & HRV via the Research ring from Biopac.
  - Configuration with electrode above right clavicle.
  - Allows for cleaner HRV signal.
- Final set-up.
]
.pull-right[
<img src="images/final_setup.png" width="70%" style="display: block; margin: auto;" />
]

---

## Design — Data

- With available ressources, we are aiming at 300 respondents.
- Student sample via mass email recruitment.
- 10$ compensation.
- 107 participants to date.
- Collection started in May 2026.

---
layout: true

.footer[[Puzzle](#puzzle) [Concept](#concept) [Measures](#measures) [Design](#design) [**Results**](#results) [Conclusion](#conclusion) ]

---
name: results

## Results — Treatment Assignment

<img src="figs/ses_distri.png" width="90%" style="display: block; margin: auto;" />

---

## Results — Across SUR

<img src="figs/coef_across_fd.png" width="90%" style="display: block; margin: auto;" />

---

## Results — Across SUR

<img src="figs/coef_across_fd_mde.png" width="90%" style="display: block; margin: auto;" />

---

## Results — Compliance

<img src="figs/eda_compliance_sweep.png" width="90%" style="display: block; margin: auto;" />

---

## Results — Compliance

<img src="figs/compliance_sweep.png" width="90%" style="display: block; margin: auto;" />

---

## Results — Within SUR

<img src="figs/coef_within_total.png" width="90%" style="display: block; margin: auto;" />

---

## Results — Within SUR

<img src="figs/coef_within_total_mde.png" width="90%" style="display: block; margin: auto;" />

---

## Results — Measurement Agreement (Post)

<img src="figs/agreement_post.png" width="90%" style="display: block; margin: auto;" />

---

## Results — Exploratory Factor Analysis (Post)

<img src="figs/factor_loadings_post.png" width="90%" style="display: block; margin: auto;" />

---
layout: true

.footer[[Puzzle](#puzzle) [Concept](#concept) [Measures](#measures) [Design](#design) [Results](#results) [**Conclusion**](#conclusion) ]

---

name: conclusion

## Conclusion — What have we learned?

- Assuming our gaze measure is right:
  1. IMCs are a bad way to measure inattention.
  2. RTAC is the closest in terms of ATE estimation (Potentially overestimating it).
  3. Within analysis confuses this trend.
  4. None of the measures scale well together.

---

class: inverse,middle,center
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/7.0.0/css/all.min.css">
## Questions? Comments? Testimonies?

.pull-left[
### Contacts

<i class="fa-regular fa-envelope"></i> wpoirier@uwo.ca

<i class="fa-brands fa-bluesky"></i> @olsisblue.bsky.social

<i class="fa-brands fa-github"></i> WilliamPo1

Or visit my website! `\(\longmapsto\)`

]
.pull-right[
<img src="figs/website_rev.png" width="70%" style="display: block; margin: auto;" />
]

---

## Appendix — DVs Distribution

<img src="figs/imc_fmc_composite.png" width="90%" style="display: block; margin: auto;" />

---

## Appendix — DVs Distribution

<img src="figs/dv_distri.png" width="90%" style="display: block; margin: auto;" />

---

## Appendix — RTAC

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-48-1.png" width="100%" style="display: block; margin: auto;" />

---
## Appendix — RTAC

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-49-1.png" width="100%" style="display: block; margin: auto;" />

---

## Appendix — RTAC

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-50-1.png" width="100%" style="display: block; margin: auto;" />

- Principal component analysis (PCA) answer:
  - **What is the minimum amount of "dimensions" (variables) I need to explain the maximum amount of variance?**

---

## Appendix — RTAC

<img src="itmi_mplmeth_files/figure-html/unnamed-chunk-51-1.png" width="100%" style="display: block; margin: auto;" />

- Gaussian mixture models (GMM) answer:
  - **To which of `\(k\)` distribution is `\(i\)` more likely to come from?**
- Assume that each observation comes from one of `\(k\)` multivariate-normal distribution.
- Expectation maximization (EM) answer:
  - **How many distributions are there?**
--

- [Read, Wolters, and Berinsky (2022)](#bib-read2022racing) impose 3 clusters.