This document discusses using neuroscience techniques like eye tracking and fMRI to study information science constructs. It outlines two current projects - one using eye tracking to model reading patterns and measure cognitive effort during information searching, and another using fMRI and eye tracking to study neural correlates of information relevance judgments. The eye tracking project analyzes fixation patterns to infer mental states and predicts how measures like reading speed and regressions relate to task characteristics. The fMRI study aims to identify different brain regions activated during relevant vs. non-relevant information searches and low-level word matching. Preliminary analysis of one participant shows differences in activation for an information search vs. word search task.
1. Towards
Neuro–Information Science
Jacek Gwizdka & Michael Cole
iSchool @ Rutgers University, NJ, USA
jacek@neuroinfoscience.org
http://jsg.tel
June 5, 2012
2. Information Science
Another IS
Information Science is about :
◦ understanding information seeking behavior (why/how/where/…)
◦ helping people find information they need
2
3. Information Systems vs. iScience
A lot of common concerns and constructs:
◦ information is digital accessed via information systems
◦ technology – task – individual
◦ IT usefulness, user interface design, usability …
◦ trust …
◦ decision making …
◦ affective and cognitive factors
◦ information search (e.g., stopping behavior) …
3
4. Information Systems vs. iScience
Also new opportunities:
◦ neural-correlates of constructs specific to Information Science
◦ Information Relevance : most commonly refers to topical relevance
or aboutness, that is: to what extent the content of a search result
matches the topic of the query or a person’s information need (e.g.,
Saracevic, 2007)
relevance judgment decision making
information stopping
4
5. Opportunities for Neuroscience to Inform IS
Seven opportunities for cognitive neuroscience to inform IS research:
1. localize the neural correlates of IS constructs to better understand
their nature and dimensionality;
2. complement existing sources of IS data with neuroscientific data;
3. capture hidden (automatic) processes that are difficult to measure
with existing measurement methods;
4. identify antecedents of IS constructs by exploring the specifics of
how IT stimuli (e.g., the design of graphical user interfaces) are
processed by the brain;
5. test the outcomes of IS constructs by showing how brain activation
predicts behavior (e.g., decisions);
6. infer causality among IS constructs by examining the timing of brain
activations due to a common stimulus;
7. challenge existing IS assumptions and enhance IS theories that do
not correspond to the brain’s functionality
(Dimoka et al. 2010)
5
7. Recent and Current Projects
1. eye-tracking: modeling reading + cognitive effort
2. fMRI + eye-tracking: information relevance
7
8. Part I: Eye-tracking
General research goal: infer and predict mental states and
context of a person engaged in interactive information
searching
Influence system design adaptive systems
Macro
user
task
characteris�cs,
cogni�ve
effort,
domain
knowledge
Meso
reading
pa�erns
Micro
eye-‐gaze
posi�ons
+
�ming
8
11. Eye-movement Patterns
New methodology to analyze eye-movement patterns
◦ Model reading and Measure cognitive effort
◦ Correlate with higher-level constructs
user task characteristics,
user knowledge, etc.
11
12. Reading Model Origins
Based on E-Z Reader model
Rayner , Pollatsek, Reichle
◦ Serial reading
◦ Words can be identified in parafovial region
◦ Early lexical access (word familiarity) + Complete lexical processing (word identification)
2o (70px) foveal region parafoveal region
MORE… 12
13. Two-State Reading Model
q
isolated
fixation fixations
sequences
p
Read
Scan
1-p 1-q
◦ Filter fixations < 150ms (min time required for lexical processing)
◦ Model states characterized by:
probability of transitions; number of lexical fixations; duration
length of eye-movement trajectory, amount of text covered
MORE… 13
14. Example Reading Sequence
Reading sequence:
Fixation model states: (F F F) S (F R F) S S S S (F F FR F F) F
R F F FFFF F S
Reading state – R | Scanning state – S
14
15. Cognitive Effort Measures of Reading
Reading Speed
foveal region
regression
Fixation Regression
a b c d
Perceptual Span
Perceptual span = Mean(a,b,c,d)
Fixation Duration excess
(“lexical processing excess”)
15
16. User Study 1: Cognitive Effort and Tasks
Journalists’
Information Search
OBI:
advanced
obituary
INT:
interview
prepara�on
CPE:
copy
edi�ng
BIC:
background
informa�on
N = 32
Do the cognitive effort measures correlate with:
task difficulty (by design), observable search effort,
user’s subjective perception of task difficulty
Can
we detect differences between task characteristics from
eye-movement patterns?
MORE…
16
17. Eye-data and Cognitive Effort Measures
Subjective Task
Cognitive effort measures Difficulty
Task difficulty derived from eye-tracking
by design reading speed
Copy Editing (CPE) mean fixation duration
Advance Obituary (OBI) perceptual span
total fixation regressions
CPE
INT
BIC
OBI
As expected:
Copy Editing CPE easiest
Search effort Advance Obituary OBI most difficult
task time Sig: Kruskal-Wallis χ2 =46.1, p<.0001
pages visited
queries entered
17
18. Eye-data and Task Characteristics
q
p
Read
Scan
1-p 1-q
Interview
prepara�on Copy
Edi�ng
Measure
Related
Task
Characteris�cs
Frequency
Advanced
obituary
and
Interview
prepara�on
tasks:
SR bias
to
read
search
for
document;
task
goal
not
specific
of
reading
state
Copy
Edi�ng
task:
search
for
segment
and
task
goal
transi�ons
RS bias
to
scan
specific
MORE…
18
19. Summary: Eye-tracking Methodology
Domain independent
◦ Document content is not involved
Culturally*and individually independent
Real-time modeling of user and tasks is possible
Adaptive systems feasible
Eye-tracking is coming to us!
Tobii
19
20. Part II: Current fMRI+eye-tracking Study
Information Relevance : refers to topical relevance
or aboutness, that is: to what extent the content of a document
(webpage) matches the topic of the query or a person’s
information need (e.g., Saracevic, 2007)
◦ Relevance multi-dimensional: topical, meaningful, useful, trust, affective…
Neuralcorrelates of topical relevance judgments
Hypothesis
◦ Brain regions that are activated when relevant information is found are
different from regions activated when no relevant info is found and when
person does “low-level” visual word search (orthographic matching)
but no hypothesis in a sense where the brain activity is located
Exploratory research
(also: a similar experiment with eye-tracking, EEG, GSR)
20
24. Current Experimental Design
Two blocks (types of tasks, balanced)
◦ WS – word search: find target word in a short news story – press yes/no
◦ IS – information search: find information that answers given question –
press yes/no. Three types of trials: relevant (R), topical (T), irrelevant (I)
◦ TR cycle: 2s
21 x
30s 4s 6s 4s 20s max
xmx ssms nsns snsns
4s
WS task
target:
jsdjsd djdjd djdj dkke ekek
+ + +
kdkddk dkdkdk dkdkdkd
instruc- kkdkd d d dd d djdj djdjdj
word
rjrjr rjr jweje ejejej ejej
tions kekekek ekeke wej e eej
eje j
21 x
30s 4s 8s + 20s max + 4s + 20s max
xmx ssms nsns snsns
+ 4s + 20s max +
IS task
xmx ssms nsns snsns
target: target: target:
jsdjsd ke ekek dkdkdkkd jsdjsd djdjd djdj dkke ekek xmx ssms nsns snsns
+
dkdkdkkd jsdjsd ke ekek dkdkdkk
kdkddk dkdkdk dkdkdkd kdkddk dkdkdkdkdkdkd
instruc- kkdkd d rjr jweje ejejej kdkddk dkdkdk dkdkdkd kkdkd d rjr jweje ejeje
info info info
ejej kkdkd d d dd d djdj djdjdj
tions
rjrjr rjjweje ejejej ejej ekeke wej e ejej fjfjf fjfjfjfjf
kekekek ekeke wee ejej fjfjrjr rreje j
fjfjf fjfjfjfjf fjfjrjr rreje j kek ekeke wej e ejej eje j
24
25. Planned Analysis
Two blocks (types of tasks, balanced)
◦ WS – word search: find target word in a short news story
◦ IS – information search: find information that answers given question –
Three types of trials: relevant (R), topical (T), irrelevant (I)
The main contrasts of interests are:
◦ IS-R - WS
◦ IS-R - IS-T
◦ IS-R - IS-I
25
26. A Very, Very Preliminary Analysis
For
one participant, aggregated for all trials in each of two
blocks (tasks)
Word search (WS)
Information Search – Relevant (IS-R)
26