Haustein, S. (2017, February). The evolution of scholarly communication and the reward system of science. Fourth Annual KnoweScape Conference 2017, 22–24 February 2017, Sofia (Bulgaria). keynote http://knowescape.org/knowescape2017/

1. The
evolution of
scholarly communication
and
the
reward system
of
science
Stefanie
Haustein
@stefhaustein

2. Outline
Scholarly
communication
From
the
16th century
to
Open
Science
Bibliometrics
From
library
management
to
research
evaluation
Altmetrics
Opportunities
and
challenges
Conclusions
and
Outlook

4. Invisible
Colleges
http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpg
http://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg
Père
Marin
Mersenne
(1588-­‐1648)
Henry
Oldenburg
(1619-­‐1677)

5. Scientific
Societies
http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-­‐Louis_XIV_und_Colbert_in_der_Akademie.jpg
https://www.rcseng.ac.uk/museums/hunterian/images/lost-­‐museums-­‐2011/the-­‐royal-­‐society-­‐repository/Image%201%20-­‐%20GreshamCollege.jpg/image_preview
http://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg
L’Académie
royale
des
sciences
22
December 1666
The
Royal
Society
28
November 1660

6. Scientific
Journals
http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpg
http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg
Le
journal
des
sçavans
5
January 1665
Philosophical Transactions
6
March
1665

7. Scientific
Articles
Harmon,
J.E.
&
Gross,
A.G.
(2007).
The
Scientific
Literature.
A
Guided
Tour.
Chicago:
University
of
Chicago
Press.
• Experiments
and
descriptions
of
the
natural
world
• Avoiding
“fine
speaking”
• Various
styles
of
arguing
• Qualitative
and
personal
judgements

8. Proportion
of
IMRaDadoption
in
medicaljournalsNumberof
references1900
to
2004
Scientific
Articles
Larivière,
V.,
Archambault,
É.
&
Gingras,
Y.
(2008).
Long-­‐term
variations
in
the
aging
of
scientific
literature:
From
exponential
growth
to
steady-­‐state
science
(1900-­‐2004).
Journal
of
the
American
Society
for
Information
Science
and
Technology,
59(2),
288-­‐296.
Sollaci,
L.B.
&
Pereira,
M.G.
(2004).
The
introduction,
methods,
results,
and
discussion
(IMRAD)
structure:
a
fifty-­‐year
survey.
Journal
of
the
Medical
Library
Association,
92(3),
364-­‐371
• Professionalized
and
highly
specialized
• Increased
focus
on
data,
graphs,
tables
and
theory
• Impersonal,
technical
and
codified
• Style
guides
and
gatekeeping
• Citations
• Introduction,
Methods,
Results
and
Discussion

9. Digital
Revolution
arXiv submission
statistics
from
http://arxiv.org/stats/monthly_submissions
Larivière,
V.,
Lozano,
G.A.
&
Gingras,
Y.
(2014).
Are
elite
journals
declining?
Journal
of
the
Association
for
Information
Science
and
Technology,
65(4),
649-­‐655.
• Improved access
• Acceleration
• Collaboration
• Peer
review
• Distribution
of
preprints
• Decreasing importance
of
scientific journal
• Journal
functions
• Diversification
of
publication
venues
• Symbolic capital
of
journals unchanged
Submissionsto
arXiv
Share
of
top
1%
mostcitedpapers

10. Academic Publishing Market
Larivière,
V.,
Haustein,
S.,
&
Mongeon,
P.
(2015).
The
oligopoly
of
academic
publishers
in
the
digital
era.
PLoS ONE,
10(6),
e0127502.
doi:
10.1371/journal.pone.0127502
• Aggravation
of
serials
crisis
• Elsevier:
30%
increase
of
subscription
price
• Profit
margins
of
commercial
publishers
up
to
40%
• Decline
of
scientific
societies
as
publishers
• >50%
of
papers
owned
by
five
major
publishers

11. Open
Access
Archambault,
É.,
Amyot,
D.,
Deschamps,
P.,
Nicol,
A.,
Rebout,
L.
&
Roberge,
G.
(2013).
Proportion
of
Open
Access
Peer-­‐Reviewed
Papers
at
the
European
and
World
Levels
2004-­‐2011.
Report
for
the
European
Commission.
http://www.science-­‐metrix.com/pdf/SM_EC_OA_Availability_2004-­‐2011.pdf
Budapest
Open
Access
Initiative
“immediate,
free
availability
on
the
public
internet,
permitting
any
users
to
read,
download,
copy,
distribute,
print,
search
or
link
to
the
full
text
of
these
articles”
• Gold
and
Green
• Libre
and
Gratis
• Hybrid
• Elsevier:
$500
to
5,000
• Springer:
$3,000
• Wiley:
$3,000
Freelyavailablejournal
papers2004
to
2011
Budapest
Open
Access
Initiative
(2002)

12. Open
Science
Kraker,
P.,
Leony,
D.,
Reinhardt,
W.
&
Beham,
G.
(2011).
The
case
for
an
open
science
in
technology
enhanced
learning.
International
Journal
of
Technology
Enhanced
Learning,
3(6),
643-­‐654.
“opening
up
the
research
process
by
making
all
of
its
outcomes,
and
the
way
in
which
these
outcomes
were
achieved,
publicly
available
on
the
World
Wide
Web”
• Open
Data
• Open
Source
• Open
Methodology
• Open
Access
• Open
Peer
Review
Krakeret
al.
(2011,
p.
645)

14. Bibliometrics
Gross,
P.L.K.
&
Gross,
E.M.
(1927).
College
libraries
and
chemical
education.
Science,
66(1713),
385-­‐389.
Library
collection
management
Journalscitedin
the
Journal
of
the
American
Chemical
Society1926

15. Bibliometrics
Garfield,
E.
(1955).
Citation
indexes
for
science.
A
new
dimension
in
documentation
through
association
of
ideas.
Science,
122,
108-­‐111.
Information
retrieval
•
“It
would
not
be
excessive
to
demand
that
the
thorough
scholar
check
all
papers
that
have
cited
or
criticized
such
papers,
if
they
could
be
located
quickly.
The
citation
index
makes
this
check
practicable.”
• Institute
for
Scientific
Information
• Science
Citation
Index
• Source
Author Index
• Citation
Index
Garfield
(1955,
p.
108)

16. Bibliometrics
Price,
D.
J.
d.
S.
(1961).
Science
Since
Babylon.
New
Haven
/
London:
Yale
University
Press,
Price,
D.
J.
d.
S.
(1963).
Little
Science,
Big
Science.
New
York:
Columbia
University
Press.
Sociology
of
science
Derek
J.
de
Solla Price
Science
since
Babylon
(1961)
Little
Science
– Big
Science
(1963)

17. Bibliometrics
Merton,
R.
K.
(1988).
The
Matthew
effect
in
science,
II:
Cumulative
advantage
and
the
symbolism
of
intellectual
property.
Isis,
79,
606–623.
Sociology
of
science
Robert
K.
Merton
• Social
norms of
science
• Communalism
• Universalism
• Disinterestedness
• Organized skepticism
• Matthew
effect
“symbolically,
[the
reference]
registers
in
the
enduring
archives
the
intellectual
property
of
the
acknowledged
source
by
providing
a
pellet
of
peer
recognition
of
the
knowledge
claim”
Merton
(1988,
p.
621)

18. Bibliometrics
Moed,
H.F.,
Burger,
W.J.M.,
Frankfort,
J.G,
van
Raan,
A.F.J.
(1985).
The
use
of
bibliometric
data
for
the
measurement
of
university
research
performance.
Research
Policy,
14(3),
131-­‐149.
Research
evaluation
• Performance
measurement and
policy instrument
“When
used
properly,
bibliometric
indicators
can
provide
a
‘monitoring
device’
for
university
research-­‐management
and
science
policy.
They
enable
research
policy-­‐makers
to
ask
relevant
questions
of
researchers
on
their
scientific
performance,
in
order
to
find
explanations
of
the
bibliometric
results
in
terms
of
factors
relevant
to
policy.”
• Commercialization
Moedet
al.
(1985,
p.
131)

19. Bibliometrics
Research
evaluation
• Part
of
hiring,
promotion
and
funding decisions
• Dashboard
tools

20. Bibliometrics
Hvistendahl,
M.
(2013).
China’s
publication
bazaar.
Science,
342(6162),
1035-­‐1039.
van
Noorden,
R.
(2013).
Brazilian
citation
scheme
outed:
Thomson
Reuters
suspends
journals
from
its
rankings
for
‘citation
stacking’,
Nature,
500(7464),
510-­‐511.
Research
evaluation
• Oversimplification
• Publications
=
productivity
• Citations
=
impact
• Uninformed
use
and
misuse
• Impact
factor
• h-­‐index
• Adverse
effects
• “Salami”
publishing
• Honorary
authorship
• Self-­‐citations
• Citation
cartels

21. Scholarly metrics
Björneborn,
L.
&
Ingwersen,
P.
(2004),
Toward
a
basic
framework
for
webometrics.
Journal
of
the
American
Society
for
Information
Science
and
Technology,
55(14),
1216–1227.
Definitions
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics
adaptedfrom:
Björneborn&
Ingwersen(2004,
p.
1217)

22. Scholarly metrics
Otlet,
P.
(1934).
Traité
de
documentation:
le
livre
sur
le
livre,
théorie
et
pratique.
Pritchard,
P.
(1927).
Statistical bibliography or
bibliometrics?
Journal
of
Documentation,
25,
348-­‐349..
Bibliometrics
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics
“La
«Bibliometrie»
sera
la
partie
définie
de
la
Bibliologie
qui
s'occupe
de
la
mesure
ou
quantité
appliquée
aux
livres.”
“the
application
of
mathematics
and
statistical
methods
to
books
and
other
media
of
communication”
Pritchard
(1969,
p.
348)
Otlet
(1934,
p.
14)

23. Scholarly metrics
Altmetrics
adaptedfrom:
Björneborn&
Ingwersen(2004,
p.
1217)
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics altmetrics
Björneborn,
L.
&
Ingwersen,
P.
(2004),
Toward
a
basic
framework
for
webometrics.
Journal
of
the
American
Society
for
Information
Science
and
Technology,
55(14),
1216–1227.

24. Scholarly metrics
Priem,
J.
(2014).
Altmetrics.
In
B.
Cronin
&
C.
R.
Sugimoto (Eds.),
Beyond
bibliometrics:
harnessing multidimensional indicators of
performance
(pp.
263–287).
Cambridge,
MA:
MIT
Press.
Rousseau,
R.
&
Ye,
F.
(2013).
A
multi-­‐metric approach for
research evaluation.
Chinese Science
Bulletin,
3288–3290.
doi:10.1007/s11434-­‐013-­‐5939-­‐3
Altmetrics
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics altmetrics
“study
and
use
of
scholarly
impact
measures
based
on
activity
in
online
tools
and
environments”
“a
good
idea
but
a
bad
name”
Rousseau
&
Ye(2013,
p.
3289)
Priem(2014,
p.
266)

25. Scholarly metrics
Definition
altmetrics
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics
adaptedfrom:
Björneborn&
Ingwersen(2004,
p.
1217)
Scholarly
metrics
Björneborn,
L.
&
Ingwersen,
P.
(2004),
Toward
a
basic
framework
for
webometrics.
Journal
of
the
American
Society
for
Information
Science
and
Technology,
55(14),
1216–1227.

26. Scholarly metrics
Haustein,
S.,
(2016).
Grand
challenges
in
altmetrics:
heterogeneity,
data
quality
and
dependencies.
Scientometrics,
108(1),
413–423.
altmetrics
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics
Scholarly
metrics
Acts: viewing, reading,
saving, diffusing,
mentioning, citing,
reusing, modifying, etc.
Scholarly documents:
papers, books, blog posts,
datasets, code, etc.
Scholarly agents:
researchers, universities,
funders, journals, etc.
“[S]cholarly metrics
are
thus
defined
as
indicators
based
on
recorded
events
of
acts
[…]
related
to
scholarly
documents
[…]
or
scholarly
agents
[…].”
Haustein
(2016,
p.
348)

28. Altmetrics
Priem,
J.,
Taraborelli,
D.,
Groth,
P.,
&
Neylon,
C.
(2010).
Alt-­‐metrics:
a
manifesto.
October.
Retrieved from http://altmetrics.org/manifesto/
• Information
overload
“We
rely
on
filters
to
make
sense
of
the
scholarly
literature,
but
the
narrow,
traditional
filters
are
being
swamped.
However,
the
growth
of
new,
online
scholarly
tools
allows
us
to
make
new
filters;
these
altmetrics
reflect
the
broad,
rapid
impact
of
scholarship
in
this
burgeoning
ecosystem.”
• Criticism
against
current
form
of
research
evaluation
• Alternative
forms
of
research
output
• Alternative
use
and
visibility
of
publications
Priem et al. (2010)

29. Altmetrics
Coverage per
platform
Haustein,
S.,
Costas,
R.,
&
Larivière,
V.
(2015).
Characterizing
social
media
metrics
of
scholarly
papers:
The
effect
of
document
properties
and
collaboration
patterns.
PLoS ONE,
10(5),
e0127830.
doi:
10.1371/journal.pone.0120495
Zahedi,
Z.,
&
Haustein,
S.
(in
preparation).
Which
document
features
attract
users
in
Mendeley?
An
analysis
of
bibliographic
characteristics
of
Web
of
Science
publications
and
Mendeley
readership
counts.

30. Mathematics
&
Computer
Science
Natural
Sciences
&
Engineering
Life
&
Earth Sciences
Biomedical &
Health Sciences
Social
Sciences
&
Humanities
76,4
%
83,7
%
91,4
%
86,5
%
81,7
%
Mendeley
7,5
%
12,9
%
21,6
%
31,7
%
26,0
%
Twitter
Altmetrics
Coverage per
discipline
Haustein,
S.,
Costas,
R.,
&
Larivière,
V.
(2015).
Characterizing
social
media
metrics
of
scholarly
papers:
The
effect
of
document
properties
and
collaboration
patterns.
PLoS ONE,
10(5),
e0127830.
doi:
10.1371/journal.pone.0120495
Zahedi,
Z.,
&
Haustein,
S.
(in
preparation).
Which
document
features
attract
users
in
Mendeley?
An
analysis
of
bibliographic
characteristics
of
Web
of
Science
publications
and
Mendeley
readership
counts.

31. Altmetrics
Density and
intensity per
platform
Intensity
Haustein,
S.,
Costas,
R.,
&
Larivière,
V.
(2015).
Characterizing
social
media
metrics
of
scholarly
papers:
The
effect
of
document
properties
and
collaboration
patterns.
PLoS ONE,
10(5),
e0127830.
doi:
10.1371/journal.pone.0120495
Zahedi,
Z.,
&
Haustein,
S.
(in
preparation).
Which
document
features
attract
users
in
Mendeley?
An
analysis
of
bibliographic
characteristics
of
Web
of
Science
publications
and
Mendeley
readership
counts.

32. Altmetrics
Spearman
correlations with citations
Perfectnegativecorrelation
Perfectpositive
correlation
Haustein,
S.,
Larivière,
V.,
Thelwall,
M.,
Amyot,
D.,
&
Peters,
I.
(2014).
Tweets
vs.
Mendeley
readers:
How
do
these two social
media
metrics differ.
Information
Technology,
56(5),
207–215.
doi:
10.1515/itit-­‐2014-­‐1048
Haustein,
S.,
Costas,
R.,
&
Larivière,
V.
(2015).
Characterizing
social
media
metrics
of
scholarly
papers:
The
effect
of
document
properties
and
collaboration
patterns.
PLoS ONE,
10(5),
e0127830.
doi:
10.1371/journal.pone.0120495

33. Altmetrics
Spearman
correlations with citations
NSF
Subdiscipline General
Biomedical Research 2011
Size of data points represents number of Mendeley readers in Twitter graph (left) and number of tweets
in Mendeley graph (right).
Haustein,
S.,
Larivière,
V.,
Thelwall,
M.,
Amyot,
D.,
&
Peters,
I.
(2014).
Tweets
vs.
Mendeley
readers:
How
do
these two social
media
metrics differ.
Information
Technology,
56(5),
207–215.
doi:
10.1515/itit-­‐2014-­‐1048

34. Altmetrics
Document
types
Haustein,
S.,
Costas,
R.,
&
Larivière,
V.
(2015).
Characterizing
social
media
metrics
of
scholarly
papers:
The
effect
of
document
properties
and
collaboration
patterns.
PLoS ONE,
10(5),
e0127830.
doi:
10.1371/journal.pone.0120495

35. Altmetrics
Highly tweeted

36. Altmetrics
Highly tweeted

37. Altmetrics
Highly tweeted

38. Altmetrics
Highly tweeted

39. Altmetrics
Altmetrics
in
the
wild

40. Opportunities and
Challenges
• Heterogeneity
• Time
and
timing
• Audiences
and
user
groups
Altmetrics

41. Opportunities
• Different acts
• Diverse
motivations
Ø Diverse
impact
Challenges
• Understanding underlying processes
Ø Determining the
meaning of
metrics
Heterogeneity of
Altmetrics

42. Heterogeneity
Saving
to
Mendeley
Mentioning
in
News

43. Heterogeneity
Recommending
on
F1000
Tweeting

44. Heterogeneity
Bertin,
M.,
Atanassova,
I.,
Gingras,
Y.,
&
Larivière,
V.
(2015).
The
invariant
distribution
of
references
in
scientific
articles.
Journal
of
the
Association
for
Information
Science
and
Technology, 67(1),
164-­‐177.
doi:
10.1002/asi.23367
Distribution
of
referencesalongthe
IMRaDstructure
Citing in
a
journal
article

45. Heterogeneity
Acts referring to
research objects
Haustein,
S.,
Bowman,
T.
D.,
&
Costas,
R.
(2016).
Interpreting “altmetrics”:
Viewing acts on
social
media
through the
lens of
citation
and
social
theories.
Dans
C.
R.
Sugimoto (dir.),
Theories of
Informetrics and
Scholarly Communication
(p.
372–405).
Berlin:
De
Gruyter Mouton.
doi:
10.1515/9783110308464-­‐022
RESEARCH OBJECT

46. Time
and
Timing
Opportunities
• Detailed life
cycle
of
scientific output
Ø Fine-­‐grained indicators and
adequate benchmarks
Challenges
• Versions
of
research output
• Publication
dates

47. Time
and
Timing
Journal
article
• Submitted manuscript
• Revised manuscript
• Accepted manuscript
• Version
of
Record
• Online
publication
• Journal
issue
• Online
date
• Issue
month
Ø Adjusting indicators
3
March
2014
15
July
2014
21
January2015
February2015

48. Time
and
Timing

49. Time
and
Timing
Tweets
before publication?

50. Time
and
Timing
Weekday effects on
Twitter
weekday of
online publication:
based on:
8,765 Springer papers with
online publication date
19,010 tweets received within one year
of online publication date

51. Audiences
and
User
Groups
Opportunities
• Differentiating between types
of
users
• Measuring societal impact
Challenges
• Identifying users and
user
groups
• Determining engagement

52. Audiences
and
User
Groups
Alperin,
J.
P.
(2015).
Moving
beyond
counts:
A
method
for
surveying
Twitter
users.
In
altmetrics15:
5
years
in,
what
do
we
know?
Amsterdam,
The
Netherlands.
Retrieved
from:
http://altmetrics.org/wp-­‐content/uploads/2015/09/altmetrics15_paper_3.pdf
Tsou,
A.,
Bowman,
T.
D.,
Ghazinejad,
A.,
&
Sugimoto,
C.
R.
(2015).
Who tweets
about
science?
In
Proceedings of
the
2015
International
Society
for
Scientometrics and
Informetrics (pp.
95–100).
Istanbul,
Turkey.
Identifying Twitter
users
• Altmetric.com
classification
• Among
a
random
sample
of
2,000
accounts
tweeting
papers,
34%
of
individuals
identified
as
having
PhD
• Of
286
users
linking
to
SciELO articles,
24%
employed
at
university,
23%
students,
36%
not
university
affiliated
*based on
Altmetric.com
data
06/2015
(Tsou, Bowman, Ghazinejad, & Sugimoto, 2010)
(Alperin, 2015)

53. 1
2
3
Audiences
and
User
Groups
Haustein,
S.,
&
Costas,
R.,
(2015).
Identifying Twitter
audiences:
who is tweeting about
scientific papers?
Communication
présentée
au
SIG/MET
Workshop,
ASIS&T
2015
Annual Meeting,
7
novembre
2015,
Saint-­‐Louis,
MO
(USA).
topics and
collectives
academic
personal
Node size
number of
accounts
associated with term
Node color
cluster
affiliation
Terms
in
Twitter
bio

54. Audiences
and
User
Groups
Engagement
with scientific papers on
Twitter
Haustein,
S.,
Bowman,
T.
D.,
Holmberg,
K.,
Tsou,
A.,
Sugimoto,
C.
R.,
&
Larivière,
V.
(2016).
Tweets
as
impact
indicators:
Examining
the
implications
of
automated
bot
accounts
on
Twitter.
Journal
of
the
Association
for
Information
Science
and
Technology,
67(1),
232–238.
doi:
10.1002/asi.23456

55. Audiences
and
User
Groups
Twitter
arXiv bots

56. Audiences
and
User
Groups
Journal
accounts

58. Conclusions
• Scholarly communication
and
the
reward system
of
science
are
changing
• Potential to
become more
transparent
and
diverse
• Open
Science
• Scholarly metrics
• Fundamental difference between posting on
social
media
and
academic publishing
• More
metrics =
more
complexity

59. Outlook
More
scientometric studies.
More
sociological research.
Avoid adverse
effects.