This talk tries to unveil some of the problems inherent in the current knowledge creation, dissemination, and evaluation practices, also based on models and quantitative analyses of the effectiveness of peer review as gatekeeping/assessment method and of citations as measure of impact. The speaker will present the recent research and development threads aiming at making the knowledge generation and dissemination process efficient, and the evaluation process (more) fair and accurate. He will in particular present the models and tools being developed to this end, which are essentially based on applying to knowledge dissemination the lessons learned from open source development and the social web. The presentation will be interactive and discussion-oriented.

1. Reshaping Scientific Knowledge
Dissemination and Evaluation in the Age of
the Web
Maurizio Marchese, Aliaksandr Birukou, Fabio Casati
and the LiquidPub team
Department of Information Engineering and Computer Science - DISI
University of Trento, Italy
The Tuesday Conversation, January 12, 2010 – DG-INFSO-Brussels

2. Challenge: doing science in the 21st
• The Web has changed many fields:
▫ News (blogs, RSS feeds, ...)
▫ Music (p2p networks, iTunes, lastFM, …)
▫ Travel (Orbiz, Google maps,…)
▫ Photos (Flikr, …)
▫ …
• Has it changed also scientific knowledge
production and dissemination processes ?

3. Challenge: doing science in the 21st
• Yes ! But - so far - mainly
▫ new and faster dissemination access channels
▫ distributed working environment
▫ …
• Scientific knowledge processes are still based on the
traditional notion of “paper” publication and on peer
review as quality assessment method
▫ Philosophical Transactions of the Royal Society of London—
founded in 1665
▫ Journal des scavans— 1665
▫ Royal Society of Edinburgh’s Medical Essays and Observations,
- 1731, introduces peer review as we would recognize it today

4. We have a
dream
Capture the lessons learned and opportunities
provided by the Web and open source, agile
development to develop concepts, models,
metrics, and tools for an efficient (for
people), effective (for science), and
sustainable (for publishers and the
community) way of creating,
disseminating, evaluating, and
consuming scientific knowledge.
Understand what’s good for science, and make
it happen
project.liquidpub.org
•Publish and perish: why the current publication and
review model is killing research and wasting your money
(ACM Ubiquity 8(3), Feb 2007), and
•Liquid Publications: Scientific Publications meet the Web From. www.52en.com/img/dream_01.jpg
Fabio Casati, Fausto Giunchiglia and Maurizio Marchese.

5. 5
Objectives
Understand Improve
▫ Peer review and innovation ▫ Better ways to do the same
▫ Evaluation processes, and things
quality/impact of research ▫ Better ways to do different
(people, papers, projects) and new things
▫ Dissemination models and
overhead ▫ Principles
▫ Scientific communities ▫ Models
▫ IT services

7. PR: Initial Goals
• Understand how well peer
review works
• Metrics + Analysis
• Understand how to improve
the process
• Gatekeeping aspects (in/out)
• Quality improvements
“Manuscript Quality before and after
Peer Review and Editing at Annals of “Not everything that can be counted counts,
Internal Medicine” and not everything that counts can
Goodman S.N., Berlin J., Fletcher be counted.” -- Albert Einstein
S.W., Fletcher R.H.

8. Metric Dimensions
Quality
Divergence
Kendall Distance
Disagreement
Robustness
Effort vs. quality
Statistics Effort invariant alternatives
Unbiasing
Biases
Min. criteria
Fairness
Efficiency

9. Quality-related Metrics:
real vs. ideal
• Real peer review ranking vs. ideal ranking
▫ Ideal ?
Subjective vs. Objective
But each process could/should define approximate
indicators of quality like: citations, downloads,
community voting, success in a second phase,
publication, citations, patents…
• IF an approximate ideal ranking is available we can
measure the difference in various ways, e.g
▫ Kendall τ distance / Kendall τ rank correlation
▫ Divergence metric

10. Comparing rankings

11. Comparing rankings

12. Divergence Metric
When the second ranking is random, we have:
t
NDivρ i ρ a (t, n, C ) = ∑ pt (i)wi
i=0
©Ndiv(1,n, C) = n-1/n
inv. correlated
e.g. √
N-Divergence
⎛ 1⎞ ⎛ 0⎞
NDivρ i ρ a (1, n, C ) = p1 (0) + p1 (1)
⎝ 1⎠ ⎝ 1⎠ indipendent
n − 1 ⎛ 1⎞
=
n ⎝ 1⎠
correlated
....
Cit C tn−t t −i ©
pt (i) = −i
;wi =
√
n
Ct t Nt = 1/n Nt = n/n
Normalized t

13. prior vs. after discussion
ca. 74 (36%) contributions have been effected
NDiv(53, 206, 206) = 0, 36 by the discussion phase

14. Results:
peer review ranking vs. citation count
Div
Normalized t
14

16. Fairness
• Definition: A review process is fair if and only
of the acceptance of a contribution does not
depend on the particular set of PC members that
reviews it
• The key is in the assignment of a paper to
reviewers: a paper assignment is unfair if the
specific assignment influences (makes more
predictable) the fate of the paper.

17. Computed Normalized Rating Biases
C1 C2 C3 C4
top accepting 2,66 3,44 1,52 1,17
top rejecting -1,74 -2,78 -2,06 -1,17
> + |min bias| 13% 5% 9% 7%
< - |min bias| 12% 4% 8% 7%
C1 C2 C3 C4
Unbiasing effect (divergence) 13% 9% 11% 14%
Unbiasing effect (reviewers affected) 10 16 5 4

18. 18/13
Disagreement metric
• Through this metric we compute the similarity
between the marks given by the reviewers on the
same contribution.
• The rationale behind this metric is that in a
review process we expect some kind of
agreement between reviewers.

19. 19/13
Normalized Disagreement

20. Disagreement vs number of reviews

21. Estimation of the "optimized”
number of proposals per reviewer
• There are different groups of papers, for
example: bad, moderate and good papers
• We want to distribute the papers among
reviewers in a way such that each reviewer will
have statistically at least one paper from each
group
▫ In this way the reviewer will have a better view of
the overall quality of the papers

22. Estimation of the "optimized”
number of papers per reviewer

23. The road ahead
Real-Time accuracy estimation
Ranking vs. marking
Speed Ranking

24. 24
Principles
Models
IT Services

25. Principles and objectives
• Everything counts!! (not just papers, not just
“innovations”)
• Minimal dissemination overhead
• Early sharing
• Early feedback/interaction
• Find diversity
• Interestingness and sharing as measures of reputation
• No gatekeeping. Use the filtering power of the
community
▫ We are not necessarily right!!
Let the community select the principles and models

26. A large number of technologies are out there

27. Blogs
Wikis
Collaborative tagging and social bookmarking
Scientific Search Services
Journals with collaborative peer review processs
More Complex Systems

28. But how they can be effectively
used ?
• Let’s explore some dimensions of the issue
▫ Agile, Collaborative, Open Source “scientific”
processes LiquidBook
▫ New models for dissemination, sharing,
interactions, evaluation LiquidJournal

29. 29

30. Liquid Books Initial Definition

31. A Scenario: Text books

32. LiquidBook Structure

33. LiquidBook Lifeycycle

34. Liquid Book: State of the art and
what’s new
o WikiBooks: open‐content textbooks ‐ community for creating a free library of
educational textbooks that anyone can edit
o Differently from WikiBook you can have different roles in the community:
o Authors
o Contributors
o People who just rate, write comments/reviews
o We want to offer a (legal) framework to authors to easier collaboration
o Tailored material for different needs (classes, professionals). Several Personalized
Editions which stay up‐to‐date with the current state of the art.
o Multi‐Faceted Content (presentations, excercises are available too)
o Sharing and reusing of content among a trusted network of authors, who
guarantee the quality

35. Examples of collaboratively written
books
o How to Think Like a Computer Scientist series of
publications by Green Tea Press, where the same
core programming text has been adapted to several
different programming languages
o 97 Things Every Programmer Should Know example
of collaborative written book, with hundreds of
contributors
o Business Model Generation example of collaborative
written book, with 470 co‐authors and without a
publisher

36. 36

37. Journals today
• Based on traditional
notion of paper
• Traditional peer-review
• Solid in nature
• Established reputation

38. The Web Era
The Web has changed the way we get, share, produce and consume
scientific content
Internet
How do I get
How do I make
interesting
my work visible!
Authors content! Readers

39. Journals: revisited
• Original reasons for the current model are gone
• Back to the roots: How to provide interesting
content?
blogs papers datasets

40. Liquid journals: Proposal

41. Scientific contributions
• Different types, maturity and certification levels

42. Filling a LJ
• Editor decides what to put
liquid
Editor

43. Consuming a LJ
Readers

44. Demo at
http://project.liquidpub.org/research-areas/liquid-journal

45. Hi Alex!
Peer-review journal
Create Issue Update Now Latests activities
Journal Created
Home 11 Apr 9.00am
Peer review, an overview [none] Springer
Content
Issues Peter, Pablo, Pedro (0) LJ references
Stats The world of scientific publications has … (4) citations Latests subscribers
Settings Peer review, scientific publications,
[no subscribers yet]
Rethinking peer review [Carl’s LJ] Arxiv
Joe (1) LJ references
General Info Exploring new ideas for peer review in … (3) citations
Created on 11 Apr 9.00am
Owner Alex Peer review, open access
Subscribers 0
Anonymized review data [Liquidpub LJ] Arxiv
Editors Juan, Alejandro (1) LJ references
Review data from 10 conferences in …
(1) citations
Alex Peer review,
Joe
Settings
Issues
[no issues yet]
We are still learning from you.
Disable
Soon you’ll get our suggestions :)
Settings
No suggestions from your friends Disable

46. Liquid journals: Characteristics
• Filled Semi-automatically (query based)
• Multi-faceted content
• Separation of knowledge production from
publication
• Use the editing power of the community
• Subscribe to the editors you trust
• Use the wisdom of the community

47. Liquid journals: Benefits
• Everybody becomes a journal editor (selfishly)
• No gatekeeping, quality enforced by the
community (gatekeeping is the noise)
• You get things you want to read (or you want
your group to read)
• You get diversity
• You leverage the selection work of your friends
• Measure interestingness and quality by sharing
• Reduce (optimize) dissemination overhead and
encourage early sharing

48. 48
Related work
• Reference management tools (Mendeley/CiteULike
collections)
▫ A way to share set of papers among a group of peers
▫ LJ are not just manually edited collections
We provide automatic feed of new content using “liquidity” of
queries
Editor can organize contributions in issues, just like in
traditional journals
Flexible workflow (maintained by Gelee)
• Overlaid journals (RIOJA)
▫ Provide an interface on top of public (open access)
repositories
LJ does not only focus on open access articles, but includes also
articles from different digital libraries and preliminary ideas
from blogs
▫ Whole idea is even older – see (Smith 2000)

50. Engineer a system that
• Accesses heterogeneous src w/no API
• Is modular so that functions can be reused
▫ In a world where modularity had little success
• Facilitates the creation of arbitrary
dissemination and evaluation models (by non-
programmers)
• Provides commonly needed research services
• Implements liquid journals (agile – our main
macro-story)
• Simple and lightweight

51. Liquid Journals dashboard
Liquid journals API
Search Personalized
Sharing
computing tagging
Diversity Caching
Reputation
Cached metadata
Search by
topic Crawling
Access
Disambigua
tion

52. UI
A
Research services P
I Crawling
Cache mgmt
Karaku client‐code
Subscriptions/stream
Disambiguation
Personalized tagging
REST API Liquidity
Karaku Rendering
REST API Cached SKOs
ResMan
Adapters
REST
REST
API
API
REST
REST
API
API
Fabio Casati - ECOWS 2009
52

53. [Insert header here] [Insert footer]

54. Vertical services
Liquid journals Liquid books …
Collaborative
programming
Gelee
Search computing
Mashart
Community
TopicSearch Reseval
discovery

55. [Insert footer] [Insert header
here]
Gelee
Paper at ICDE 09. Video avail online and demo at icsoc

56. Mashart
Paper at ER’09, demo at icsoc ‘09

57. project.liquidpu
b.org
RESEVAL http://demo.liquidpub.org/reseval/ complex’09

58. Group Management

59. Overview of the Community Network
SOFTWARE ENGINEERING(kbse,icse)
DIST. SYSTEM/COMPILER(ipps,iccS) APPLIED COMPUTING/CRYPTO(sac,compsac)
TELETEACHING/HUM_INT (chi,hicss)
TELECOM (icc,globecom)
HUMMAN –COMP INTER(icchp,hci)
GENETIC AND EVO ALG(cec,gecco)
AI/DB (icai,aaai)
ROBOTIC/M.MEDIA (icra,icpr)

60. Take-Home Message
• Flaws of current practices (or, lack of evidence that
they work as expected)
• Research services for novel dissemination model
• Principles, models, composable IT services in a
restricted domain
• Use cases:
▫ LiquidBook: Sharing and reusing of content among a
trusted network of authors, who guarantee the quality
▫ LiquidJournals: Interestingness, reward innovation,
sharing, everything contributes, use the filtering power
of the community

61. Collabiration with ICST.org
• The Institute for Computer Sciences, Social-
Informatics and Telecommunications Engineering:
an international society, focuses on ICT in its
broadest sense
• Supports research, innovation and technology
transfer in IT
• Collaboration on:
▫ LiquidJournals will be fed with data from ICST’s
eScripts (electronic journals) and distributed via
PeerNet (social network)
▫ Courseware platform – integration with LiquidBooks
▫ Review analysis – using Assyst (conf management
system) as a data source

62. Collaboration with other projects
http://project.liquidpub.org/collaboration/collaboration

63. 63
To know more
• Google liquidpub-announce and subscribe if you
are interested (VERY low traffic)
• First release at the end of the month

64. Thank you