New Perspectives on Assessment How Altmetrics Measure Scholarly Impact

Introduction

Beyond Traditional Impact: What Can Altmetrics Do for You?

Altmetrics hold a lot of promise but it's also important not to get too carried away by hype. Euan will be looking at what altmetrics are good for right now, what they're not good for, and what they might be useful for in the future. We'll also take a look at some concrete use cases to help put things in context.

Altmetric was founded by Euan Adie in 2011 and grew out of the burgeoning altmetrics movement. Euan had previously worked on Postgenomic.com, an open source scientific blog aggregator founded in 2006.

Disciplinary Differences and Other Biases: Exploring Social Media Metrics in Scholarly Context

What are institutions and researchers looking for in metrics, and how are newer metrics useful to them? What are ways that these numbers can be presented to be most helpful for what the institutions want to do? How / why is an API useful to an institution? Haustein will explore the need for systematic analysis of discipline-specific difference of social media uptake and usage behavior and its effect on altmetrics counts including the possible need for normalization fit into precisely what researchers are looking for. 

Stefanie Haustein is a research analyst at Science-Metrix in Montreal, Canada. She is also pursuing her Post Doctoral studies at the Universite de Montreal. She is a visiting lecturer at Heinrich Heine University in Dusseldorf, Germany.

Scholarly Connections: Citations, Social Media, ORCID and Authorship Networks

For his first twelve years at Elsevier, Taylor worked in book publishing. For the last four years, he has worked in the research and development group, Elsevier Labs, where he has tackled such diverse areas as question and answering technology, hypothesis classification and extraction, co-authorship networks and research identity. As part of the last subject, he has been engaged with the cross-industry Open Researcher and Contributor ID (ORCID) project since its inception in various technical capacities.

Event Q&A With Our Speakers

Q: Could you recommend case studies or success stories for a government agency which uses altmetric data?

Stefanie Haustein: From my point of view, this is just starting, although I was surprised by the organizations that showed interest in using altmetrics at the PLoS ALM workshop in San Francisco this October. This is also where Adam Dinsmore presented on the case study done by the Wellcome trust, his slides can be accessed here: http://lanyrd.com/2013/alm13/scrdpm/.

Euan and other altmetrics aggregators (Jason Priem and Heather Piwowar at impactstory and Andrea Michalek and Mike Buschman at Plum) probably have more insights which government agencies already make use of altmetric data.

Q: How do you determine if tweets, blog posts, etc. are from scholars or the general public?  It doesn't seem scalable to evaluate on an individual basis.

SH: It is currently a problem to find out by which audience or in what manner a social media metric is created. We know that Mendeley has an academic audience because of it's nature as a scholarly reference manager and we also know that a large number of users are young researchers. However, we do not know what a reader count actually means: having read a paper, intending to read a paper, letting other think to have read the paper etc.

With Twitter it is even more problematic to determine the who, how and why, because Twitter is used by all types of audiences: the general public but also authors, teachers, students, librarians, journal editors, journalists, politicians etc. Altmetric.com is doing quite a good job in differentiating between "Members of the public", "Scientists", "Practicioners" and "Science communicators" based on keywords that appear in the Twitter user profile. However, even if someone identifies themself as being a "scientist", this does not necessarily mean that he or she discusses scientific articles in a scientific manner (see example on page 39 of my slides). It is subject to future research to determine the type of impact of various social media metrics based on content and context analysis.

Q. How are publishers/researchers changing their behavior to take advantage of altmetrics?

Mike Taylor: Taking advantage of is an interesting way of putting the question, there is an implication that publishers might respond to altmetric data in order to change their behaviour. I'm not aware that this is happening. Inside Elsevier there are a lot of people who are aware of altmetrics - we frequently have meetings with 30 or 40 people on the call to talk about the latest. For the most part people are just interested. We have used the Altmetric.com donut on Scopus for 18 months and some journals on Sciencedirect are doing similarly. Obviously we like to promote our content, and we have social marketing teams that do this work. Also we like to help our authors promote the visibility of their work too, and a number of our projects are aimed at making it easier for them to (for example) tweet about their work and for our readers to share / save articles.

There is a lot of research work needed to be done to make sense of the data and to develop user interfaces that reflect that sense, so although there is a clear view that this data is of interest in itself, the extent to which it is helpful in terms of guiding researchers to newly published influential work, or in terms of guiding publishers to understand what articles will be highly cited is unknown at present. That said: it will be known over the course of the next few years.

Q: Is the altmetrics movement considering the number of downloads and page views of "green" articles deposited in institutional repositories?

SH: Downloads and page views are sometimes counted as an altmetric and sometimes they are not. Download statisitcs have been there long before the altmetrics movement but except for a few open access journals, availability of global usage data has always been the main issue. So in the strict sense, download counts are not altmetrics. Access counts are, however, similar to altmetrics as they reflect impact in a "broader" sense than citations. Right now, it is still mainly a question of availability of data and identifiers. ImpactStory and Plum present views and downloads for gold open access journals if they are available, e.g. for PLoS.

Plum also collects views from ePrints and DSpace: http://www.plumanalytics.com/metrics.html. Stacy Konkiel wrote something about altmetrics and repositories here: http://www.asis.org/Bulletin/Apr-13/AprMay13_Konkiel_Scherer.html and frequently presents on how they implemented Altmetric data in the IU repository: https://scholarworks.iu.edu/dspace/bitstream/handle/2022/16980/2013_DLBB_DSpace%20altmetrics.pdf?sequence=1

There is also a group in Germany (Daniel Beucke, SUB Göttingen, http://dini.de/projekte/oa-statistik/english/) that gathers usage statistics and plans to integrate altmetrics into repositories as well. So I assume that on the one hand, the number of altmetrics sources will increase and on the other hand, for-profit publishers are hopefully becoming less reluctant to provide download statistics as more and more usage and social media data is accessible.