Journal Editors on Peer Review, Paywalls, and Preprints

; Joseph Hill, MD, PhD; Brahmajee K. Nallamothu, MD, MPH


February 20, 2019

This transcript has been edited for clarity.

Robert A. Harrington, MD: Hi. I'm Bob Harrington from Stanford University, here at the American Heart Association (AHA) meetings in Chicago. Our readers and listenership on and Medscape Cardiology have shown great interest in how journal editors are thinking about such complicated issues as peer review, open access, preprints, and the free flow of information through social media. These topics are worth some discussion, and here at the AHA meetings, I'm really privileged to be joined by two colleagues who are editors of major journals in the Circulation family of journals.

Next to me is Dr Joe Hill, a professor of medicine at University of Texas Southwestern. Next to him is another friend and colleague, Dr Brahmajee Nallamothu, who is a professor of medicine and a cardiologist practicing at the University of Michigan.

Joe, you are the editor of Circulation. Brahmajee, you are the editor of Circulation: Cardiovascular Quality and Outcomes. Both of those are important journals.

Let's talk about the peer review question first. It's come up a lot recently. I often liken peer review to what Churchill said about democracy: "It's the worst form of government except for all the others." Joe, is peer review the best we've got?

Peer-Review Process

Joseph A. Hill, MD, PhD: It has not evolved much in recent years, but it is a service that people provide to the body of cardiovascular science and medicine and to our profession. Of course, we as editors are enormously grateful for that.

Some of our best reviewers are early career professionals...and some of our worst reviewers are big-name people.

We see good reviews, and we see sloppy reviews. We have a practice at our journal of tracking the reviewers who do a particularly good job and acknowledging them at our editorial board dinner and writing a letter to their boss letting them know that they do these.

Some of our best reviewers are early-career professionals, including fellows, and some of our worst reviewers are big-name people. You can't tell how someone will do based on the thickness of their curriculum vitae.

Harrington: Do you think that has to do with the time that people have to put into it, or do you think there is something else at play?

Hill: That is part of it. An early-career person who has the opportunity to review for one of our journals often takes it very seriously. They read around it and evaluate the novelty of the report, whereas other people who are busier or less engaged with the process zip through it pretty fast.

Harrington: Brahmajee, how do we prepare people to be peer reviewers? I know that you are working on some programs to pair junior and senior people. And should we pay peer reviewers? This topic has come up recently with a lot of buzz on social media.

Brahmajee Nallamothu, MD, MPH: Those are both great questions. Building on what Joe mentioned, it's difficult sometimes to get big names to review papers, even when it's in their own research space. We found a lot of enthusiasm among junior faculty as well as fellows to participate, but there is some learning. Our running joke is that when you get a fellow, the review is almost longer than the paper and that can be a lot to wade through. Being an author myself, you have to be cognizant about what is expected of the review process.

We started an assistant reviewer program. Many of our papers now go to typical peer reviewers within our system, and we tag on an assistant reviewer and this has been very successful. Lots of people have joined in. We have a process now where we "graduate" the assistant reviewers. I'm not sure some of them want to be graduated, because then they start getting a lot more papers.

[I]f we make it transactional, then the community feeling within science starts to dissipate a little bit.

Whenever you review a paper, I'm sure the first thing you do whenever you get the decision letter is see what the other reviewers thought. For new reviewers, to have a little bit of the peer-review process opened up to them at an early part in their career has been special. The more we can tap into this new generation, the better.

Your second question about paying reviewers is something that I've had mixed feelings about. We can never pay people the value of their time. It is an immensely valuable thing if Bob Harrington spends an hour to review a paper, but there is no way our journal can pay him for that time. In one sense, I really want to appreciate what people are doing for the journal, but I feel like if we make it transactional, then the community feeling within science starts to dissipate a little bit.

What we try to do is exactly what Joe said, which is recognize people in other ways. Borrowing off of Circulation, we send out a note to chairs and med school deans talking about the program. We recognize them at our annual meeting. I think those things mean a lot to folks more than $50 or $100 for a review.

Harrington: Over the years, I've decreased the number of journals that I review for, but I review for the Circulation journals, the Journal of the American College of Cardiology journals, New England Journal of Medicine, JAMA, et cetera. It adds up. I probably review a paper a week. Every review takes me probably 2 hours, and that is a lot of time. But I agree with you. If you are in the community of science, you need to give back.

Hill: For an early-career professional, one of the best ways for him or her to get their name out is to review for a journal and do a good job, because after you have done that two or three times, we notice. We have a team of 50 pretty impressive editors, and we notice that this early-career person is doing a great job, and that is a powerful thing.

Harrington: I completely agree with you. When my fellows or junior faculty ask me, "Should I review for this journal?" My answer is, "If it's one of the big journals, like a Circulation journal, absolutely. That is going to get you into the community of science."

Preprint Concept

Harrington: Let's go to the preprint idea. Part of the concept behind preprints is that it deals with the crowdsourcing of peer review. You put something out there that maybe is not quite ready, it gets molded in the open-access preprint space, and then it goes through the regular peer-review process.

I feel strongly that you risk doing harm by just throwing your study out there, because some people are more fastidious in their science than others.

Hill: Brahmajee and I wrote something about this a few months ago.[1] It has long since been established in the domains of physics, mathematics, economics, and so forth. If I come up with a new mathematical theorem and I throw it out for crowdsourcing evaluation, human lives are not at stake. Whereas we see papers every single day where the interpretation is either overblown or wrong and human lives are at stake.

I feel strongly that you risk doing harm by just throwing your study out there, because some people are more fastidious in their science than others. A notion about what blood pressure to target or what drug to use matters to human beings, and so I'm very hesitant in that space. It's very different from physics, math, and economics when you are talking about human lives.

Harrington: What about basic science or laboratory science? Could you see those benefiting from the preprint space?

Hill: Maybe. If you make a discovery that changes clinical practice, then people are in a hurry to move on it. Whereas in basic science, if we discover something new, it's not like the world will jump on it tomorrow necessarily. I think there is a little less pressure in that space. Certainly, there is less risk to folks.

Harrington: Brahmajee, the journal you edit has a lot of observational outcomes research in it, which by its nature of being nonrandomized has confounding and some challenges. Different methods might yield different results. How do you feel about the notion of preprints? Can we work out some of those methodologic differences by exposing it to a broader community first?

Nallamothu: Like Joe, I worry that people take these preprints at a different stage within the research process. As a community, if we start to move in that direction, we should think about them more like abstracts being presented here at AHA. Nobody thinks that any of this stuff is completed science.

Harrington: Most of it won't get published in the next few years.

Nallamothu: Absolutely. Preprints make sense in that aspect, but we have to teach the community that that is what it really is about.

We have tried experimenting with the idea of preprints. After Joe's publication and once in a while on Twitter, you will see all this social media saying, "Why don't we do this?" We still have not gotten one preprint. It's crazy, because I don't know whether it's a cultural standpoint, but the clinical science community especially just has not caught up with it. It's one of these things making a lot of noise, but I'm not seeing them come across my desk every day.

Harrington: In the clinical science space, as you have indicated, there has been a bit more hesitancy. There are certain papers for which it would not be a bad idea, such as a secondary analysis of some large data set that maybe the community can actually make better. I worry a little more about the big clinical trial where people might change what they do; maybe we should step that back and make sure that the peer review is strong.

Social Media

Harrington: Let's talk about a related topic, social media, and how you as journal editors are dealing with such questions as how to use social media to promote the journals. Both of you are doing that quite well. How are you dealing with the fact that people want information right now? If they know that the AHA is coming up and there is going be a late-breaking clinical trial and one of you guys is going to publish the concurrent paper, the community does not necessarily want to wait. Should we keep doing what we are doing, which is waiting? Or should we push stuff out faster?

Sometimes when papers get talked about... [w]e are just liking it and retweeting it, but nobody is reading the paper.

Nallamothu: It's something I've thought a lot about. I was not very active on social media until I took this role. Part of it was feeling convinced that you have to be active and engaged, and so that made me break out of my own mold. You can be on it and sometimes you take a step back and ask yourself, "What is the purpose?" Is it just a couple of hundred of us talking to ourselves? But I've actually been shocked at how widespread social media use is, especially across networks of junior folks. This is a part of their lives in a way that I don't think is necessarily true for us. Because of that, we have been super-excited.

We started a program largely because of a couple of concerns I had. Sometimes when papers get talked about, two phenomena can happen. One is that everybody is "liking" it—it feels like my daughter's Facebook page. There is a great term someone coined about it: astroturfing. We are just liking it and retweeting it, but nobody is reading the paper.

The second thing is that sometimes the discussion can veer way off in a negative way. I'm not going to shock anybody by saying that that is a bit of an issue about a lot of controversial topics and stuff right now. We started a group with Madeline Sterling from New York and Boback Ziaeian from UCLA, and we call them "CQO commentators." We want them to have independence from our group. They are not on our editorial board and they don't decide which papers get accepted, but we give them a preview. We send them the papers and we tell them to be critical and to tell us if we made a wrong decision.

Engage the authors and engage other people on social media who are interested in this topic. We have started it and are still finding our way along, but our goal is to have a deeper discussion around these papers and work.

Harrington: Joe, in Circulation, I've seen viewpoint papers[2] on whether Twitter use should become a core competency for cardiologists or whether all cardiologists should have a Twitter handle. You guys are definitely thinking about this.

With late-breaking clinical trials, I hope people don't think that a paper comes in and we hold onto it for 3 months before the AHA meeting.

Hill: We are quite a bit, and it's fair to say that as an editor-in-chief, my obligation is to bring content before people in the way they want to sample it. We estimate that probably only 10% of our readership actually touch a physical print journal. Some people want it as electronic tables of contents, a website, a Twitter feed, a Facebook post, or a podcast, and so we provide all of that. Different people at different stages of their lives want it in different ways.

With late-breaking clinical trials, I hope people don't think that a paper comes in and we hold onto it for 3 months before the AHA meeting. These things come in at the last minute, and we are hoofing it to process it. I would submit that if you conceived a clinical problem and wrote a grant and did the study and then at the very end, you allow 2 months for careful peer review, that pales in comparison to the years of effort that went into that. That last 2 months of making sure you get the message right is critical.

Harrington: As a clinical trialist who has done a lot of these large-scale trials, I worry about the increasing pressure to present at one of the three big cardiology meetings: European Society of Cardiology, AHA, American College of Cardiology. We sometimes unblind results a week or two ahead of the meeting, and the pressure is on to get it out there. On more than one occasion, I've said, "Maybe we should skip the meeting and think about the data we have just spent 5 years working on. Maybe we should present at another meeting, or just get it into the peer review."

Nallamothu: There is a great letter from the BMJ[3] years ago where someone talked about sending in a note to the editors and saying, "Can you slow-track my publication? We just spent 5 years working on this and getting grants. Be thoughtful, be really considerate of this."

Harrington: That absolutely takes place. We all review for late-breaking science, and you are given two notes: Are you available to review this, and can you review it within 48 hours? Then you need to put time aside to do it. But it's an important part of information flowing quicker, and the community wants it out there. At this meeting, we saw the cholesterol guidelines released and concurrently published, and the information is moving fast like nothing we have ever seen.

Hill: It was already translated into Chinese within 2 hours.

Harrington: It's incredible the way and the speed with which things move. And the commentary has been pretty good too, where people have said, "Have we encouraged too much liberal use of coronary calcium? Have we risk-stratified well enough? Are we being as aggressive as we should be?" These are good conversations that the community needs to have.

Paywalls and Data Sharing

Harrington: The final topic is a sensitive one. Both of your journals exist behind a paywall, and we just talked about the free flow of information. There are business realities around the journals, but both of you, though, want to share data. What is the balance of these two things? I'll start with you, Joe, since you are the overall editor-in-chief.

[A] lot of public money has gone into research, and then to restrict public access to the information just does not seem right.

Hill: There are several funding agencies in Europe that now will not allow research that they fund be published in a journal that has a hybrid mechanism where you can opt for open access or not. Recently the big gorilla in the room joined in, and that is the Wellcome Trust. That process is well down the road in Europe. My guess is that we will see that on this side of the Atlantic in the not-too-distant future.

Harrington: While I appreciate the business aspects of all of this, we want information flow and we want to get it to people. We just have to figure out how to do that.

Hill: Absolutely. The model that we do science for free and we evaluate it for free, and we give it to a publisher who then charges us to see it, is not likely to remain unchanged for very much longer.

Harrington: Yes, I agree with that comment.

Nallamothu: I would just echo what Joe says. We are all investigators, and then we are editors, and I still see myself as an investigator first and that is a core principle. I'm glad to see the sponsors starting to move in this direction. I think we all know it's going to happen. The question is when and in what way. Wellcome Trust even scoped out a plan for how this will happen. That is all good news. Like you said, a lot of public money has gone into research, and then to restrict public access to the information just does not seem right.

Harrington: That does not feel right, given that we definitely want people to have this available to make decisions, whether it's to do other types of science or to take care of patients. That is certainly why we do it.

Hill: We work at institutions where we can access hundreds of journals, but there are places in the world where you cannot.

Harrington: There is nothing more frustrating than to be on a plane and I don't have access to the Stanford library and I'm like, "I need that article—I can see it right there."

I want to thank both of you for taking time out of a busy AHA meeting to talk about some of these issues that have been going around in the social media space as well as at conferences around peer review, open access, and how information moves today in a social media age. Thanks to both of you. Keep up the terrific work. You have done great jobs with these journals, and we are looking forward to seeing more creative innovative things from you.

For more Cardiology, follow us on Twitter

Follow Medscape on Facebook, Twitter, Instagram, and YouTube


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.