How to manage peer review for academic conferences

Start from records, not from a mailbox

If abstracts live in a form tool and comments live in email, the chair cannot answer a simple question: who saw this paper, who recused, and what the author was told. A defensible workflow keeps assignment, scores, committee notes and the decision letter on the same abstract record that intake already created.

Conflicts before scoring

Co-authors, recent collaborators and same-institution rules should block assignment, not appear as a footnote after a score exists. Conference committees also have custom conflicts: a topic chair who proposed the symposium should not quietly review it. Declare those rules while the call is still open, not the weekend after the deadline.

Split what authors see from what the committee sees

Author-facing decision text and confidential remarks are different objects. Mixing them in a spreadsheet column is how leaked comments happen. Configure visibility by role: reviewer, topic lead, programme chair, secretariat.

Decisions have to become sessions

Accept, reject, poster-instead-of-talk, or invited revision only matter if the programme builder can consume the outcome. A review tool that exports a CSV into last year’s timetable is still two products. That join is the reason Eventumbot treats peer review as part of conference operations rather than as a standalone judging app.

Leave an audit trail the next chair can inherit

Societies rotate officers. The incoming chair should see who declined, who was late, and which papers were discussed in committee, without a handover novel. If your current process cannot survive a new volunteer, it is not a process yet.

The workflow in seven steps

Written down, the process is short. Every step should have an owner and a date.

• The call closes and the chair freezes the list of abstracts.
• Conflicts are declared and checked.
• Reviewers are assigned by topic and workload.
• Reviewers score against a rubric and write a comment for the author.
• The committee resolves borderline cases.
• Decisions are issued and letters sent.
• Accepted abstracts move into the programme.

Worked example: assigning reviewers

Take a meeting with 300 abstracts and 40 reviewers. At three reviews per abstract you need 900 reviews, or 22 to 23 per reviewer. That is heavy, so cap the load, recruit a reserve, and assign by topic first. Keep five to ten reviewers unassigned until the first withdrawals arrive, and reassign quickly when someone declines.

A simple decision matrix

Agree the rules before scores arrive, so borderline cases are handled the same way.

• High average score, reviewers agree: accept, as oral or poster according to preference and capacity.
• High average score, reviewers disagree: the track lead reads the reviews and decides.
• Middle score: accept as a poster where capacity allows.
• Low score, reviewers agree: reject with the comments written for the author.
• Any unresolved conflict: the chair reassigns before a decision is made.

Before the review period opens

Use the checklist below, then read our abstract review rubric and reviewer email templates for the documents you will need. The form fields collected at intake, described in our abstract submission form guide, determine how well reviewers can be matched.

• Rubric agreed and shared with reviewers.
• Conflict policy published and declarations collected.
• Reviewer invitations sent and accepted.
• Workload cap and reserve list decided.
• Visibility rules set for reviewers, track leads and secretariat.
• Decision matrix and letter templates ready.

Choosing a review model

The model decides who can see whose identity, and it shapes both fairness and workload. Choose it before the call opens and state it in the announcement.

• Single-blind: reviewers see the authors, authors do not see reviewers. Simple, and lets reviewers spot conflicts, but can favour well-known names.
• Double-blind: neither side sees the other. Reduces prestige bias, but authors must anonymise their abstract and conflicts are harder to detect.
• Open review: identities are known to both. Encourages courtesy and accountability, but some reviewers will hold back.

Reviewer workload and fairness

A reviewer who receives fifty abstracts will skim them. Cap the load, and spread it by expertise so that nobody is asked to judge work far from their field. Keep a record of how many abstracts each reviewer holds and how many are complete, and rebalance at the midpoint of the review period. Where a small number of reviewers cover a narrow topic, tell them in advance and thank them; they are carrying the meeting.

Talking to reviewers during the period

Reviewers need three messages: the instructions when the period opens, a reminder at the midpoint with the number of abstracts still open, and a final reminder two days before the deadline. Keep each short, link to the instructions and give a named contact. When someone reports a problem, such as an abstract that will not open or a conflict, reply within a day. Reviewers are volunteers, and speed of response is part of how you treat them.

A committee meeting agenda for borderline abstracts

Not every abstract needs discussion. Prepare a list of the borderline ones only, and use a short agenda so that the meeting finishes on time.

1. Confirm the decision rules (5 min) 2. Abstracts with conflicting reviews, by track (30 min) For each: the two reviewers’ comments are read; the track lead proposes; the committee decides 3. Abstracts near the threshold for posters (15 min) 4. Capacity check: number of oral slots and poster boards available (10 min) 5. Confirm decisions and the date letters will be sent (5 min)

Keep a record you could show to an author

Sooner or later an author will ask why their abstract was declined. Keep the scores, the comments and the decision reason together with the abstract. A one-line note such as “Below threshold on soundness; reviewers agreed” is enough. If a decision changed after committee discussion, record why. The aim is not to defend every decision, but to be able to explain each one, and to show that the process was applied evenly.

Common failure modes

These are the problems that damage a review process most often.

• Reviewers assigned before conflicts were checked.
• A rubric shared only after some reviewers had started.
• No reserve of reviewers, so declines leave abstracts with one review.
• Decision letters that repeat the confidential comments.
• Late changes to the acceptance threshold without a record.
• No one responsible for chasing reviewers who go silent.

Measuring how the process worked

After the decisions, look at a few numbers: the share of reviews completed on time, the time from the submission deadline to the decision letters, the number of abstracts reassigned, and the spread of scores between reviewers. Write them down and compare them with the next edition. Also ask reviewers, in a short survey, whether the workload and instructions were reasonable. The numbers show where the process was slow; the comments show why.