Trends and Challenges in Measuring Performance of Reviewer Paper Assignment

Abstract

The remarkable increase in the number of authors and publications is observed in almost all fields in last decade. Inherently publication engages process of peer review and is a vital segment in research and development progression. The Reviewer Assignment Problem (RAP) is one of the most important and apparent process that involves the process of assigning a reviewer to paper. RAP has motivated many researchers in the recent decade as the conference organizers and journal editors are demanding for automatic assignment with fair and accurate reviews. Fair and accurate reviews mostly depend on accurate expert assignment to the paper. RAP entails two core phases: matching submitted manuscript and experts, and expert assignment to paper for review. Matching manuscript and expert involves building both paper and expert profiles correctly and then matching them accurately. Similarly assigning an expert to paper accurately involves solving assignment problem by satisfying constraints like a conflict of interest, fulfilling coverage requirements and balancing the load of the expert reviewer. Many researchers have contributed to the topic by proposing solutions for RAP. Each proposed methodology necessitates using an appropriate performance metric for proving its effectiveness. In the case of RAP, we can measure the performance of the whole process; one can measure the performance of the core phases separately as both phases engross different processes and different metric. Testing performance and reducing errors in phases will lead to overall improved results. Both subjective (also known as qualitative) and objective (also known as quantitative) measures have been practiced by the researchers. In this paper, we present trends in using performance measures, their applicability and suitability for RAP. It is observed that ‘Unanimity’ is the most appropriate measure for the overall review process. Use of performance measures- matching score, authority, coverage, and diversity are the most suitable for measuring the performance of phase one- matching paper and reviewer. It is noticed that load balance, fairness and accuracy are suitable for measuring the performance of phase two that assigns a reviewer to paper by satisfying constraints. The study derives the conclusion that hybrid approach with the proper combination of these measures is the best for measuring the performance of process of assigning reviewer to the paper.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 9 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.