Questions?
Please send any questions to: ml4h.2021@gmail.com
Goals for Reviewing Process
Similar to prior years, we will have two tracks at ML4H: a formal proceedings track for polished work with technical sophistication and clear healthcare impact as well as an extended abstract track for non-traditional research artifacts. Our goal in providing two tracks is to establish ML4H as a strong venue for publishing excellent ML4H work while also providing a forum for insightful discussion of creative and probing ideas.
There are two complementary goals of the reviewing process. One goal is to provide actionable, constructive, and respectful feedback to the authors that will help them improve the work, both through positive recognition of the paper’s strengths and targeted commentary on opportunities for improvement. The other goal is to help meta-reviewers and organizers make decisions such that the highest-impact work is identified and accepted.
This year, ML4H also features an explicit consensus-building discussion period. In this period, we expect all reviewers to engage with one another and their meta-reviewer to present a unified consensus (see below for specific instructions on how to do this in HotCRP). In this period, we expect reviewers to consider the author’s rebuttal, align with other reviewers on commonly identified strengths and weaknesses, and update their review in light of these discussions. The goal of this period is not to induce homogeneity, but rather to build consensus and provide a unified recommendation to the author. If you feel after this period that your beliefs about the quality of the work are not reflected by other reviewers, it is acceptable and encouraged to present these beliefs in your final review, provided you have made a good-faith effort to engage with other reviewers and have been open to other perspectives or arguments. Meta-reviewers are encouraged to seek consensus in this period, so you will be asked by your meta-reviewer to justify your stance if you disagree with the majority of reviewers, and failing to do so may result in your review not being featured as heavily in the meta-reviewers overall recommendation.
Timeline
Sep. 13th AoE: Submission deadline.
Sep. 16th: Reviews & Meta-reviews assigned. Reviewing period begins!
Oct. 6th AoE: Review deadline
Oct. 11th: Reviewer Discussion Period Starts; Author Response Period Starts
Oct. 15th AoE: Author Response Period Ends
Oct. 20th: Reviewer Discussion Period Ends
Oct. 27th: Final Decisions Released
How do I participate in the discussion period?
To interact with other reviewers or your meta-reviewer, please use the “Add comment” functionality in HotCRP. All comments added to your papers should have visibility set to “Hidden from authors” (or “Administrators only” if you would like to leave a private comment for the organizers, and not the paper’s reviewer committee). Authors will not be able to see comments during the discussion period regardless of visibility settings, but comments with “Visible to authors” may be exposed after the discussion process.
How can I write a good review?
There are numerous external resources that provide helpful advice on writing reviews. To get a sense of what a good review might look like, the 2017 ACL PC chairs blog contains some useful advice. To understand the difference between a negative and a constructive review, check out the review content section of the NeurIPS 2021 guidelines (for inspiration only; follow the ML4H reviewer guidelines for any specific instructions). To assess the healthcare-specific aspects of a paper/abstract, Nature has a small paragraph on how to assess the authors’ contributions.
What counts as a good ML4H Paper/Abstract?
All reviewers should refer to the guidelines we provided authors on how to write a good ML4H paper or extended abstract. The criteria for the proceedings and abstracts tracks are distinct -- while both tracks require high quality submissions, the abstract submissions should also be judged on their likelihood to lead to a good discussion at the workshop. Please refer to the writing guidelines document for exemplary submissions in each track.
When should a paper switch to the abstract track?
Works submitted to the Proceedings track can be considered for consideration to the extended abstract track. If you are reviewing a paper and think that a shortened 4-page version of the work would be strong enough to warrant consideration in the extended abstract track, make sure to select that option in the reviewer form on HotCRP.
Keep in mind that the primary goal of the extended abstract track is to generate productive and interesting discussion amongst conference attendees. Creative, insightful, and/or potentially divisive contributions (even if less fully developed or not performant) make great additions to the abstract track. As a result, an extended abstract may not necessarily meet the same level of technical sophistication as a paper, as long as it meets the criteria of generating productive / interesting discussions.
Confidentiality Guidelines
All aspects of the review process should be confidential. Do not discuss or distribute the submissions or reviews or use any ideas from the submissions in your own work until they are publicly available.
Conflict Guidelines
Please check ASAP that your assigned papers do not represent conflicts of interest. You should not recognize any paper you are reviewing as work done by someone you work closely with.
If you find a conflict, report it immediately via email: ml4h.2021@gmail.com
Anonymity Guidelines
Papers should be anonymous. If you find a major violation, please report it to the organizers:ml4h.2021@gmail.com.
Our process is double-blind. The authors do not know the reviewers' identities, and the reviewers do not know the authors' identities. Of course, no process is perfect: the reviewers might be able to guess the authors' identities based on the data set or the approaches used, or by technical reports posted on the internet. As a reviewer, we expect you will not actively attempt to discover the identity of the author. We also caution you against assuming that you've discovered an author's identity based on a data set or approach: multiple independent inventions are common, and different groups often share data sets.
Formatting Guidelines
All submissions should be formatted using the ML4H 2021 LaTeX template. Proceedings track papers should have their main content limited to 9 pages (including figures and text, but excluding references), and extended abstracts should be limited to 4 pages (excluding references and appendices).
ML4H 2021 template files
ML4H 2021 template on Overleaf
Please note that references and appendices can extend beyond the page limit. However, reviewers should not feel obligated to read any supplementary material. Your time is precious!
In general, we’d rather accept good work than nitpick about little formatting issues (especially because some clinicians may not be familiar with LaTeX). However, big issues (e.g. a 20 page journal-like paper) would be grounds for rejection.
Reviewing Quota
We’ve tried our best to grow a large pool of reviewers, so we expect that each reviewer will be assigned no more than 5 submissions. Please help us keep this number small by completing your assigned reviews on time -- and if you can complete them early, all the better!
Review Assignment Process
Authors labeled each paper with a set of topic areas, and each reviewer was asked to list their topic areas of expertise. These topic areas are used to assign reviewers to the most relevant papers possible.
FAQ
- Can PC members also be authors of papers or extended abstracts?
Yes. Though of course a PC member will not be assigned to review their own papers. That’s a conflict of interest.
- What if I don't have sufficient expertise to review a paper assigned to me?
While papers should provide adequate background that nearly all reviewers can provide a reasonable review, we understand that occasionally a paper may fall far outside your area of expertise. If this happens, please contact ml4h.2021@gmail.com immediately.