Methodology Review Process
Methodology Review Process
The methodology review process consists of four stages before the methodology is published on the Regen Registry Methodology Library:
Internal Review (4 weeks)
Expert Peer Review (approx. 3-6 months)
Public Comment (14 - 30 days)
Each stage is described below. The length of time to complete the expert peer review depends on availability and response time of expert peer reviewers as well as the amount of comments to address and number of review cycles needed for the methodology review lead to approve the methodology.
Informal Feedback from Regen Registry & Regen Network Community
In order to implement an effective and efficient review process we encourage methodology developers to submit high quality methodologies for review. As a methodology is being developed, the Regen Registry, Science Team, and other community members with expertise and knowledge of the methods you are proposing will provide informal feedback. This feedback will help ensure the methodology being submitted for the expert review process is well written and the methodology framework is robust.
Internal Review
The first formal step in the review process after a methodology developer submits a methodology to the Regen Registry, is an internal review performed by Regen Registry and Science Teams. The intent of the internal review is to ensure that the methodology adheres to the requirements outlined in this Handbook and that the quality of the methodology framework is sufficient to warrant review by expert peers. We will also verify that three or more suitable potential reviewers have been identified to provide an expert peer reviewer.
The Registry ad Science Teams will determine if it the methodology is complete and of suitable quality (language and content) for expert peer review. Regen Registry aims to complete the internal review in less than four weeks. If the methodology still needs work, it will be returned to methodology developers to inform them whether or not the methodology is potentially suitable to be hosted on the Regen Registry, and if it is not, provide feedback on how to improve it. Otherwise, the methodology review lead will approve the methodology to proceed to the Expert Peer Review stage.
Expert Peer Review
The expert peer review will focus on ensuring that the proposed methodology is scientifically robust and likely to produce the expected results using the methods outlined in the methodology. To accomplish this, we will enlist the help of a minimum of two peers from the expert community that have sufficient expertise to evaluate the proposed method. We may use the list of potential expert peer reviewers submitted with the methodology and will identify additional reviewers at the discretion of the methodology review lead. This selection process can take a few weeks to accomplish. More information about the process that reviewers will follow can be found in the Expert Peer Reviewer Guidelines section below. When the methodology review lead is satisfied that the methodology is acceptable the methodology developer will be notified and the final review phase, the Public Comment, can begin.
Handling a Review
All steps of the expert peer review will be managed by the methodology review lead. When the number of methodologies being submitted increases beyond the capacity that can be effectively handled by internal Regen Registry review leads, Regen Registry will solicit, through a public announcement, additional methodology review handlers. The additional handlers would work under the direction of the methodology review lead and be responsible for identifying and coordinating recommendations to the methodology review lead. When using additional review handlers, Regen Registry will aim to engage with individuals that represent a broad range of expertise so methodologies can be sent to handlers based on how familiar they are with the methods being used. All final decisions will be made by the methodology review lead.
Anonymity
We will be using a single-blind review process where the names of the methodology developers are known but the reviewers have the option of being named in the review. The default option is that reviewers remain anonymous. We are using this approach initially since it will make it easier to find reviewers.
Review
Once a reviewer accepts to review a methodology they will have three weeks to provide feedback. In some cases more time will be allotted to accommodate reviewer scheduling constraints.
When the methodology review lead receives comments from reviewers they will assess the proposed comments and a decision will be made to accept the methodology as is, request changes (minor or major), reject and resubmit the methodology, or reject the methodology. If necessary, the methodology review lead will request additional feedback to clarify reviewer feedback that is not clear or is contradictory. The decision along with feedback from the reviewers and the methodology review lead will then be sent to the methodology developer.
Revisions
If revisions are requested, the methodology developers will have a maximum of six weeks to address reviewer feedback and resubmit a revised methodology along with clear notification of changes that were made with explicit note of responses to expert peer reviewer feedback. When the methodology review lead receives a revised methodology, they will decide if it needs to go out for a second (or more) round of expert peer review. If it goes out for review again, an effort will be made to use the same reviewers (if they formerly accepted to review future versions) or new reviewers will need to be found. If new reviewers are used, they will be made aware this is a revised methodology and they will need to have access to the revisions and past comments.
Expert Peer Reviewer Guidelines
The task of a reviewer is to provide critical feedback to the methodology developer to help them improve the methodology. The reviewer should consider themselves an ally of the methodology developers.
The methodology review lead will contact potential reviewers until a minimum of two people commit to review the methodology. When a potential expert peer reviewer is contacted they will be offered a stipend between $100 and $300 to complete a review, depending on the complexity of the methodology, at the discretion of the methodology review lead. The methodology developer will have to pay for the expert peer review. The range of stipends might be adjusted as the peer review program matures. When contacted, potential expert peer reviewers will have one week to accept or decline the offer to review. If there is no response, a follow-up letter will be sent to the potential expert peer reviewer. If no response or a negative response is received, then a search for another expert peer reviewer will begin. If a potential expert peer reviewer declines to review they will be requested to provide one or more alternates.
If a review cannot be submitted within the three week time period, the expert peer reviewer should contact the methodology review lead as early as possible to request an extension. Extensions will be granted at the discretion of the methodology review lead.
The review should provide feedback on the validity of the methodology. Specifically it should address the following:
Is the methodology clearly written with adequate detail for implementation?
Is the underlying foundation of the methodology clear?
Is the methodology feasible?
Will the proposed methods achieve the results defined in the methodology?
Are the sampling and measurement protocols robust?
Are there any alternative or additional steps that should be considered?
Any additional questions noted by the methodology review lead
In addition to review comments, the expert peer reviewer should also indicate if they are willing to review a future version of the methodology and if they want to be named in the review.
Public Comment
After a methodology has successfully completed the expert peer review process it will advance to the final review stage, Public Comment. Comments about the methodology will be solicited from the general public for a period of 15 days. The public commenting period will be announced on Regen Registry Hylo and on Regen Network Development (RND) typical social media platforms (ie Twitter).
Methodologies open to Public Comment are posted on the Call for Public Comment page can be made through the Public Comment templates provided there. People submitting comments directly will have the option to remain anonymous (the default option) or they can request to have their name attached to the comments.
Comments should be constructive and developed with the intent of improving the methodology and in the spirit of an ally with the methodology developer. The points noted in the Expert Peer Reviewer Guidelines section can be used to guide the types of comments we are looking for.
After the 15 day Public Comment period, the methodology review lead will compile comments and synthesize them before sending them to the methodology developer. If there are major concerns that cannot be confidently resolved within Regen Registry, input from the expert community will be requested. Also, some clarification might be requested from the commenters when necessary by the methodology review lead.
The methodology developer will have six weeks to respond to the synthesized comments and return the revised methodology, along with responses to feedback, to the methodology review lead. The methodology review lead will review the revised methodology and will decide if the methodology is sufficient to be accepted into the Regen Registry Methodology Library, or if more work is required. If more work is required, the methodology review lead will request the methodology developer to make specific changes to the methodology.
Last updated