IARPA is seeking information on methods for evaluating the quality of reasoning in analytic reports on real-life complex issues (e.g., political, social, economic, military, environmental). This RFI is issued for planning purposes only and does not constitute a formal solicitation for proposals or suggest the procurement of any material, data sets, etc.
Background and Scope
IARPA is seeking information on methods for evaluating the quality of reasoning in analytic reports on real-life complex issues (e.g., political, social, economic, military, environmental). This RFI is issued for planning purposes only and does not constitute a formal solicitation for proposals or suggest the procurement of any material, data sets, etc.
Background and Scope
IARPA is interested in techniques to measure how well analytic reports:
• Address all key pieces of evidence, both supporting and contrary;
• Properly evaluate the evidence; and
• Make appropriate and accurate inferences from the evidence.
IARPA is especially interested in how to measure the quality of reasoning for analytic reports on real-life, complex issues. IARPA is less interested in artificial problems for which all relevant information has been provided. IARPA is particularly interested in objective measures.
Measures of reasoning quality should:
• Be appropriate both for reports that make forecasts and for those that do not;
• Assess quality of reasoning in cases where ground truth is not known and may never be known; and
• Apply to a wide range of analytic subjects including political, social, economic, military, and environmental.
The purpose of this RFI is to identify innovative methods to measure the reasoning quality of analytic reports. Responses to this RFI should answer any or all of the following questions, providing relevant references to support your answers:
1. What methods exist for assessing the quality of reasoning in analytic reports? Are these methods domain-specific or do they have broader topical applicability?
2. If a reasoning quality evaluation method has been tested, what performance statistics have been collected? Are these in-sample or out-of-sample results?
3. If a measurement includes expert judgments, how are those judgments elicited? What is the inter-rater reliability for these judgments? How much time is required to train the judges on the process? How long does it take to evaluate a typical report?
4. If a method is automated, how was it trained? How often is the method retrained and/or reevaluated?
5. How does the method distinguish quality of reasoning from quality of writing?
Preparation Instructions to Respondents
IARPA requests that respondents submit responses to the above questions for use by the Government in formulating the evaluation methodology for a potential program. IARPA requests that submittals briefly and clearly address the questions above. This announcement contains all
of the information required to submit a response. No additional forms, kits, or other materials are needed. IARPA welcomes responses from all capable and qualified sources from within and outside of the U.S. Reponses must meet the following formatting requirements:
1. A one-page cover sheet that identifies the title, organization(s), respondent’s technical and administrative points of contact – including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating its association with IARPA-RFI-22-02;
2. A substantive, focused, one-half page executive summary;
3. Responses to the above questions. Responses should be limited to 1 page per question addressed. Responses should be in minimum 10-point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11-inch paper, with 0.5-inch margins;
4. A list of citations (any significant claims or reports of success must be accompanied by citations). There is no page limit for citations.
Submission Instructions to Respondents
Responses to this RFI are due no later than 5:00 p.m., Eastern Time, December 13, 2021. All submissions must be electronically submitted to as a PDF document. Inquiries to this RFI must be submitted to . Do not send questions with proprietary content. No telephone inquiries will be accepted.
Disclaimers and Important Notes
This is an RFI issued solely for information and planning purposes and does not constitute a solicitation or authority to enter into negotiations for a contract. Respondents are advised that IARPA is under no obligation to acknowledge receipt of the information or to provide feedback to respondents with respect to any information submitted under this RFI. Responses to this notice are not offers and cannot be accepted by the Government to form a binding contract. Respondents are solely responsible for all expenses associated with responding to this RFI. IARPA will not provide reimbursement for costs incurred in responding to this RFI. It is the respondent’s responsibility to ensure that the submitted material has been approved for public release by the information owner. The Government does not intend to award a contract on the basis of this RFI or to otherwise pay for the information solicited, nor is the Government obligated to issue a solicitation based on responses received. No proprietary and no classified concepts or information shall be included in the submittal. However, should a respondent wish to submit classified concepts or information, prior coordination must be made with the IARPA Chief of Security. Email the Primary Point of Contact with a request for coordination with the IARPA Chief of Security. Input on technical aspects of the responses may be solicited by IARPA from non-Government consultants/experts who are bound by appropriate non-disclosure requirements.
Background and Scope
IARPA is interested in techniques to measure how well analytic reports:
• Address all key pieces of evidence, both supporting and contrary;
• Properly evaluate the evidence; and
• Make appropriate and accurate inferences from the evidence.
IARPA is especially interested in how to measure the quality of reasoning for analytic reports on real-life, complex issues. IARPA is less interested in artificial problems for which all relevant information has been provided. IARPA is particularly interested in objective measures.
Measures of reasoning quality should:
• Be appropriate both for reports that make forecasts and for those that do not;
• Assess quality of reasoning in cases where ground truth is not known and may never be known; and
• Apply to a wide range of analytic subjects including political, social, economic, military, and environmental.
The purpose of this RFI is to identify innovative methods to measure the reasoning quality of analytic reports. Responses to this RFI should answer any or all of the following questions, providing relevant references to support your answers:
1. What methods exist for assessing the quality of reasoning in analytic reports? Are these methods domain-specific or do they have broader topical applicability?
2. If a reasoning quality evaluation method has been tested, what performance statistics have been collected? Are these in-sample or out-of-sample results?
3. If a measurement includes expert judgments, how are those judgments elicited? What is the inter-rater reliability for these judgments? How much time is required to train the judges on the process? How long does it take to evaluate a typical report?
4. If a method is automated, how was it trained? How often is the method retrained and/or reevaluated?
5. How does the method distinguish quality of reasoning from quality of writing?
Preparation Instructions to Respondents
IARPA requests that respondents submit responses to the above questions for use by the Government in formulating the evaluation methodology for a potential program. IARPA requests that submittals briefly and clearly address the questions above. This announcement contains all
of the information required to submit a response. No additional forms, kits, or other materials are needed. IARPA welcomes responses from all capable and qualified sources from within and outside of the U.S. Reponses must meet the following formatting requirements:
1. A one-page cover sheet that identifies the title, organization(s), respondent’s technical and administrative points of contact – including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating its association with IARPA-RFI-22-02;
2. A substantive, focused, one-half page executive summary;
3. Responses to the above questions. Responses should be limited to 1 page per question addressed. Responses should be in minimum 10-point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11-inch paper, with 0.5-inch margins;
4. A list of citations (any significant claims or reports of success must be accompanied by citations). There is no page limit for citations.
Submission Instructions to Respondents
Responses to this RFI are due no later than 5:00 p.m., Eastern Time, December 13, 2021. All submissions must be electronically submitted to as a PDF document. Inquiries to this RFI must be submitted to . Do not send questions with proprietary content. No telephone inquiries will be accepted.
Disclaimers and Important Notes
This is an RFI issued solely for information and planning purposes and does not constitute a solicitation or authority to enter into negotiations for a contract. Respondents are advised that IARPA is under no obligation to acknowledge receipt of the information or to provide feedback to respondents with respect to any information submitted under this RFI. Responses to this notice are not offers and cannot be accepted by the Government to form a binding contract. Respondents are solely responsible for all expenses associated with responding to this RFI. IARPA will not provide reimbursement for costs incurred in responding to this RFI. It is the respondent’s responsibility to ensure that the submitted material has been approved for public release by the information owner. The Government does not intend to award a contract on the basis of this RFI or to otherwise pay for the information solicited, nor is the Government obligated to issue a solicitation based on responses received. No proprietary and no classified concepts or information shall be included in the submittal. However, should a respondent wish to submit classified concepts or information, prior coordination must be made with the IARPA Chief of Security. Email the Primary Point of Contact with a request for coordination with the IARPA Chief of Security. Input on technical aspects of the responses may be solicited by IARPA from non-Government consultants/experts who are bound by appropriate non-disclosure requirements.
Contracting Office Address
Office of the Director of National Intelligence, Intelligence Advanced Research Projects Activity Washington, District of Columbia 20511 United States
Primary Point of Contact
Dr. Steven Rieber
Program Manager – Office of Analysis Research
Intelligence Advanced Research Projects Activity
Office of the Director of National Intelligence, Intelligence Advanced Research Projects Activity Washington, District of Columbia 20511 United States
Primary Point of Contact
Dr. Steven Rieber
Program Manager – Office of Analysis Research
Intelligence Advanced Research Projects Activity