Plan

Defining the Goals of the Assessment

The overall goal of the assessment in Peru was to inform future public health surveillance strengthening efforts in the country, ensuring that such efforts are optimally designed and implemented to respond to the unique surveillance needs in Peru. This broad assessment identified major gaps in current surveillance and opportunities for strengthening.

Assessment Objectives:

1. Identify decision makers and stakeholders essential to implementing surveillance activities in Peru.
2. Describe existing system and operation of public health surveillance systems.
3. Assess surveillance system performances for priority pathogens in Peru.
4. Assess the quality and use of epidemiological information produced by the surveillance system.
5. Describe areas of further integration, growth and optimization of human, animal, and environmental data streams for public health action.
6. Assess which policies, regulations, and laws regulate public health surveillance in Peru to identify any facilitators or inhibitors to any recommendations for making changes in existing surveillance.

Key Assessment Questions Included:

1. What are the objectives and operation of the public health surveillance systems in Peru?
2. Do surveillance system participants receive training on case definitions and reporting requirements?
3. How does information get feedback to the different levels of the health system (e.g., national to facility level)?
4. Are health events and priority pathogens kept under surveillance with the current systems?
5. What are the successes and deficiencies of the surveillance system?
6. What is the quality of the epidemiological information produced?
7. Which elements of the surveillance systems can be enhanced to improve the quality of information?
8. How does surveillance both support and benefit stakeholders including decision makers?
9. How do surveillance results affect control and policy?
10. What are the levels of integration with other systems, including animals, and health information systems?

Logistics

To determine the extent of what the assessment could include, the Assessment Team aligned the budget and timeline, as well as reviewed the budget to consider how many people could be hired and how many sites the team could visit in person within the timeline.

Logistics Considerations:

1. Contractors needed to be hired to conduct data collection. The Assessment Team reviewed the objectives and geographic areas identified by MINSA to estimate the time needed to accomplish data collection. In the end, balancing timeline, budget, and logistics considerations, it was determined that six data collectors could be hired for a two-month period to collect data from six geographic locations with support from core staff.
2. Supplies needed to conduct the assessment for each data collector. The Assessment Team made sure that each data collector had access to a tablet for data entry, as well as access to pre-paid data bundles for transmitting collected data to the central data store via cellular networks.

Assessment Team

The team that was assembled for the assessment was comprised of faculty, staff, and students from UW and UPCH, including medical epidemiologists with disease surveillance experience, informatics experts, statisticians, legal experts with health policy experience, and individuals with expertise in data collection. The CDC-Peru team of epidemiologists and statisticians used the data analysis plan for conducting the secondary data analysis of the surveillance quality indicators for the priority pathogens. Through a cooperative agreement with CDC, the team had ongoing access to additional experts in disease surveillance and data science as needed.

Role	Tasks
Tool Design
Infectious Disease Experts (x2)	Design pathogen-specific questions, include specific COVID-19 questions
One Health Technical Expert	Align human and animal health questions, incorporate climate questions
Epidemiologist	Inform study design, questions, and methodology, and train data collectors in methods
Qualitative Researcher	Inform qualitative study design aspects
Data Scientists (x2)	Develop data analysis plan including dummy tables for the analysis and advise on data collection tools
Informatics Experts (x2)	Develop questions to assess health system architecture and infrastructure
Attorney	Conduct public health surveillance policy assessment
Assessment Implementation
Data Collectors (x5)	Collect data according to SOPs
Data Manager	Review data quality and storage
Team Lead	Coordinate the data collection teams and site visits, as well as collect data
CDC-Peru Epidemiologists and Statisticians	Lead the analysis of the secondary data for six priority pathogens
Data Analysis and Report Writing
Data Scientists (x3)	Lead data analysis in R
Qualitative Researchers (x2)	Conduct qualitative data analysis using grounded theory
One Health Specialist/ Infectious Disease Doctor	Assess One Health components and interpret zoonotic infection data

Planning Process

Literature Review

To start, the Assessment Team conducted a literature review of public health surveillance system assessments to determine what could be leveraged as a best practice for the Peru assessment. The review included surveillance reports from the Ministry of Health, published scientific reports related to surveillance in Peru, international assessments specific to Peru to perform country level assessments of compliance with International Health Regulations (IHR), Integrated Disease Surveillance Response (IDSR), WHO State Party Self-Assessment Annual Reporting (SPAR) Tool, WHO Animal Health and Performance for Veterinary Services (OIE-PVS) Assessment Tool, and the pilot Global Health Security Agenda (GHSA) Assessment. which includes metrics related to global health security. Because of the interconnectivity of animal, human, and environmental health including climate change, the Assessment Team decided to use a One Health framework to holistically assess the Peru surveillance systems ability to target health hazards that involve humans, animals, and their environment. This included assessing collaboration across institutions and disciplines operating in different sectors to implement comprehensive surveillance processes, including communication between sectors, for priority pathogens. Based on the literature review, the Assessment Team developed methods and a protocol for the surveillance system survey and secondary data analysis.

Human Subjects Review

The team submitted the protocol to institutional review boards for human subjects at the University of Washington (UW), University of Peru Cayetano Heredia (UPCH) and the Centers for Disease Control and Prevention (CDC). The protocol received a non-research determination for both UW and CDC and was deemed minimal risk by UPCH.

Stakeholder Engagement

Once the Assessment Team had approval from institutional review boards and the Peruvian National Center for Epidemiology, Prevention and Control of Diseases (CDC-Peru), they conducted preliminary stakeholder engagement, inviting a wider group of stakeholders to discuss the assessment. Stakeholders were identified by University of Peru Cayetano Heredia based on their role in public health surveillance in the government and aligned to Peru processes for program engagement. The team used a “collaborate” approach to stakeholder engagement, inviting key stakeholders in Peru to participate in the assessment process, beginning with planning for the assessment.  Stakeholders included representatives from the CDC-Peru, the Peruvian National Institute of Health (INS), the Ministry of Agriculture, the World Bank, Pan American Health Organization (PAHO), USAID, and heads of regional health departments.

A virtual sensitization meeting was held to explain the goal of the assessment, solicit feedback from stakeholders, discuss timeline, and determine additional approvals that were needed. With input from stakeholders, the methods were further modified to identify seven priority pathogens for the study specifically related to surveillance system performance, and to clarify the roles of some government partners.

Assessment Methods & Tool Development

The Assessment Team identified four components to assess in the study: legal and regulatory frameworks, surveillance system, surveillance data, and interoperability. Planning for each component is described below.

1. Legal and Regulatory Framework Assessment

To assess the legal frameworks, a legal consultant was hired from Peru to conduct a desk review of existing policies and legislative actions found through the gray and white literature, including guidelines and regulations that were published or approved by Peruvian government. The consultant was selected because of her long-term work with the Peruvian government laws and regulations related to the public health landscape. The consultant also conducted interviews with relevant stakeholders. Snowball sampling (when a study participant identifies or refers you to addition people to contact until saturation point) was used to follow-up any new leads that emerged from interviews or the desk review.

Presentations and resources from the Workshop on Public Health Surveillance Policy Evaluation to learn more about how to conduct a policy evaluation:

• Policy Workshop Agenda
• Policy Analysis – Key Questions
• Session 1  – Introduction
• Session 2 – Governance, Policy and Public Health Structures
• Session 3 – Policy Evaluation Framework
• Session 4 – Governmental Mandates
  • • Policy Description Instrument
• Session 5 – Policies and National Documents
  • • Search Tracking Tool Instrument
    • Example of a Search Tracking Tool
• Session 6 – Identification of Other Relevant Policies
• Session 7 – Advocacy Strategies
  • • Advocacy Strategy Resource
    • Policy Analysis for Managing the Legal Profession
    • Policy Analysis Table

2. Surveillance System Assessment

Mixed methods were used to develop a survey tool for in-person and virtual interviews based on the literature review. Following the One Health approach, the questionnaire included sections on surveillance for human and animal health, as well as sections on the physical environment and environmental health. Subsections on surveillance, outbreak investigation, analysis, reporting and interpretation, as well as communication and dissemination were included to explore details of the public health surveillance system in Peru. To detect other strengths and gaps in the system, additional cross-cutting subsections were explored, including subsections on policy, interoperability, training, and laboratory support.

Tool development

Once a comprehensive tool was drafted in Excel, the file was translated to Spanish.  The team chose KoboToolbox for data collection because it was open-source software that could capture audio recording for qualitative questions and be used offline. Because of the anticipated wide range of respondents coming from different levels of the government, agencies, and organizations, the tool was further modified to allow respondents to only answer questions relevant to their role. Qualitative responses were audio recorded through KoboToolbox. Quantitative questions had pre-populated dropdown selections or checkboxes and where there was a request to see a document, a photo was taken to capture the document in the tool. The draft tool was tested in Lima, Peru with five public health practitioners to further refine respondents’ roles to adapt the survey tool to the respondent group and address any other challenges experienced in using the tool during the pilot.

Site selection

Based on the budget and recommendations by the CDC-Peru, seven regions were purposefully selected for in-person interviews and site visits. Loreto (Iquitos health centers), Amazonas (Bagua), and Madre de Dios (Puerto Maldonado, Iñapari) were selected because of the significant amount of wild animals living near human communities; Junín (Jauja, Tarma, La Merced) was purposefully selected because it is located between the Andes and the jungle, its endemic status for yellow fever, and its role as a point of entry for people traveling to the capital, Lima; Pasco (Oxapampa) was added based on its proximity to Junín, making it easily accessible during the site visit; Piura (El Alamor, Lancones, Querocotillo) was included because it is a significant point of entry for the large number of Venezuelan migrants coming from Ecuador to Lima; finally, Lima was selected because it is the capital and one-third of Peru’s population is based there. All other regions were invited to participate for Zoom or phone interviews, with 14 regions opting to participate in virtual interviews. The distribution of interviews is shown in Figure 2.

• KoboToolbox compatible survey tool English | Spanish
• Clean final survey tool Spanish
• Manual for KoboToolbox tool use in English | Spanish 

3. Surveillance Data Assessment

Seven priority pathogens were identified to include a mix of zoonotic infections and conditions that had greater impact to population health. The Assessment Team used the U.S. CDC Updated Guidelines for Evaluating Public Health Surveillance Systems Attributes for the seven priority pathogens to assess system performance using datasets from CDC-Peru and COVID-19-specific and related open datasets (e.g., vital registration system). The attributes chosen for analysis included data quality, sensitivity, positive predictive value, representativeness, and timeliness. Their derivations are detailed in Table 1 (non-COVID-19 data) and Table 2 (COVID-19 data). Based on the data privacy of surveillance data, CDC-Peru conducted the secondary data analysis of the non-COVID-19 priority pathogens to maintain patient privacy. The Assessment Team developed the data analysis plan and oriented CDC-Peru to the indicators to conduct the analysis. Raw datasets were analyzed by CDC-Peru for all pathogens that were reported as part of the national system except COVID-19. CDC-Peru generated tables for the analysis based on the data analysis plan for those pathogens. Tables were aggregated at the district level for each disease assessed. See the report for additional information on data sources and analyses.

#	Indicator	Numerator and Denominator	Level
A1. Completeness
1a	Proportion of cases in a database with no missing required information	Numerator: Number of deaths with complete information on age, gender, location. Denominator: Total deaths recorded.	Year Region Disease
1b	Proportion of missing information required by field (age, gender, location)	Numerator: Number of deaths with complete information for each of the three fields Denominator: Total deaths recorded	Year Disease (for each field)
A2. Validity
2a	Proportion of coding errors within a dataset	Numerator: Number of deaths with the specific disease described as a cause of death in any of the  fields and a compatible ICD10. Denominator: Number of deaths with the specific disease described as a cause of death in any of the six fields	Year Region Disease
A3. Sensitivity
	Sensitivity of surveillance system (NotiWeb): proportion of cases in one data set found in another dataset	Outpatient Information Numerator: Number of confirmed cases in NotiWeb Denominator: Number of cases for the same condition registered in HIS	Year Region Disease
A4. Positive Predictive Value
4a	Proportion of cases reported through the surveillance system in case-based surveillance that were confirmed	Numerator: Number of confirmed cases of a condition Denominator: Number of confirmed plus discarded cases	Year Region Disease
A5. Timeliness
5a	Time delays between onset and report	Time to report: (Date reported) – (date onset) for a specific condition	Year Region Disease
A6. Representativeness
6a	Ratio of the number of districts with notifications in surveillance system to the number of districts with cases in HIS	Numerator: Number of districts with notification in NotiWeb Denominator: Number of districts with cases registered in HIS	Year Disease

#	Indicator Definition	Numerator and Denominator	Level
A1. Completeness
1a	Proportion of cases recorded in a database with no missing required information	Numerator: Total cases recorded with no missing information Denominator: Total cases recorded, including unknown and missing items	Year Region Sex
A2. Validity
2a	Proportion of inconsistencies and errors within a dataset (inconsistent dates or out of range)	Numerator: Total cases with dates out of the range of the pandemic (less than “March 2020” or greater than dataset date). Denominator: Total cases in the dataset	Sex Institution
A3. Concordance (Polymerase Chain Reaction (PCR) Test Results)
3a	Proportion of concordance in positive PCR tests between two different data sources	Numerator: Number of positive PCR results in PCR dataset Denominator: Number of positive PCR results in positive dataset	Year Region Sex
A4. Timeliness
4a	Main time delays	Delay to report: Date of report (call) – date of onset (date of symptoms) Delay to lab result: Date of lab result – date of onset of symptoms Delay to attention: Date of admission to hospital– date of onset of symptoms	Year Region Sex

Data Analysis Resources:

• Peru Secondary Data Analysis Plan (Seven Pathogens of Interest)
• Peru Secondary Data Analysis Plan (COVID-19)
  • COVID-19 Data Set Inventory
• Final Report with Additional Details about Analysis

4. Interoperability Assessment

Questions related to interoperability of systems were included in the desk review and incorporated into the surveillance system survey.

Previous: Background