Which Aspects of a Surveillance System Can Be Assessed?

There are five main aspects of a surveillance system that can be evaluated separately or simultaneously depending on the goals of your assessment:

• System Design and Characterization: Models and approaches describing the overarching goals and characteristics of a system including equity, how modern the data system is, whether it includes more holistic surveillance approaches like One Health, and how compliant the system is to government regulations.
• System Regulatory Environment: The policies and legal frameworks which are relevant to disease surveillance; social and political context.
• System Resources: Funding, adequately trained workforce, infrastructure/equipment to support the surveillance system.
• System Processes and Data Governance: The management of data assets; the processes around data, including data collection, storage, cleaning, analysis, reporting, and flow; tools and procedures; and the roles and responsibilities around data, such as the institutional capacity to implement processes.
• System Efficacy and Performance: Data quality, coverage, system performance attributes, impact, and use.

System Design and Characterization

There are five cross-cutting principles and concepts that help define and characterize a well-functioning system as a public service that were identified by the team. These principles conceptualize surveillance systems as public goods and serve as frameworks to understand and address problems of effectiveness, access, and equity. Several of these frameworks are related, for instance data interoperability is difficult to achieve without modernization, and One Health surveillance requires interoperability.

1. The “One Health” approach to disease surveillance is a holistic and collaborative framework that recognizes the interconnectedness of human health, animal health, and the environment. It emphasizes the interdependence of these three domains and advocates for integrated surveillance systems that monitor and address health threats at the human-animal-environment interface. By considering the complex interactions between humans, animals (both domestic and wild), and their shared environments, the One Health approach aims to prevent, detect, and respond to disease outbreaks more effectively. It involves cooperation among medical professionals, veterinarians, ecologists, environmental scientists, and other relevant stakeholders to gather and analyze data from diverse sources. This integrated approach enables early detection of zoonotic diseases (diseases that can spread between animals and humans), enhances understanding of disease emergence and transmission dynamics, and supports the development of comprehensive strategies to safeguard both human and animal populations while promoting environmental health.
2. Data Modernization refers to the comprehensive process of maintenance, updating, and upgrading data systems, technologies, and practices to meet current and future needs. This involves transitioning from traditional, often paper-based methods to digital, automated, and technologically advanced approaches for collecting, storing, managing, analyzing, and sharing health-related data. Data modernization aims to enhance the efficiency, accuracy, and accessibility of surveillance information, enabling real-time monitoring, timely reporting, and informed decision-making. It often involves implementing standardized data formats, utilizing advanced analytics, ensuring data security and privacy compliance, and adopting cloud-based solutions. By embracing data modernization, public health agencies can improve their capacity to detect outbreaks, track disease trends, and respond effectively to health emergencies, ultimately contributing to better overall population health outcomes.
3. Data Interoperability is often a part of data modernization but focuses on the ability of different information systems and databases to seamlessly exchange, share, and make use of data in a standardized and efficient manner. It involves designing and implementing systems in such a way that they can communicate, understand, and use data from various sources, regardless of differences in data formats, structures, or platforms. Interoperability enables health authorities, researchers, and organizations to integrate data from diverse sources, such as laboratories, hospitals, clinics, and public health agencies, into a cohesive and comprehensive picture. This holistic view of data supports better decision-making, timely detection of outbreaks, and effective responses to health threats by enabling the aggregation and analysis of information across multiple sources. The integration of surveillance data from multiple organizations is particularly important in implementing a One Health approach to surveillance as part of efforts to strengthen prevention of epidemics and pandemics.
4. Regulatory compliance with international requirements is an important part of contributing to global health security. International Health Regulations (IHR) disease reporting requirements refers to the adherence of countries and their respective health authorities to the guidelines and obligations set forth by the World Health Organization (WHO) under the These regulations are a legally binding international framework designed to prevent, detect, and respond to the international spread of diseases, particularly those that pose a public health threat. Compliance involves timely and accurate reporting of specified diseases and events of international concern to the WHO, as well as the implementation of necessary measures to mitigate the spread of such diseases. By ensuring compliance with IHR reporting requirements, countries facilitate international cooperation in responding to public health emergencies. In addition to IHR, there may be additional national and regional regulations governing disease surveillance.
5. Equity should be considered at each step in a surveillance assessment from design to dissemination. This is critical to evaluating whether the right populations are included in the surveillance system. An example of how a gap can arise in a surveillance system when equity is not taken into consideration would be surveillance of maternal mortality that only captures deaths that occur in healthcare facilities. In this example, data may be missing on homebirths, which may be the population that needs support the most. When working with data collected from Indigenous communities, Indigenous Data Sovereignty should be centered on those communities.

Questions to ask around equity include:

• • Who is the intended audience of this assessment?
  • Who is deciding which surveillance criteria to assess?
  • Which populations are included in this system?
    • Is the surveillance system tracking all intended disease outcomes and any unintended consequences?
    • Is this system capturing data from all levels of care? (including community-level)
    • Are there differences in data quality, capacity or responsiveness among different populations, regions, level of service, or between community and facility surveillance systems?
    • Are recommendations from the assessment shared with all end users appropriate to their context, computer access, languages spoken, and literacy level?

System Regulatory Environment

Regulatory policies and procedures are critical to surveillance. Policies often define how, where, when, and who conducts surveillance activities. Additionally, surveillance happens across different ministries and sectors contributing to fragmented laws and regulations affecting surveillance activities.

Factors external to the surveillance system can heavily influence system implementation and impact including policy. Understanding the context of the surveillance system to be assessed is a critical step in determining why a system is or isn’t performing at the desired level and what changes need to occur outside of the system to improve performance. Policy dictates data sharing, data governance and data use to shape public health action.

Indicators for targeting system regulatory environment as an aspect of your assessment include:

• There is a suggestion of external barriers affecting system performance
• There has been no previous context assessment
• During system evaluation, it is found that identified data are not being collected as intended
• You would like to determine if the surveillance system design is appropriate for the context

Questions to Consider

In order to assess the regulatory environment of the system, consider breaking down policies and legal frameworks into questions:

• What government structures, actors and institutions have a role or mandate in public health surveillance?
• What policies, laws, and regulations affect public health surveillance in your country?
• What overlaps, gaps, or enablers exist in policies related to public health surveillance?

Common Approaches

• Desk review
• Business Process Analysis

System Resources

A surveillance system cannot meet target performance goals with inadequate funding, workforce, or infrastructure. Characterizing these resources is critical to understanding why some systems out-perform others, certain attributes or features are not met or why data governance is problematic and to identifying opportunities to strengthen the surveillance system. This is particularly important when data are not being collected as intended even after external factors have been addressed. Key resources include:

1. Human resources: There must be adequate personnel with sufficient education, training, and expertise, who have sufficient dedicated time to engage with surveillance system activities during work. If there are local standards for adequate human resources for surveillance system management, use those. Access to professional development opportunities and continuing education to new and changing technologies and approaches is also part of human resources development.
2. Physical infrastructure: Physical infrastructure includes reliable computer hardware, electricity, internet service, and office space, as well as an environment that provides safe working conditions and fosters necessary collaborative relationships. For paper-based systems, physical infrastructure includes record storage and management.
3. Information technology (IT): IT infrastructure includes the software and information services necessary to conduct surveillance activities, the specific configuration of those information services for the context of the surveillance system, the computers, mobile devices, and servers that host and serve this software, and the data that has been collected and generated that is necessary for pursuing public health goals and outcomes. For its continual operation, IT resources also depend on physical infrastructure, such as reliable internet service, and human resources (e.g., database management, software maintenance, and IT support personnel).

Common Approaches

• In-depth interviews
• Desk review
• Time use study or working with an economist

System Processes and Data Governance

Assessment of data governance is key to characterizing a surveillance system. Data governance includes the management of data assets and the processes, roles, institutional responsibilities, and polices. This governance applies to data throughout its lifecycle, from collection or creation, transmission, or storage, use or analysis, and eventual archiving or destruction. These practices aim to ensure the security, privacy, availability, reliability, integrity, and accuracy of data, and that its lifecycle complies with regulatory demands. Conducting a data governance assessment begins by understanding and accounting for data assets.

1. Identifying and Cataloging Data Assets: This involves producing a detailed inventory of the data assets in an organization relevant to public health surveillance. The goal of this cataloging exercise is to understand how the data are involved in the surveillance process, how the data are structured, the provenance and contents of the data, and their purpose to the processes and goals of the organization. This includes identifying data that may be consumed as inputs, such as case reports, medical encounters, laboratory tests, mortality data, news articles or vaccination records, or produced as outputs, such as internal dashboards or external reports.
2. Describing Data Ownership and Roles: As described in the data catalogue, data assets have a specific purpose in specific surveillance processes, and these purposes intersect with the processes and practices of professionals in the relevant organization. For each data asset, the assessment should describe who interacts with the asset and their role(s) as the data flows through the public health surveillance process. A person’s relationship with data can vary. They can be users of that data, producers, stewards, or owners, and may fulfill multiple roles for the same or different data asset.
3. Understanding Processes Around Data: Data is not static but is an integral part of the processes of an organization. These processes—including human processes like the actions of users in accordance with their roles and automated processes—transform and generate data while also being informed by data in-turn. Mapping and understanding these processes is a first step towards identifying potential inefficiencies that might need to be addressed. Some examples of potential inefficiencies might include issues in an automated data handling chain (e.g., inability to handle unclean data), logistical challenges that prevent data collection (e.g., lack of reagents in a laboratory), human processes that bottleneck data handling (e.g., manual entry of paper forms by a healthcare worker), and poorly integrated systems (e.g., manual data transfer between a laboratory and epidemiological data systems).
4. Identifying Potential Information Security Risks: As data flows through organizational processes, it is exposed to potential access, modification, or deletion by unauthorized users. These vulnerabilities can occur at different places throughout the dataflow map, such as each time the data are available for access by users, transmitted between systems, or stored in place. An information security risk assessment should be conducted to enable the organization to prioritize vulnerabilities to address. This judgment must consider how impactful and sensitive a data asset is to exploitation, the likelihood or ease in which a vulnerability can be exploited, and the cost of implementing controls to address or reduce these vulnerabilities.
5. Assessing Data Quality: An important part of a data governance process is to be able to evaluate and manage the quality of data assets. In this context, the term data quality is used more broadly than it is in the CDC attributes. Data quality can be defined across several dimensions or characteristics, such as the accuracy, completeness, integrity, timeliness, and accessibility of the data. A multitude of techniques and metrics are available for evaluating data quality across these dimensions. An assessment of data quality for public health surveillance should consider the importance and relevance of specific characteristics based upon on the purpose of the data asset.

Common Approaches

• Business Process Analysis (BPA) is a high-level process for reviewing operations and procedures relevant to the functioning of an organization, analyzing them for opportunities for improvement, and managing and evaluating changes to these processes. BPA functions best when scoped to a review of a specific process with the aim of making identifiable and measurable improvements.
• Interoperability assessment is an assessment focused on the ability of two or more organizations or systems to communicate and transfer data, make meaningful use of data across organization boundaries, and address legal, process, technical, and other barriers to interoperability. There are several interoperability assessment frameworks.

System Efficacy and Performance

While certain aspects of a surveillance system assessment should be tailored to surveillance system type, there are key attributes that should be part of all surveillance system assessments. Attributes are an element of system performance that reflect system design (environment, resources, and characteristics) and system function (processes). If data quality are poor or not representative of the target population, system performance will be inadequate.

The US Centers for Disease Control and Prevention (CDC) has identified 9 attributes of surveillance systems that should be assessed to gauge overall performance including simplicity, flexibility, data quality, acceptability, sensitivity, positive predictive value, representativeness, timeliness, and stability/sustainability.

• Simplicity highlights the importance of user-friendly systems that facilitate data collection and interpretation.
• Flexibility reflects the ability of the system to adapt easily to changing demands for information and operating conditions in the face of changing circumstances and emerging health threats.
• Data quality emphasizes accurate, complete, and reliable information to inform decision-making.
• Acceptability underscores the willingness of individuals to participate in the surveillance process.
• Sensitivity and Positive Predictive Value focus on a system’s ability to detect true cases and minimize false positives.
• Representativeness ensures that the surveillance system does not systematically exclude subpopulations, for instance groups defined by geography, age, ethnic group, gender identity, sexual orientation, incarceration status, etc.
• Timeliness emphasizes the need for swift data collection and reporting to enable timely response and intervention, including the ability for disease events to be detected in “real time”. What counts as “timely” is different for different diseases, reflecting the urgency of response.
•  Stability and Sustainability highlight the need for consistent, long-term operation of the system

View the full Surveillance Systems Attributes Table with examples, definitions, and questions to ask.

Questions to Consider

If the system performs poorly in one or more of these attributes, questions to consider include:

• Is the case definition too broad or narrow?
• Are the data specific enough?
• Are we collecting data from all relevant sources (i.e., only collecting facility data and not community data)?

System performance also encompasses the impacts of the surveillance system and how data are used to improve population health. This aspect represents the ultimate purpose of the surveillance system: public health action. If the data that are being collected are not being used, the surveillance system is failing.

Questions to Consider

• Who are the different end users of this system (global, national, subnational, local)?
• What data are needed for the different users in terms of accessibility/access, ability to interpret data, and actionability?

Common Approaches

• Quantitative Secondary Analysis
• Modeling (expected vs observed) based on external data