In a sea of possibilities, how do we choose how to measure the digital economy?

As digital technologies take on an ever-expanding role in everyday life, there is a growing demand among policymakers for tools and methods to effectively measure the digital economy. 

Understanding how the digital transformation is impacting our economies and societies is crucial for informing and guiding decision-making. Data is needed for governments to produce precise diagnostics, assess the potential impact of policy options, monitor progress, and evaluate the efficiency and efficacy of implemented actions. But as the digital landscape continues to evolve, conducting timely and frequent research is critical to ensure that the data keeps pace with this rapidly changing environment. 

What tools and methods exist to capture the diversity and complexity of the digital economy? What options do policymakers have to measure the advancement of their countries’ digital transformation, considering their local context and needs? What trade-offs do they face?

As part of its efforts to support African countries advance their digital transformation, the AU-EU Digital for Development (D4D) Hub conducted in-depth research aimed at answering these questions. The purpose is to help policymakers understand the existing approaches for mapping and analysing the digital economy and society, as well as their gaps and limitations, so they can be better equipped to choose the tools and methods that can best meet their measuring needs.

The AU-EU D4D Hub analysis covers a wide array of 25 diagnostic tools and indices produced by UN agencies, regional organisations, international donors, and even the private sector (see table in the section below). Some try to cover the entire digital ecosystem, such as the World Bank’s Digital Economy for Africa diagnostic tool, while others focus on measuring connectivity, like GSMA’s Mobile Connectivity Index or the Global Data Barometer. Other tools focus instead on key themes, such as the Gender Digital Divide Index, or the Global Cybersecurity Index. 

Just as we see a wide variety of tools, there is also a diverse range of methods used to assess the digital economy. While some attempt to measure maturity in order to identify needs and gaps, others use repeated surveys to measure and track progress over time. Data sources can be quantitative, qualitative, or a mix of both. Some tools rely on secondary sources while others conduct in-depth data collection with primary sources. 

To better understand the different tools and methods available for measuring the digital economy, below is an overview of the main findings of the AU-EU D4D Hub’s analysis. 

Tools for measuring the digital economy

Despite its diversity, all methods that were analysed by the AU-EU D4D Hub feature some combination of the four tools listed below: 

1. Indices for evaluating a related topic in the digital economy.
2. Diagnostic assessments or frameworks.
3. Theoretical frameworks that lay out an understanding of the central concepts of a given topic.
4. Resources and toolkits that provide background information on relevant topics related to the digital economy or existing data sources. 

The table below offers some categories to which the different tools and methods can be allocated. For example, a tool could be structured around change or impact, it could focus on a specific category of society or the economy, it could focus on a specific goal (like inclusion) and the methods can either give a snapshot or measure progress over time. For each option, there are trade-offs, both in terms of level of detail and depth but also in terms of how realistic it is for the methods or tools to be applied again in future. 

Because tools are designed as a result of an organisation’s specific rationales and research goals, there is a great deal of variation found in the four categories listed above. In some cases, a tool can develop a framework for understanding an aspect of the digital economy, like UNDP’s Digital Transformation Framework, or evaluate a broad concept, like an entire country’s digital economy, as is the case with USAID’s Digital Ecosystem Country Assessment (DECA). In other cases, a tool can have a much narrower scope, focusing on a more specific theme connected to the digital transformation, such as ITU’s Global Cybersecurity Index. The Toolkit for Measuring the Digital Economy by G20 DETF is an example of a resource providing background information on how to measure the digital economy. 

The table below offers a non-exhaustive list of tools measuring aspects of the digital economy and society that are currently available. The list provides a good overview for policymakers looking to select a tool or methodology, including a classification by type.

Data sources 

One of the main characteristics to distinguish different tools and methods is based on the type of data they use. Data sources can be broken down into two main dichotomies: quantitative vs. qualitative data and primary vs secondary data.

Quantitative data is a numerical measurement of the subject, while qualitative data describes the subject. Some common sources for quantitative data include surveys, censuses, administrative records, data portals, and even social media. Index assessments use quantitative data (and in many cases also secondary data sources) which are weighted through a methodology to form a desired index score. This is standardised, allowing for opportunities for comparison, like in the case of the GSMA Mobile Connectivity Index, which uses indicators to measure mobile connection performance across different areas, such as infrastructure, affordability, consumer readiness, and content and services. Indices in the Mobile Connectivity Index are directly collected by GSMA and are mostly primary data sources. 

Qualitative data, on the other hand, can be obtained through surveys and social media, as well as focus groups and interviews. Unlike index assessments, which draw solely on quantitative data, diagnostic assessments combine both quantitative and qualitative data sources, and tend to rely more on qualitative data. This is because, in addition to tracking the evolution of a variable over time (quantitative), diagnostic assessments also seek to understand the driving factors behind it, using methods like interviews or desk research. The World Bank’s Digital Economy for Africa (DE4A) Diagnostic and USAID’s Digital Ecosystem Country Assessment (DECA) are both examples of diagnostic assessments. ITU and UNICEF’s Framework for Universal and Meaningful Digital Connectivity, which drew on multi-stakeholder consultations to determine its outcomes, is also an example of a tool using qualitative data sources. 

Assessments do not need to limit themselves to a single type of data. Understanding the role of different types of data can help determine the sources that are most appropriate for any given context. 

The graph below shows where some of the tools listed above are placed taking into consideration the type of data they use. Primary data may be more reliable but will be more costly to obtain and more difficult to verify. Secondary data, on the other hand, can introduce unwanted bias and might not be available for a number of key indicators such as digital skills, digital inclusion, or digital rights.

A tailored measurement solution?

There is an expanding deck of tools and methods to choose from when considering the best approach for measuring the digital economy. From broad assessments and frameworks seeking to capture an entire ecosystem, to drilling down to certain themes, like data connectivity or e-governance, there is a wealth of options for policymakers to choose from. Yet in some cases, policymakers might find themselves confronted with the need to have a tailored methodology for a specific context.

In such cases, it is important to ask: What is the assessment seeking to achieve? What kinds of resources are available? The overview of tools and data sources provided by the AU-EU D4D Hub can also help policymakers understand the possible paths. Exchanging with other partners who have been involved in such measurement endeavours can also be valuable to identify synergies, avoid duplications, and leverage shared resources.