3.7. Roadmap: Measuring broadband quality

Why are indicators on broadband quality needed?

The actual performance of broadband connections, (i.e. broadband quality), is critical to meeting consumers’, policy makers’, and regulators’ various objectives. Broadband performance is a fundamental metric for consumers to make informed choices, as it reflects the quality of their experience and enables them to assess any differences between advertised speeds and actual speeds. For policy-makers and regulators, being able to assess broadband performance is essential to ensuring the accessibility of online services and to ascertaining whether services are meeting their goals for overall market development (e.g. competitiveness, coverage). In 2012, the OECD was tasked with assessing available datasets that provide robust data offering like-for-like comparisons over time, and with working towards a long-term goal of co-ordinated measurement of speed and other service qualities (OECD, 2012).

One key aspect of connection quality is download and upload speed. Measures can refer to advertised speeds (describing the theoretical maximum speed that can be expected) or the actual experienced speed. While widespread penetration of broadband is observed in OECD countries, there is a vast difference in the speeds available to users and hence the applications from which they can benefit. To reflect these differences, the OECD broadband portal, (https://oe.cd/broadband), provides a breakdown of fixed broadband subscriptions by speed tiers, ranging from subscriptions with speeds as low as 256 Kbps to over 100 Mbps. However, these are advertised speeds and not actual experienced speeds, which can be significantly lower.

In addition to broadband speeds, other quality factors such as latency or data packet loss have become increasingly important. Latency – the round-trip time for information between two devices on the network – is key for many advanced applications such as Virtual Reality and Augmented Reality, remote robotics, fully automated vehicles, and haptic technologies (present in remote surgery and industrial IoT applications). These require ultra-reliable broadband. In this respect, the fifth generation (5G) of broadband wireless networks, and increasing deployment of fixed backhaul (e.g. fibre) necessary for both mobile and fixed networks will help to meet these increasing demands. Another quality measure, data packet loss, is more common in wireless networks and can significantly affect their reliability, and hence critical services that need to be delivered error-free and in real-time, such as remote surgery or air-traffic monitoring. These factors, in addition to security considerations and the robustness of networks in the event of disasters, affect the ways in which digital services can be accessed, used, and in turn, the value generated for businesses and consumers. They should therefore be included in a rounded assessment of broadband quality.

One example of a co-ordinated quality assessment of communication networks comes from the National Information Society Agency (NIA) in Korea. The assessment focuses primarily on coverage and speed, including speeds experienced in challenging environments such as in costal or mountain areas and on sea routes. It also incorporates multiple perspectives, including official testing by regulators, self-evaluation by operators, quality evaluations by users.

Summary of Korea Quality of Communication services evaluation framework and select results, 2017

Source: NIA, 2017.

Quality should also be considered in the context of “access divides”, such as between businesses of different sizes, or households of different incomes or locations. When quality is not considered, disparities in broadband uptake between urban and rural areas appear small in many countries. Indeed, in several countries such as Luxembourg, Norway and the United Kingdom, rural households are more likely to have subscriptions to broadband than urban households. However, there is evidence that, despite advances in recent years, rural areas still tend to have relatively slower, less reliable connections than urban areas (see page 3.5). Recognising these challenges to broader economic and social developments, various OECD countries have been developing programmes, within national broadband strategies or regional development strategies, to close the urban-rural digital divide. In order to meaningfully assess such divides, both within and across countries, robust and comparable information on relevant aspects of quality must be made available.

What are the challenges?

The download speeds advertised by communication operators, as collected in OECD broadband subscriptions statistics, can be very different to the average speeds experienced by subscribers. A rounded and nuanced assessment of speed therefore needs to encompass multiple perspectives and sources including information on connection technology (e.g. fibre), the type of subscriber (e.g. retail or business), and indicators of the speeds actually experienced by users.

The OECD collects information on connection technology but, as with other indicators, businesses cannot yet be distinguished from retail customers. Indicators on experienced speed are available from market players such as content delivery networks (e.g. Akamai), online service providers (e.g. Netflix), online speed test tools (e.g. Ookla and M-Lab, see page 3.5), and other firms operating key components of the Internet. Together these could provide a cross-section of all types of users, but each gives only give a partial perspective on experienced speeds and the broader Internet experience.

Furthermore, statistics on the availability of high-speed broadband networks in rural and remote areas are not available across all OECD countries. While national broadband maps are insightful, data on coverage by speed tiers in rural versus urban areas would allow for cross-country comparison and a meaningful evaluation of quality gaps in broadband access.

Looking at quality more broadly, information on service reliability (outages, packet loss rates, etc.) is not widely available.

New opportunities are emerging in terms of “crowd-sourced” and open data that have the potential to empower consumers by making unprecedented information available to them. These sources, however, may not always provide the information needed to inform specific policy and regulatory goals. As a number of factors can influence results, broadband quality measurement faces greater potential hurdles on the path to international agreement and acceptance than telecommunication subscriptions measures did.

Options for international action

In June 2014, as a follow-up to the 2012 Broadband metrics workshop, the OECD published a report on Access Network Speed Tests (OECD, 2014). The report reviewed information on official speed tests to date, as well as the strengths and drawbacks of their methodologies, emerging good practices, and the challenges to undertaking a harmonised approach across OECD countries. Network speed tests in OECD countries can be found at the following link: http://www.oecd.org/internet/speed-tests.htm.

To build upon this foundation, examples of gathering data on coverage by speed tiers in rural versus urban areas (e.g. NIA, 2017) could be collated to serve as models for others and provide a template for co-ordinated indicators.

One further avenue involves collating and comparing information from third party sources that have the scale to generate useful indicators of the performance of different networks spread around the world (e.g. the Akamai content delivery network, Netflix streaming video service, and online gaming services such as STEAM). This includes data on speeds across individual networks and also aggregated statistics at the national level (OECD, forthcoming).