Chapter 5. Monitoring and evaluation: How are SMEs performing?
This chapter discusses current approaches for assessing implementation of SME support and public procurement policies, strategies and measures. While the importance of measuring and monitoring the performance is widely acknowledged, those activities not consistently carried out by countries. Governments that do measure SME policies focus on participation and success in tender processes. Such measures are useful for measuring changes in SME engagement over time, but they do not shed light on the location or size of the barriers that SMEs face. The proliferation of e-procurement systems has unlocked a great deal more data that can be analysed towards those ends. This chapter identifies the different measurement techniques employed by countries to improve the development and implementation of policies and regulations. The key role played by central purchasing bodies (CPBs) is also examined, followed by a close look at the various challenges to measuring the effectiveness of SME support policies.
The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.
Evaluating the results of policies is crucial for monitoring their implementation, as well as for managing the performance of the public procurement system, as highlighted by the OECD Recommendation of the Council on Public Procurement (Box 5.1).
The Council:
X. RECOMMENDS that Adherents drive performance improvements through evaluation of the effectiveness of the public procurement system from individual procurements to the system as a whole, at all levels of government where feasible and appropriate.
To this end, Adherents should:
i) Assess periodically and consistently the results of the procurement process. Public procurement systems should collect consistent, up-to-date and reliable information and use data on prior procurements, particularly regarding price and overall costs, in structuring new needs assessments, as they provide a valuable source of insight and could guide future procurement decisions.
ii) Develop indicators to measure performance, effectiveness and savings of the public procurement system for benchmarking and to support strategic policy making on public procurement.
Source: (OECD, 2015[1]).
In most OECD countries, SME policies focus on facilitating access to public procurement opportunities by ensuring a level playing field for all economic operators. Governments moreover seek economic and societal development through SME growth. It is therefore crucial to evaluate whether the measures in place achieve these objectives. Given the plethora of different policy options that can be pursued by government, many of which are unrelated to public procurement, a holistic assessment can help to identify the most effective policy mix. Such evaluation makes it possible to gauge performance, and in turn better manage the public procurement function by helping government to identify implementation gaps and, if any, the unexpected adverse effects of such measures.
This chapter discusses current approaches for assessing implementation of SME facilitation and support public procurement policies, strategies and measures. In addition to discussing how implementation is monitored, it addresses measurement of the effectiveness of procurement in achieving various objectives, as stressed by the OECD Recommendation.
5.1. Measuring SMEs’ ease of access through their market share in public procurement
The share of SMEs’ participation in public procurement markets is one way of measuring their ease of access. Indeed, the share of contracts that have been awarded to SMEs, in terms of both number and value, is the most commonly measured area, yet it is still only measured by less than half of OECD countries (Figure 5.1). Some countries, such as Germany and Poland, are currently planning to systematic collect data on the share of public procurement awarded to SMEs.
Note:. In Austria, this measurement is only performed for the Federal Procurement Agency (BBG), the country’s CPB. In Germany, the data will be measured once the new regulation on statistics on public procurement takes effect.
Source: 2017 OECD Survey on strategic use of public procurement to support SMEs.
Some countries have defined specific targets for SME participation in public procurement markets – namely, Australia, Canada, Japan, Korea, Mexico and the United Kingdom (Table 5.1). These targets are also described in terms of contracts won by SMEs. Countries sometimes also define the timeline by which the goal must be achieved. For instance, the United Kingdom has a target of 33% of central government procurement spending to go to SMEs, directly and through the supply chain, by 2022. This replaces a previous target that was met, to achieve 25% by 2015.
Targets defined in public procurement regulatory frameworks can demonstrate governments’ commitment to the objective of increasing SME participation. Statutory goals exist in some countries, such as Japan, Korea and Mexico, as described below:
-
In Japan, the Basic Policy on State Contracts with Small and Medium-sized Enterprises, which is annually enacted based on the Act on Ensuring the Receipt of Orders from the Government and Other Public Agencies by Small and Medium-sized Enterprises, sets the goal for the fiscal year, including the target amount and the ratio of contracts with SMEs and micro-enterprises against the total budget of state contracts. Cabinet approval was given in August 2015 for setting 54.7% as the target contract ratio among SMEs and micro-businesses in fiscal year 2015, and for roughly doubling the ratio of state contracts with new SMEs over the three fiscal years from 2015 to 2017, compared to 2014 (estimated at around 1%).
-
In Korea, the Enforcement Decree of the Act on Facilitation of Purchase of SME-Manufactured Products and Support for Development of their Markets sets an annual goal for SME-manufactured product purchases, of 50% or more of total purchasing value. Furthermore, Korea has priority purchase goal of 10% for SME-developed technology products. The Ministry of SMEs and Startups centrally monitors these goals by collecting purchasing performance records and purchasing plans, and manages a database system containing these statistics. The results are submitted to the National Assembly and released to the general public.
-
In Mexico, the Law of Acquisitions, Leases and Services of the Public Sector stipulates that in order to promote the development and participation of SMEs in public procurement, agencies must award them at least 50% of the contracts that were carried out as exceptions to open tender each year. Furthermore, the Law for the Development of the Competitiveness of Micro, Small and Medium-sized Enterprises states that agencies must plan on gradually awarding at least 35% of their procurement of goods, services and works to SMEs. An inter-ministry committee establishes an annual goal for each agency in accordance with their budget.
SMEs represent a higher share in terms of the number of public procurement contracts awarded than in terms of value (Table 5.2). The contract share figure also varies widely across countries and, in some countries, between years. In addition to that share, the number of bids submitted by SMEs also reflects their ease of access to public procurement markets and could offer real insight into SMEs’ performance in winning contracts. However, the survey showed that assessing the number of bids received from SMEs is not a common practice.
Assessing the share of contracts awarded to SMEs may underestimate the level of their actual participation by leaving out those participating as subcontractors or in the supply chain. For instance, the United Kingdom had 27.1% of central government purchases from small businesses in 2014-15. This figure included SMEs in the supply chain, which represented 16.2% of that share.
The aggregate level of SMEs in the public procurement market does not give a complete picture of their ease in accessing that market. The SME population is typically composed of very diverse businesses, in terms of age, size, ownership, business models and entrepreneur profiles and aspirations. To reflect the heterogeneity of the SME population, countries must study SME participation in public procurement markets broken down by, for instance, market sector, SME type or size, owners’ backgrounds (including gender aspects), firms’ social characteristics, level of available technologies, skills and experiences, or the regions in which they operate.
As is acknowledged by Nicholas and Fruhmann (2014[2]), variable characteristics – including spend portfolios, priorities, cultures and stakeholder requirements – affect contracting authorities’ procurement decisions (Caldwell et al., 2005[3]). Thus, measuring aggregated results may not allow a more micro-level understanding – for example, a grasp of the benefits that contracting authorities can achieve by purchasing from SMEs (Curran, 2000[4]). Collecting and analysing data at the contracting authority level could help to deepen this understanding.
5.2. The effectiveness of public procurement in achieving broader goals through supporting SMEs
Despite increasing recognition of the strategic use of public procurement for achieving broader objectives, the results of these efforts are not being fully assessed. In balancing the use of public procurement to pursue broader policy objectives against the primary objectives, the OECD Recommendation on Public Procurement highlights the importance of “employ(ing) appropriate impact assessment methodology to measure the effectiveness of procurement in achieving secondary policy objectives (OECD, 2015[1]).”
In addition to opening up public procurement markets to give equal opportunities to suppliers, the countries’ objectives in these policies and strategies extend to economic and societal development through increasing the competitiveness of SMEs, stimulating employment and job creation and boosting innovation (Figure 1.7). Where these types of policy objectives are presented in descriptive terms, it is difficult to effectively measure their achievement (Nicholas and Fruhmann, 2014[2]).
Gauging the effectiveness of such policies and measures involves assessing whether they achieve their objectives, in relation both to the benefits and costs of using public procurement to support SMEs, and to the effectiveness of procurement as a policy lever compared to other means.
Where changes are made to regulatory frameworks, countries can undertake impact assessments and many OECD countries do so with respect to their impact on SMEs. For instance, the legal framework in Austria requires an impact assessment for new legal provisions, and one of the dimensions covered by the assessment is the law’s impact on SMEs. The Dutch experience (Box 5.2) following introduction of the Dutch Public Procurement Act of 2012 provides an example of such an assessment, where the public procurement system was evaluated in terms of SME access and participation.
Following enactment of the Dutch Public Procurement Act of 2012, in 2014 the Netherlands Government evaluated its effect with regard to the ambitions formulated at its drafting, which included:
-
enhancing access for companies – especially SMEs – to government contracts
-
standardisation of the procurement practice
-
creating room for innovation and sustainability
-
decreasing administrative burdens
-
increasing compliance with procurement rules.
The evaluation was divided into three parts: one study dealt with access, standardisation, innovation and sustainability, a second examined administrative burdens and a third focused on compliance.
The evaluation was based on a data study based on the Netherlands Government procurement platform TenderNed and, where necessary to complete the picture, from the private platform Aanbestedingskalender. To complement the data from these platforms, a survey was sent to a representative sample of contracting authorities, selected on the basis of European and national procurement procedures they had conducted in the relevant period. This process was employed for all three parts of the study.
Afterwards, additional data were collected for each particular study to arrive at a relevant data set for each study. As an example, for the study on SME participation, the following methods were used:
-
Data derived from the above methods were checked against information registered at the Dutch Chamber of Commerce to supplement missing information.
-
Contract values were checked, and where values were unrealistic (EUR 1 for instance), the contracts were left out of the analysis.
-
Collected data were supplemented with data from the CBS (Dutch Central Bureau for Statistics).
All this data gathering produced the most thorough and complete picture of SME participation possible. Then it was a simple matter of counting the number of procurement procedures in which SMEs in fact participated compared to the total number of procurement procedures in the relevant period, so as to be able to evaluate their participation. Another aspect examined was the number of contracts actually awarded to SMEs. The study found that:
-
1. About two-thirds of government contracts procured in 2014 were awarded to SMEs. Within that group, most contracts went to SMEs with up to 100 employees.
-
3. In 2014, larger companies (not falling within the definition of a SME with < 250 employees) more often won larger, higher-value contracts than smaller companies (who did come within the definition of SME).
-
4. SMEs that participated in procurement procedures in 2014 were proportionally less likely to win than larger companies.
-
5. The percentage of SMEs awarded contracts has not or barely increased between 2012 and 2014.
-
6. Since 2006 there has only been a small, gradual increase in SME participation in procurement procedures.
-
7. The participation of SMEs in European procurement procedures for goods, services and works remained the same in 2014 as in 2012.
-
8. In national procurement procedures for works, SME participation decreased in 2014 from 2012. In services, there was an increase in the same period.
-
9. Participation of SMEs in European procurement procedures has increased for companies in special utilities (for which special procurement procedures apply) and with the central government.
For more detail, the evaluation (in Dutch) is available at: www.pianoo.nl/sites/default/files/documents/documents/effecten-van-aanbestedingswet-2012.pdf.
At the time this evaluation was published, the government expressed that SME participation and frequency of contract awards will be regularly monitored. The form of this regular monitoring and the methodology to be used are still under discussion.
Source: Country response to the OECD Survey 2017 on strategic use of public procurement to support SMEs.
While public procurement is subject to a growing number of objectives, the primary objective remains delivery of the goods and services necessary to accomplish the government’s mission in a timely, economical and efficient manner (OECD, 2015[1]). In the context of SMEs’ participation in public procurement, there are tensions between potentially competing objectives, for example between SME preference and both fair competition principles and the tendency towards contract consolidation. Especially in the case of the latter, while consolidating procurement needs within one large contract enables cost savings through economies of scale and lower administrative costs, it can also limit SME participation (Kidalov and Snider, 2011[5]).
Table 5.3 shows some of the methodologies adopted by countries when measuring the effectiveness of their policies and strategies.
The methodologies used by countries to assess their procurement system with regard to SME participation and performance reflect not only the different objectives of such policies, but also the institutional frameworks and the organisation of the public procurement system in their respective countries. Relevant and consistent methodologies would greatly improve management of the public procurement function through objective monitoring and performance evaluation.
Canada provides an example of monitoring and evaluation of several aspects of the procurement system, such as the objectives of fairness, openness and transparency, the overall level of client and supplier satisfaction, and the timeliness of services (Box 5.3).
The Office of the Auditor General of Canada and the Office of the Procurement Ombudsman periodically conduct performance audits and reviews of procurement by the government of Canada. The Office of the Procurement Ombudsman routinely conducts Procurement Practice Reviews in order to assess whether departmental practices support the objectives of fairness, openness, and transparency.
PSPC measures the cost of procurement services per CAD 100 of contract value awarded annually by the CPB on behalf of government departments; the overall level of client and supplier satisfaction; and the timeliness of services. It is also developing indicators to measure achievement of socio-economic objectives.
Source: Canada’s response to the OECD Survey 2017 on strategic use of public procurement to support SMEs.
In addition to quantitative data, countries use survey data to understand users’ opinions. When defining the methodology to measure the effectiveness of the public procurement system, countries could consider collecting the opinions of stakeholders, and then use that perception data to assess the functioning of public procurement processes. Engaging with stakeholders helps build a common understanding between the public and private sectors of the system and its challenges. For instance, countries have identified administrative simplification and implementation of e-procurement systems as key measures to facilitate SME access to public procurement opportunities.
At the same time however, certain business associations have pointed to challenges SMEs have had in using e-procurement systems, namely a high administrative burden and lack of visibility on smaller-value contracts. Furthermore, while business associations agree that use of e-procurement solutions is one of the most valuable measures to help SMEs prepare bids for public contracts, no common understanding has been established as to whether it has improved SMEs’ access to procurement markets. While differences in how countries design their e-procurement systems need to be acknowledged, engagement with users – including bidders, suppliers and contracting authorities – could help to better understand discrepancies between the expected and real impacts of measures.
5.3. Monitoring CPBs’ performance on facilitating SME participation
As previously discussed, CPBs – as contracting authorities that are often established to centralise public procurement activities and aggregate purchasing power – have implemented measures to support SMEs in their public procurement activities. The institutional mission, settings, objectives and the scope of activities of CPBs differ from country to country. The measures they adopt and their respective assessments of results reflect these discrepancies.
Regardless of the specificities of each CPB, an essential part of their performance management involves measuring results. As previously discussed in Section 2.5, some CPBs define their institutional objectives around providing all economic operators, including SMEs, with a fair chance of participating in their public procurement opportunities. Yet, in less than half of OECD countries do CPBs carry out evaluations in order to assess the results of these SME-support measures (Figure 5.2).
Note: Sweden carries out a supplier satisfaction survey to collect information on how to improve practices; the survey is given not only to SMEs but also to large suppliers.
Source: 2017 OECD Survey on strategic use of public procurement to support SMEs.
For CPBs as well, the share of contracts in terms of value and number that have been awarded to SMEs represents the most commonly used evaluation methodology. In Korea the Public Procurement Service (PPS), the country’s CPB, makes these statistics available on a monthly basis, broken down according to procurement type (e.g. goods, services, works). CPBs in some countries, such as Sweden and New Zealand, use supplier satisfaction surveys in addition to that methodology.
CPBs indeed provide substantial procurement opportunities to SMEs, especially considering the significant share of the total number of contracts that have been awarded to SMEs (Figure 5.3).
Note: Data for Korea is only partially available. The data for Australia refer to the contracts that had been awarded by the Department of Finance in 2015-16. The line the graph is an equality line. The horizontal or vertical distance between the line and each data point represents any discrepancy between the share of contracts awarded to SMEs in terms of number and that in terms of value.
Source: 2017 OECD Survey on strategic use of public procurement to support SMEs.
SMEs are generally awarded smaller-value contracts by CPBs as well – their share is greater in terms of number of contracts than value. While subject to the institutional objectives of aggregating the purchasing needs of contracting authorities to find better terms and conditions for government purchases, CPBs could also produce spill-over effects for the economy by increasing the participation of SMEs in their public procurement processes. In this sense, some CPBs monitor also the number of SMEs participating in their contracts, such as BBG in Austria (Figure 5.4).
Note: The number of small and medium-sized enterprises is constantly growing, also in part due to the SME strategy of the BBG, which can look forward to a share of 66 percent in small and medium-sized enterprises
Source: www.bbg.gv.at/fileadmin/daten/Downloads/T%C3%A4tigkeitsbericht/Zahlen_Daten_Fakten_2017__v2_.pdf
5.4. Current limitations to measuring the effectiveness of SME support policies
The limitations of current methodologies behind quantitative evaluations of SME support policies generally have been highlighted by governments and by academics. For instance, one study noted that quantitative methodologies do not adequately take into account net positive outcomes, deadweight and displacement (Curran, 2000[4]). SME support policies, being a horizontal policy area, are multi-dimensional, a fact that points to the need to take into account synergies and trade-offs among policies. It follows that robust evaluation of an SME support policy could not be done without considering its relative costs and benefits. Difficulties related to identifying matched control samples due to the heterogeneity of the SME population also add to the complexity of quantitatively evaluating policy effectiveness (Storey, 1999[6]). This suggests that sound design of SME support policies must rely on a thorough understanding of SMEs’ role in the economy. However, “poor thinking and muddled conclusions, problems compounded by a plethora of issues relating to the availability and quality of relevant data” further adds to the complexity (Freeman, 2013[7]).
Governments also face challenges in measuring the effectiveness of SME-support policies in public procurement. Practical difficulties in collecting and processing data, as well as a lack of methodology, constitute some of the main constraints on measurement (Figure 5.5).
Note: The survey respondents were asked to indicate how relevant each option is in their country – from 1 (not relevant) to 4 (very relevant).
Source: 2017 OECD survey on strategic use of public procurement to support SMEs.
The information collected by public procurement authorities may not be sufficient to carry out an evaluation of SME support policies. For instance, the status of bidders may not be collected during the procurement process for reasons of fairness, making it impossible to know which of them were SMEs. Quantitative evaluation of policies could thus require inter-institutional co-ordination or integration of e-government systems. In Israel, for instance, measurement of SME participation in public procurement is carried out in collaboration with the Ministry of Economy and with the Taxes Authority. This enables the governmental expenditure report to be classified by suppliers’ size, according to the definition of the Ministry of Economy. Israel moreover is currently working on creating technological solutions in order to facilitate further data gathering.
5.4.1. Defining an evaluation framework for SMEs’ participation and performance in public procurement
Implementation of e-procurement systems has facilitated the collection of data and information that make performance evaluation and management possible. However, many governments still face difficulties in collecting the right data to establish performance indicators for the public procurement system. Achieving that would typically require additional functionalities to be embedded into existing e-procurement systems, as well as integration of systems or exchange of data with other e-government technologies. Indeed, the development of comprehensive electronic infrastructure for information exchange among public sector agencies, individuals and businesses is enabling public administration to offer better quality services, as well as to better adapt public sector operations to end-users’ needs and preferences.
When they are defined, performance indicators on SME participation and performance in public procurement, they should be in line with the objectives of the system. Accordingly, prior to addressing technical difficulties, governments first should define the areas in which performance assessments will be carried out. In light of this, the following indicators could be considered, disaggregated by aforementioned SME characteristics where possible, as part of the performance assessment framework for measuring different objectives:
-
1. Number of SMEs registered within the supplier registry – First of all, the groups of SMEs that are concerned by public procurement markets need to be identified. This is because, as previous discussed, activities of certain SMEs do not match the public procurement needs. Once this SME population is identified, this population could be compared with the number of SMEs registered in public procurement system. Relatively low representation of SMEs in the public procurement supplier registry, compared to that of large enterprises, could imply that SMEs’ perceive public procurement to be an unattractive business opportunity, or reflect the burdensome registration process.
-
2. Share of bids received from SMEs – This indicator relates more directly to SMEs’ participation in public procurement market opportunities. It encompasses implications such as whether the tender opportunities were clearly and promptly communicated to potential suppliers, or whether the design of the tender itself unnecessarily discouraged participation of potential suppliers.
-
3. Share of contracts awarded to SMEs, in terms of both number and value – This most widely used indicator measures SMEs’ success in tender processes. In particular, the value indicator can also be interpreted as the share of public procurement spending going to SMEs. For this purpose, it would be especially relevant to take into account the SMEs that participate in supply chains. Monitoring these indicators over time, together with indicators 1 and 2, could also shed light on SMEs’ relative competitiveness in public procurement markets, which may affected by SMEs’ economic performance in general or by changes to the public procurement system.
-
4. SMEs’ compliance with the terms of contracts (e.g. delivery deadlines) – This indicator could be of particular relevance in the cases where certain SME enablement programmes are thought to directly increase SMEs’ chances of winning, and thus are believed to incur efficiency costs. The most direct way of comparison in this case would be that of the prices submitted in bids. When that direct comparison is not possible – and even when it is possible – SMEs’ performance at the post-tendering stage could also be monitored. This indicator could furthermore demonstrate whether SMEs are indeed more responsive in cases where there are changes in contracting authorities’ needs.
-
These indicators could be considered at different levels – at the procurement process level, the level of a contracting authority, and the aggregated government level – as well as in relation to types of goods, services and works. These indicators should also be monitored in connection with other indicators of the public procurement system in general, as well as the general economic and market conditions for SMEs. Robust empirical experiments face numerous limitations, such as the absence of comparable samples, as well as difficulties related to the endogeneity of indicators. Nonetheless, continuous monitoring of these indicators, together with qualitative assessment through interviews and stakeholder engagement exercises, would help governments better understand SMEs’ challenges, and shape policies and measures towards a more efficient and effective public procurement system.
References
[3] Caldwell, N. et al. (2005), “Promoting competitive markets: The role of public procurement”, Journal of Purchasing and Supply Management, Vol. 11/5-6, pp. 242-51, https://doi.org/10.1016/J.PURSUP.2005.12.002.
[4] Curran, J. (2000), “What is small business policy in the UK for? Evaluation and assessing small business policies”, International Small Business Journal, Vol. 18/3, pp. 36-50, https://doi.org/10.1177/0266242600183002.
[7] Freeman, A. (2013), Challenging Myths about the Funding of Small Businesses...: Finance for Growth, Demos, London, http://www.demos.co.uk/files/DF_-_Finance_for_Growth_-_web.pdf?1378216438.
[5] Kidalov, M. and K. Snider (2011), “US and European Public Procurement Policies for Small and Medium-Sized Enterprises (SME): A Comparative Perspective”, Business and Politics, Vol. 13/04, pp. 1-41, https://doi.org/10.2202/1469-3569.1367.
[2] Nicholas, C. and M. Fruhmann (2014), “Small and medium-sized enterprises policies in public procurement: Time for a rethink?”, Journal of Public Procurement, Vol. 14/3, pp. 328-360, https://doi.org/10.1108/JOPP-14-03-2014-B002.
[1] OECD (2015), OECD Recommendation of the Council on Public Procurement, https://www.oecd.org/gov/ethics/OECD-Recommendation-on-Public-Procurement.pdf (accessed on 18 May 2017).
[6] Storey, D. (1999), “Six Steps to Heaven: Evaluating the impact of public policies to support small businesses in developed economies”, in Sexton, D. and H. Landström (eds.), The Blackwell Handbook Of Entrepreneurship, Blackwell Publishing Ltd, Oxford, United Kingdom, https://doi.org/10.1111/b.9780631215738.1999.00012.x.