Azerbaijan

1

United Nations Development Account

Terminal Evaluation of 2023N “Strengthening Innovation Policies for SPECA Countries in Support

of the 2030”

(2020-2023)

Report completed on: July 2024

Evaluation conducted by: Nelly Dolidze

2

Evaluator: Ms. Nelly Dolidze

Evaluation Manager: Mr. Christopher Athey, Economic Affairs Officer, Innovative Policies

Development Section

The United Nations Economic Commission for Europe (UNECE), Palais des Nations, 8-14,

avenue de la Paix, CH-1211 Geneva 10, Switzerland

The evaluator would like to express her gratitude to the member States, the United Nations

Economic Commission for Europe (UNECE) representatives, non-state institutions, and the

subject-matter experts who provided inputs to the online survey and during an intensive

interview process.

The evaluator also wishes to express her most profound appreciation to all those who

supported the evaluation and made it possible to complete this assignment. Furthermore, the

evaluator would also like to warmly acknowledge the crucial role of the Innovative Policies

Development Section and Programme Management Unit staff at the UNECE for their invaluable

support in this evaluation.

Moreover, many thanks also go to Ms. Elisabeth Türk (Director of the Economic Cooperation

and Trade Division (ECTD)), Mr. Christopher Athey (Economic Affairs Officer), Mr. Nicolas Dath-

Baron (Officer in Charge of the Program Management Unit), Mr. Oscar Fast (Associate

Economic Affairs Officer), and any others to have invested their efforts to provide valuable

comments and advice.

This report was commissioned by the UNECE The findings, conclusions and recommendations of this

report are those of the external evaluator and do not necessarily reflect the views of the UNECE.

3

Table of Contents

List of Acronyms and Abbreviations ....................................................................................................... 5

Tables and Figures ..................................................................................................................................... 6

Executive summary ................................................................................................................................... 7

1. Introduction .......................................................................................................................................... 11

2. Description of the Project .................................................................................................................. 11

2.1 Background.............................................................................................................................. 11

2.2 Project objectives and expected results ............................................................................. 11

2.3 Project strategies and key activities ................................................................................... 11

2.4 Target countries and beneficiaries ...................................................................................... 14

2.5 Key partners and other key stakeholders ........................................................................... 14

2.6 Resources ................................................................................................................................ 15

2.7 Link to the Sustainable Development Goals (SDGs) ......................................................... 15

2.8.1 Innovative elements ............................................................................................................... 15

3. Evaluation objectives, scope and questions ................................................................................... 15

3.1 Purpose and objectives ......................................................................................................... 15

3.2.1 Evaluation scope, criteria and questions ............................................................................ 16

4. Methodology ........................................................................................................................................ 17

4.1 Evaluation Limitations ........................................................................................................... 19

5. Findings ................................................................................................................................................. 19

5.1 Relevance ................................................................................................................................. 19

5.2 Effectiveness ........................................................................................................................... 34

5.3 Efficiency ................................................................................................................................. 48

5.4 Sustainability .......................................................................................................................... 52

6. Conclusions and Recommendations ................................................................................................ 55

4

7. Lessons learned and good practices ................................................................................................ 57

Annexes ..................................................................................................................................................... 58

Annex 1: Evaluation ToR ......................................................................................................................... 59

Annex 2: Evaluation Matrix ..................................................................................................................... 64

Annex 3: Data Gathering Tools .............................................................................................................. 66

Annex 4: List of Stakeholders Interviewed .......................................................................................... 78

Annex 5: List of Documents Reviewed ................................................................................................. 79

Annex 6: Management Response and Recommendation Action Plan ............................................ 82

5

List of Acronyms and Abbreviations

AIIB Asian Infrastructure Investment Bank

BA Business Accelerator

BI Business Incubator

DAC Development Assistance Committee

DSCSD Digital Solutions Centre for Sustainable Development

EA Expected Accomplishments

EC The European Commission

EPR Environment Performance Review

FCDO Foreign, Commonwealth and Development Office

GII Global Innovation Index

ICT Information-Communication Technologies

I4SDR Innovation for Sustainable Development Review

IsDB Islamic Development Bank

ITU International Telecommunications Union

KII Key Informant Interview

MPTF Multi-Partner Trust Fund

NBIASD Network of Business Incubators and Accelerators for Sustainable

Development

OECD Organisation for Economic Co-operation and Development

SDG Sustainable Development Goal

SPECA The United Nations Special Programme for the Economies of Central Asia

STI Science, Technology and Innovation

ToC Theory of Change

ToR Terms of Reference

TTSTI Task Team on Science, Technology and Innovation

UN United Nations

UNECE United Nations Economic Commission of Europe

UNEG United Nations Evaluation Group

UNESCAP United Nations Economic Commission for Asia and the Pacific

UNESCO United Nations Educational, Scientific and Cultural Organization

UNDA United Nations Development Account

UNDP United Nations Development Programme

UNIDO United Nations Industrial Development Organization

UNITAR United Nations Institute for Training and Research

EPR Environment Performance Review

WG on ITSD Working Group on Innovation and Technology for Sustainable Development

WIPO World Intellectual Property Organization of the United Nations

WS Workstream

VNR Voluntary National Review

6

Tables and Figures

Table 1 The Project Structure

Table 2 The Project Activities

Table 3 Approved Budget of the Project

Table 4 Key Evaluation Questions

Table 5 Cluster of Stakeholders Contacted

Table 6 A Snapshot of the Project Activities

Table 7 National Policy Documents and Legal Acts Related to Promoting Innovations

Table 8 The Contribution of the Project Activities to the SDGs

Table 9 Status of the SPECA participating States in Achieving the SDGs (2018 data)

Table 10 Status of the VNRs in the Targeted Countries

Table 11 The I4SDRs and EPRs Carried out in the SPECA participating States

Table 12 Progress Made Against Outcome-level Indicators

Table 13 Progress Made Against Outputs

Table 14 Project Activities Under OP 2.2

Table 15 Project Activities Under OP 2.3

Table 16 International Donor-funded Initiatives to Develop the Innovation Ecosystem in the

Targeted Countries

Table 17 The Project Budget in USD (Planned vs. Actual)

Figure 1 Profile of the Online Mini-survey Respondents

Figure 2 Feedback from Mini-survey Respondents on the Project Alignment with SDGs and

Regional Priorities

Figure 3 Global Innovation Index (GII) for the Project’s Beneficiary Countries (2019-2023)

Figure 4 Conceptual Roadmap of the Project and its Link to the UNECE’s Mandate

Figure 5 Opinions of Mini-survey Respondents on Gender Equality & Mainstreaming, Human

Rights, Disability Perspectives, and Climate Change Considerations

Figure 6 Feedback from Mini-Survey Respondents on their Satisfaction with Project Events

Figure 7 Budget Idem share to the Total Budget (Planned vs. Actual)

7

Executive summary

1. The evaluation of the United Nations Development Account (UNDA) Project 2023N

“Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable

Development” (hereafter called “the Project”) was carried out by an external evaluator in February-June

2024.

2. The Project was launched in January 2020 and concluded in December 2023. It was financed

through the UNDA and had a total budget of USD 483,316. By design, the Project targeted six (6) member

States of both the UN Economic Commission for Europe (UNECE) and the UN Economic and Social

Commission for Asia and the Pacific (ESCAP), and one member State of UN ESCAP: Afghanistan1, the

Republic of Azerbaijan, the Republic of Kazakhstan, the Kyrgyz Republic, the Republic of Tajikistan,

the Republic of Turkmenistan, and the Republic of Uzbekistan.

3. Pursuant to the Terms of Reference (ToR) of the evaluation (ANNEX 1), the evaluation was

conducted in accordance with the Development Assistance Committee’s (DAC) criteria introduced by the

Organisation for Economic Co-operation and Development (OECD) for evaluating the development

projects and programs: relevance, effectiveness, efficiency, and sustainability. UNECE is the primary user

of this evaluation.

4. Overall, the evaluation covered the full duration of the Project and resulted in the following key

findings and conclusions:

➢ The Project's participatory design and implementation modalities, including the sessions and the

composition of the SPECA WG on the ITSD and expert groups, allowed for its complete alignment

with the national and global development agendas. It is noteworthy that six out of the seven

beneficiary countries prioritized innovation and digitalization. At the same time, the project

design and implementation modalities were not heavily focused on the inclusion of the third

sector (i.e., private sector, associations, and relevant non-state actors).

➢ The evaluation also validated that the Project was fully aligned with global and regional priorities

and four SDGs 2 . Moreover, the Project activities were thematically coherent with and

complementary to the interventions (which vary per country) funded by other international

donors (including UN agencies).

1 Afghanistan a member of SPECA but not of the UNECE - participated in subregional activities supported by UN

ESCAP, of which Afghanistan is a member. 2 I.e., SDG 8 (“Promote sustained, inclusive, and sustainable economic growth, full and productive employment and

decent work for all”), SDG 9 (“Build resilient infrastructure, promote inclusive and sustainable industrialization, and

foster innovation”), SDG 12 (“Ensure sustainable consumption and production patterns”), and SDG 17 (“Strengthen

the means of implementation and revitalize the Global Partnership for Sustainable Development”).

8

➢ The operational, structural, and conceptual arrangement of the Project safeguarded its direct

contribution to the UNECE’s overall mandate, and its work programme related to economic

cooperation and integration as well as the environment. The key stakeholders highlighted the

exceptional position held by the UNECE in securing high-level political support, cross-regional

engagement and collaboration, and the provision of multisector technical and analytical expertise.

➢ The evaluation found that the Project’s objectives and results were achieved at the outcome and

output levels. Noteworthily, the most significant results/outcomes of the Project were the

development and adoption of the Action Plan for the SPECA Innovation Strategy, the development

of the SPECA Network of Business Incubators and Accelerators for Sustainable Development, and

the Project’s contribution to the development of the Kazakhstan Association of the Universities

Business Incubators and Accelerators (KAUBIA, www.kaubia.kz). The stakeholders reported on

the capacity building events conducted within the framework of the Associations mentioned

above. However, the overlap between the outcome level and output level indicators3 renders a

distinctive qualitative assessment of the reported results impossible.

➢ The Project provided a platform for policy-level discussions within the framework of the SPECA

Working Group on Innovation and Technology for Sustainable Development and the 16th session

of the SPECA Governing Council. It facilitated science, technology, and innovation gap analysis for

all the targeted countries. It also supported the production of a background paper related to the

prospects for SPECA regional cooperation on innovation. UNECE posted all the reports online to

make them easily accessible to the public. The Project also contributed to building the capacity of

the targeted countries through face-to-face and online workshops, side events, and training

sessions.

➢ The Project results framework was aligned with the relevant development priorities of the

targeted countries but lacked gender, human rights, disability, and climate-change-sensitive

indicators by design. The absence of indicators to measure progress across gender, human rights,

disability, and climate change dimensions resulted in an information gap and low awareness

among beneficiary countries regarding the correlation between the innovation agenda and these

dimensions. This might have lessened the impact of the Project deliverables on the capacity of

the participating countries to prioritize progress in the relevant cross-sectoral processes and

structures that would eventually contribute to the achievement of the relevant SDGs (i.e., SDG 54,

SDG 105, and SDG 136).

3 Outcome level indicator IA1.1 (“An action plan for the SPECA innovation strategy with performance indicators

aligned with the SDGs is developed and agreed with SPECA countries and adopted by the SPECA Governing

Council”); Output level indicator OP1.4 (“Produce a final draft of the SPECA Innovation Strategy Action Plan and

secure its formal approval by the SPECA Governing Council, with the attendance of the national focal points and

experts (a session within the framework of the SPECA Economic Forum and the SPECA Governing Council”). 4 SDG 5: “Gender Equality.” 5 SDG 10: “Reduced Inequality.” 6 SDG 13: “Climate Action.”

9

➢ Despite lacking gender equality, human rights, disability, and climate change-sensitive indicators,

the UNECE incorporated climate change and gender mainstreaming agenda in the Project’s

activities. The side event to the 16th session of the UNECE Committee on Innovation,

Competitiveness and Public-Private Partnerships promoted women’s entrepreneurship in

transition economies for sustainable development, and the Work Plan for the SPECA WG on ITSD

approved at the 4th Session of the SPECA WG on ITSD referred to climate change dimension.

➢ The Project experienced some external challenges (e.g., the COVID-19 pandemic and political

changes in the host and beneficiary countries) that delayed the project implementation. The

Project team successfully applied an agile management approach to secure an iterative and

incremental implementation of the Project, focusing on flexibility, collaboration, and the

satisfaction of the participating countries and institutions. This also included the budget

modifications that allowed for all the results to be achieved as planned at the outcome and output

levels and switching to an online capacity building mode. Therefore, the budget analysis showed

a significant increase in spending on budget items related to external contractual services and

expert fees.

➢ The Project backstopping and administration were limited to one P-level staff, and a full-time

administrative support staff was lacking. According to key stakeholders, the Project's human

resource structure needed readjustment to secure advanced backstopping (at the administration

level) and continuous engagement of external thematic experts and service providers.

➢ The evaluation confirmed the interest of the beneficiary countries and a sense of ownership

regarding the Project results. At the same time, the evaluation acknowledged that several

strategic challenges directly affected the sustainability, scale-up, and replication of the Project

results, including the lack of financial resources, political influences from neighbouring countries,

changes in the political agenda, staff turnover, and the shortage of a structured knowledge

transfer system in the beneficiary countries. The evaluation concluded that the sustainability,

replication, and scale-up of the Project’s results cannot be guaranteed unless the UNECE and

SPECA participating States proactively address the external risks and challenges associated with

financial limitations through the sustainability strategy (i.e., SPECA Multi-Partner Trust Fund

(MPTF)) to support SPECA participating States across all relevant projects.

➢ The evaluation also acknowledged the proactive actions of the Secretariat to raise funds and

continue supporting SPECA participating States in strengthening their innovation policies. Its new

project proposal (pending donor approval) also incorporated inclusive digital transformation and

leaving no one behind agenda with a focus on gender equality and the empowerment of women

and girls.

5. Based on the above findings, the evaluation resulted in the following five recommendations:

10

➢ The evaluation recommends continuing to apply participatory and agile management

modalities in future project design and implementation to allow the project teams to get real-

time insights into project progress and potential issues.

➢ The evaluation strongly advises continuing engaging the private sector, associations, and non-

state actors in project activities to strengthen in-country and regional collaboration across

sectors and secure a better balance of political support and a sense of ownership with industry-

specific insights.

➢ The evaluation also heavily recommends revising the formulation of outcome and output level

indicators of future projects to avoid overlap and ensure that complex concepts are broken

down into measurable components at the outcome level. Furthermore, the evaluation

recommends incorporating gender, human rights, disability, and climate change indicators into

the results framework to enable continuous reporting on progress made.

➢ With regard to the implementation modalities of the capacity building activities, the evaluation

recommends the development of online webinars (for general capacity building activities at the

national level) with face-to-face meetings (those addressing regional, strategic and more

advanced knowledge sharing). However, face-to-face regional meetings and site visits would be

more relevant for the sort of capacity building activities that would potentially lead to cross-sector

strategic partnerships and provide first-hand experience of the practical implications and

challenges of the approaches applied in the host countries.

➢ The evaluation recommends reconsidering project staffing patterns to align their level of

engagement to the lifetime of the project. The Secretariat needs to include administrative support

and professional-level staff in the project proposals to ensure the smooth continuity of new

project activities. This is particularly crucial, taking into account the level of effort required for

quality assurance of the developed concept papers and assessment notes and for organizing

cross-country and regional face-to-face capacity building activities, workshops, and side events.

Page 11 of 84

1. Introduction

6. The current document presents the evaluation report for the UNDA’s “Strengthening

Innovation Policies for SPECA Countries in Support of the 2030 Agenda for Sustainable Development”

project (hereinafter, “the Project”).

2. Description of the Project

2.1 Background

7. The Project was initially launched in January 2020, following a decision taken in 2017 at the

12th session of the Governing Council of the United Nations Special Programme for the Economies of

Central Asia (SPECA) to develop an innovation strategy to promote sustainable development in the

SPECA region. Later, the innovation strategy was submitted for approval to the 2019 SPECA Governing

Council, aiming to “…achieve a sustained improvement in the national capacity and capabilities of the

SPECA countries to formulate and implement innovation policies for sustainable development and to

raise the level and quality of regional cooperation in the implementation of innovations that target

and support sustainable development.”7 The Project was expected to be concluded in December 2023.

It was financed through the UNDA and had a total budget of USD 483,316.

2.2 Project objectives and expected results

8. The Project was set out to support and create further sustainable momentum in the course of

taking the first steps toward putting the above-mentioned strategy into practice. In addition, it

intended to support the seven SPECA participating States in their efforts to spur innovation as a central

driver of the increasingly urgent transition from a resource-intensive model of economic development

characterized by low productivity to a knowledge-based model delivering more sustainable economic

growth, in line with the Sustainable Development Goals (SDGs).

9. The Project encompassed the following two main objectives:

➢ To support and create further sustainable momentum while putting into practice an

innovation strategy to promote sustainable development in the SPECA region, as decided at

the 12th session of the SPECA Governing Council in 2017; and

➢ To strengthen the institutional capacities of the landlocked SPECA countries to harness

innovation as a driver of sustainable development and regional integration.

2.3 Project strategies and key activities

10. The Project was implemented in line with two main workstreams (WSs) corresponding to the

two expected outcomes and seven outputs (Table 1).

7 SPECA_Innovation_Strategy_English.pdf (unece.org).

Page 12 of 84

Table 1: The Project Structure

Workstreams (WS)

Outcomes Outputs Indicator of Achievement (IA)

WS 1

Strengthened co-operation on innovation to promote sustainable development and deliver the Agenda 2030 in the SPECA subregion.

Output 1.1: Develop a subregional gap analysis covering the 7 SPECA countries to inform the first Action Plan for the SPECA Innovation Strategy.

IA 1.1: An Action Plan for the SPECA innovation strategy with performance indicators aligned with the SDGs is developed and agreed with SPECA countries and adopted by the SPECA Governing Council.

IA 1.2: At least two joint initiatives on innovation for sustainable development involving two or more SPECA countries developed and endorsed by the beneficiary countries.

Output 1.2: Organize a subregional workshop to present and validate the gap analysis.

Output 1.3: Organize a subregional workshop to develop an Action Plan and its performance indicators and develop joint activities between countries on innovation for sustainable development.

Output 1.4: Produce a final draft of the SPECA Innovation Strategy Action Plan and secure its formal approval by the SPECA Governing Council, with attendance of the national focal points and experts (a session within the framework of the SPECA Economic Forum and the SPECA Governing Council).

Output 1.5: Organize a subregional workshop to measure the implementation progress of the Action Plan based on its key performance indicators.

WS 2

Enhanced capacity to design and carry out effective innovation policy and institutional reform.

Output 2.1: Develop training materials on good practices on innovation policies and institutions for sustainable development, including technology transfer and science and technology parks (based on activities with ESCAP and IATT on science, technology, and innovation under A2.1).

IA 2.1: At least 3 of the 7 SPECA countries designed at least one initiative to improve support for innovation for sustainable development.

Output 2.1: Within the framework of a national capacity building agreement with 3 SPECA countries, organize 6 national workshops (2 for each country) on a specific topic to address key issues at the national level identified by the gap analysis as part of national consultations carried out under OP1.1, as reflected in the SPECA Innovation Strategy Action Plan. In parallel with these events, UNECE and ESCAP will provide targeted support to three countries on three specific issues.

Source: The Project Design Document, UNECE (2020).

11. In total, in accordance with the Project framework, the United Nations Economic Commission

for Europe (UNECE) team carried out 23 capacity building activities (including face-to-face and online

workshops, study tours and side events, SPECA Working Group meetings, and Expert and Technical

Group meetings) (Table 2).

Page 13 of 84

Table 2: The Project Activities8

# Activity Date and venue

1 1st Session of the SPECA Working Group on Innovation and

Technology for Sustainable Development.

30 July 2020, virtual modality

(Country chair - Kazakhstan).

2 Subregional Workshop: Science, Technology and Innovation

(STI) gap assessment of SPECA countries.

26 November 2020, virtual

modality.

3 Capacity building: 1st meeting of the Task Force on the

Roadmap for the Development of Innovation Support

Infrastructure in Kyrgyz Republic until 2025.

9 June 2021, virtual modality.

4 Capacity building: 2nd meeting of the Task Force on the

Roadmap for the Development of the Innovation Ecosystem of

Kyrgyz Republic until 2025.

8 September 2021, virtual

modality.

5 2nd Session of the SPECA Working Group on Innovation and

Technology for Sustainable Development and Subregional

workshop: Action plan of the SPECA Innovation Strategy for

Sustainable Development.

20-21 October 2021, Hybrid

mode: online platform,

(Bishkek, Kyrgyz Republic).

6 16th session of the SPECA Governing Council: Approval of the

Action Plan of the SPECA Innovation Strategy for Sustainable

Development.

19 November 2021, (Hybrid

session) Tashkent,

Uzbekistan.

7 Capacity building: Challenges and opportunities for supporting

innovative, high-growth enterprises in the SPECA countries.

23 December 2021, Virtual

mode: online

platform (Geneva,

Switzerland).

8 Capacity building for SPECA policymakers on supporting

innovative high-growth enterprises in the SPECA subregion.

15 February and 17 February

2022, Virtual mode: online

platform (Geneva,

Switzerland).

9 Capacity building: “New approaches to innovation policy in the

transition economies of the SPECA subregion.”

19 May 2022, Virtual mode:

online platform (Geneva,

Switzerland).

10 Expert Group Meeting on Innovation and Technology for

Sustainable Development: Proposal for a SPECA Network of

Business Incubators and Accelerators for Sustainable

Development (NBIASD).

19-20 July 2022, Hybrid

mode (Almaty, Kazakhstan)

11 3rd Session of the SPECA Working Group on Innovation and

Technology for Sustainable Development: Approval of

the creation of a SPECA NBIASD with national focal points.

20 July 2022, Hybrid

mode (Almaty, Kazakhstan)

12 Capacity building: Fostering innovative entrepreneurship

through educational curricula in the SPECA subregion.

19 September 2022, Virtual

mode: online platform,

(Geneva, Switzerland)

13 Capacity building for SPECA countries "Effective management

and development of business incubators and accelerators."

27-29 September 2022,

Virtual mode: online platform,

(Geneva, Switzerland).

8 Innovation and Technology for Sustainable Development | UNECE.

Page 14 of 84

14 Capacity building for SPECA countries “Development of business

incubators and start-up accelerators: training for university

management.”

21 February 2023, Virtual

mode: online platform,

(Geneva, Switzerland).

15 Working with venture funds and business angels: UNECE

capacity building for staff of business incubators and

accelerators in the SPECA subregion.

16 March 2023, Virtual mode:

online platform, (Geneva,

Switzerland).

16 Side event to the 70th session of the Economic Commission for

Europe: Fostering circular solutions through innovation.

3 April 2023 (online, Geneva).

17 UNECE capacity building workshop for Chief Innovation Officers

on innovation for the circular economy.

2 May 2023 (Tashkent,

Uzbekistan).

18 Side event to the 16th session of the UNECE Committee on

Innovation, Competitiveness and Public-Private Partnerships:

Promoting women’s entrepreneurship in transition economies

for sustainable development.

1 June 2023 (Geneva,

Switzerland).

19 Study tour to Georgia for innovation policymakers from the

SPECA subregion.

10-11 July 2023 (Tbilisi,

Georgia).

20 Capacity building: Enhancing venture capital in Azerbaijan.

July-August 2023, Virtual

mode: online platform.

21 UNECE B2B capacity building “Innovation for the Circular

Economy: Bridging Start-Ups and Corporations”).

18 October 2023 (Tashkent,

Uzbekistan

22 4th Session of the SPECA Working Group on Innovation and

Technology for Sustainable Development and Subregional

workshop: Progress Update on the Action Plan of the SPECA

Strategy on Innovation for Sustainable Development.

18-19 October 2023

(Tashkent, Uzbekistan)

23 Innovation for the Circular Economy: Empowering Start-ups to

Achieve More with Less: UNECE Study Tour for SPECA

participating States.

20-21 December 2023 (Tbilisi,

Georgia)

Source: Innovation and Technology for Sustainable Development | UNECE.

2.4 Target countries and beneficiaries

12. At its inception, the Project was intended to target six member states of the UNECE and one

member state of the United Nations Economic and Social Commission for Asia and the Pacific (ESCAP):

Afghanistan, the Republic of Azerbaijan, the Republic of Kazakhstan, the Kyrgyz Republic, the Republic

of Tajikistan, the Republic of Turkmenistan, and the Republic of Uzbekistan. During the

implementation, Afghanistan (a member of SPECA and UN ESCAP) participated in subregional

activities supported by UN ESCAP.

2.5 Key partners and other key stakeholders

13. UNECE implemented the Project in close cooperation with the UN ESCAP. Both regional

commissions also provided analytical support to the intergovernmental discussions among SPECA

participating States. In addition, under the framework of the SPECA Working Group meetings, UNECE

cooperated with the United Nations Development Programme (UNDP), Asian Infrastructure

Investment Bank (AIIB), International Telecommunications Union (ITU), United Nations Institute for

Training and Research (UNITAR), and with various ministries and governmental agencies in the SPECA

participating States.

Page 15 of 84

2.6 Resources

14. The approved budget for the Project was $ 483,316. About 45 % of the budget was allocated

for consultancy and expert fees and 24 % for workshop and study tours (Table 3).

Table 3: Approved Budget of the Project

Budget Item Amount ($) % of the Total Budget

Other staff cost - general temporary assistance $ 25,000 5.2 %

Consultants and experts $ 221,316 45.8 %

Travel of staff $ 81,000 16.8 %

Contractual services $ 23,500 4.9 %

General operating expenses $ 16,500 3.4 %

Supply and materials - 0

Furniture and equipment - 0

Workshops/Study tours (grants and contributions) $ 116,000 24.0 %

Total $ 483,316 100 %

Source: The Project Design Document, UNECE (2020).

2.7 Link to the Sustainable Development Goals (SDGs)

15. The Project was fully aligned with four SDGs such as:

- SDG 8 (“Promote sustained, inclusive, and sustainable economic growth, full and productive

employment and decent work for all”),

- SDG 9 (“Build resilient infrastructure, promote inclusive and sustainable industrialization,

and foster innovation”),

- SDG 12 (“Ensure sustainable consumption and production patterns”), and

- SDG 17 (“Strengthen the means of implementation and revitalize the Global Partnership for

Sustainable Development”).

2.8.1 Innovative elements

16. The Project was built on the strong interest of the SPECA participating States in innovative

entrepreneurship and designed to support the implementation of the SPECA Innovation Strategy for

Sustainable Development and its Action Plan. In addition, the Project activities were expected to

strengthen the beneficiary countries' capacity to integrate circular economy principles and innovative

practices into their agenda.

3. Evaluation objectives, scope and questions

3.1 Purpose and objectives

17. The evaluation was commissioned by the UNECE and was implemented in February–June

2024. Its aim was to assess the extent to which the objectives of the Project had been achieved. In

addition, the evaluation outlined any impacts the Project may have had in relation to progressing

human rights, gender equality, disability inclusion, climate change, and disaster risk reduction. The

Page 16 of 84

evaluation also addressed the impact of the COVID-19 crisis and, where relevant, the UNECE’s COVID-

19 early response in relation to the Project.

3.2.1 Evaluation scope, criteria and questions

18. The evaluation applied the criteria of relevance, efficiency, effectiveness, and sustainability

introduced by the Organisation for Economic Co-operation and Development (OECD)9. In addition, the

evaluation addressed gender mainstreaming and assessed any potential or actual impact of the

Project on progressing human rights, gender equality, disability inclusion, climate change, and disaster

risk reduction. The evaluation provided answers to key questions listed below to assess whether the

Project had delivered the optimal outcomes in the most efficient way and to identify key lessons

learned (Table 4).

Table 4: Key Evaluation Questions

CRITERIA KEY EVALUATION QUESTIONS

Relevance 1. To what extent did the Project respond to the priorities and needs of participating

countries?

2. To what extent were the Project activities consistent with global and regional priorities and

aligned with the SDGs?

3. How relevant were the Project activities vis-à-vis the programme of work of the UNECE?

What value have UNECE’s efforts added in this area?

4. To what extent were gender, human rights, disability perspectives and climate change

considerations integrated into the design and implementation of the Project? How can

these perspectives be better included in future projects’ design and implementation?

Effectiveness 5. To what extent were the Project design and set-up effective for meeting the needs of the

beneficiary countries?

6. To what extent were the Project objectives and expected results achieved?

7. To what extent were the Project activities coherent and harmonized with those of other

partners operating within the same context, particularly those of other UN system entities?

8. What were the challenges/obstacles (including COVID-19) to achieving the expected

results? How successfully did the Project overcome these?

Efficiency 9. Were the resources adequate for achieving the results?

10. Were the results achieved on time and were all activities organized efficiently?

11. To what extent were the resources used economically and how could the use of resources

be improved?

Sustainability 12. What measures were adopted to ensure that project outcomes would continue after the

Project ended and to what extent have these measures addressed the existing risks for

sustainability?

13. To what extent do the partners and beneficiaries own the outcomes of the Project? How

is the stakeholders’ engagement likely to continue, be scaled up, replicated, or

institutionalized?

Source: The TOR of the Evaluation, UNECE (2023).

9 Organization for Economic Co-operation and Development – Development Assistance Committee (OECD –

DAC)

Page 17 of 84

4. Methodology

19. The evaluation applied a participatory approach by including all relevant stakeholders in the

process to achieve a high level of ownership with respect to its output. Furthermore, the evaluation

methodology incorporated the analysis of quantitative and qualitative data, with particular attention

given to the cross-validation of information. Data were collected through different methods, including

document review, online mini-surveys among participants of the capacity building activities, as well

as online meetings and key informant interviews (KIIs) with representatives of the UNECE and the

United Nations Economic and Social Commission for Asia and the Pacific (UN ESCAP), international

and national experts engaged in the Project’s implementation, national delegates who delivered

presentations during the Project’s implementation, and the key experts from the SPECA Network of

Business Incubators and Accelerators for Sustainable Development. In addition, the evaluator

attended the workshops and discussions that took place in the course of the following selected

activities of the Project as a direct observer:

➢ “The 4th Session of the SPECA Working Group on Innovation and Technology for Sustainable

Development and Subregional Workshop: Progress Update on the Action Plan of the SPECA

Strategy on Innovation for Sustainable Development” (8-19 October 2023; Tashkent,

Uzbekistan); and

➢ “Innovation for the Circular Economy: Empowering Start-ups to Achieve More with Less:

UNECE Study Tour for SPECA participating States” (20-21 December 2023; Tbilisi, Georgia).

20. Overall, the evaluation consisted of the following main phases: the inception phase; the

fieldwork phase; the reporting phase; and the closure phase. During the inception phase, the evaluator:

➢ Reviewed the relevant documentation (primary and secondary data and Project papers).

➢ Carried out an inventory of the Project’s activities.

➢ Finalized the list of internal and external stakeholders to be reached out to during the

evaluation.

➢ Developed the evaluation framework and data collection tools such as the mini-survey and

interview protocols.

➢ Developed the sampling, data-gathering strategies, and evaluation timeframe; and

➢ Prepared the inception report for the evaluation.

21. During the fieldwork phase, the evaluator gathered data through different methods/sources

such as document analysis, KIIs (online and in-person), direct observations of the workshops, and

online mini-survey results. Meanwhile, the evaluator ensured adequate interaction and consultation

with different internal and external stakeholders in a participatory manner. Overall, the table below

(Table 5) presents the clusters of key stakeholders consulted during the course of this evaluation and

the relevant data-gathering methods (e.g., interviews and online mini-surveys).

Page 18 of 84

Table 5: Clusters of Stakeholders Contacted

Clusters of Stakeholders Data Gathering Means

The Project Team In-depth interviews (online)

Participants of the capacity building workshops Online mini- survey

Invited speakers (workshop presenters) In-depth interviews (face-to-face and online)

Members/participants of the Working and

Technical Groups

Online mini-surveys and in-depth interviews

(face-to-face and online)

Source: The Evaluation Dataset (2024).

22. The evaluator used the most reliable and appropriate sources of information and triangulated

(i.e. cross-validated) primary data (e.g., technical and financial reports of the Project, planning and

monitoring documents, reviews, and studies) and secondary data (e.g., relevant studies/notes

produced by the beneficiary countries, and the UNECE documents and standards).

23. Furthermore, the evaluation entailed the implementation of online in-depth interviews and

the distribution of online mini-surveys to the selected beneficiaries of the Project as well as key

stakeholders to gather relevant qualitative and qualitative data. In total, the evaluation conducted in-

depth interviews with key stakeholders such as the UNECE staff, independent experts, and the direct

beneficiaries/participants of the Project. ANNEX 4 presents the list of stakeholders interviewed over

the course of this evaluation.

24. Furthermore, the evaluation gathered online feedback from 25 respondents, of whom 28

percent (seven out of 25) represented state agencies of UNECE member States, 24 percent (six out of

25) came from the non-governmental sector, 24 percent (six out of 25) were independent experts,

and 20 percent (five out of 25) represented academia (Figure 1). Notably, some respondents reported

several functionalities (e.g., representing both the state sector and academia, or representing a non-

governmental organization and academia at the same time). In addition, 42 percent (10 out of 24)

respondents were female, and 54 percent (13 out of 24) of them were male10.

Figure 1: Profile of the Online Mini-survey Respondents

Source: Evaluation Dataset (2024).

10 One respondent did not answer a gender-related question.

7 (28%)

0 (0%)

3 (12%)

6 (24%)

6 (24%)

5 (20%)

3 (12%)

0 1 2 3 4 5 6 7 8 9

Government - UNECE Member State

Government - Non UNECE Member State

UN Agency

Non-Governmental Organization

Independent Expert

Academia

Other

Page 19 of 84

25. The reporting phase was mainly devoted to the preparation of the evaluation report. Here,

the evaluator ensured that the report’s assessments were objective and balanced, that its findings were

accurate and verifiable, and that its recommendations were realistic. The evaluator prepared the

evaluation report in accordance with the norms and standards of the United Nations Evaluation Group

(UNEG). More specifically, the evaluation process was aligned with the principles of integrity,

accountability, respect, and beneficence:

- Integrity: The evaluator adhered to high professional standards and acted with honesty,

transparency, and objectivity.

- Accountability: The evaluator was accountable for the quality and usefulness of the

evaluation work and ensured that the findings were communicated clearly and accurately.

- Respect: The evaluator respected the rights and dignity of all stakeholders, including the

Project staff, and national decision-makers.

- Beneficence: The evaluator strove to conduct evaluations that was beneficial to the targeted

stakeholders to promote positive social change.

26. Eventually, a draft report was sent to the UNECE to obtain final feedback.

27. During the closure phase, the evaluator gathered feedback from the UNECE and incorporated

it into the final evaluation report.

4.1 Evaluation Limitations 28. The several inherent limitations in the design of this evaluation include the following:

a. Key informants were interviewed on the basis of their availability, which did not allow for proper randomization and left open the possibility of selection bias. In addition, some key informants declined an interview. Accordingly, the sample obtained is not fully representative of the population that is supposed to be analyzed. b. The response rate for the online mini-survey distributed among the Project’s key stakeholders was modest. c. Due to the seizure of power by the Taliban in August 2021, gathering data in Afghanistan was less feasible than initially planned, and Afghanistan was thus removed from the scope of this evaluation.

5. Findings

5.1 Relevance

Finding 1: In-person interviews, online survey responses and desk research confirmed that the Project

to a great extent accommodated the needs and priorities of six out of the seven targeted countries.

However, due to the challenging political situation in Afghanistan since 2021, the country has not

participated in the Project’s activities, although it was one of the seven beneficiary countries of the

Project.

Finding 2: The evaluation validated that the Project was fully aligned with global and regional priorities

and four SDGs, namely SDG 8 (“Promote sustained, inclusive, and sustainable economic growth, full

and productive employment and decent work for all”), SDG 9 (“Build resilient infrastructure, promote

inclusive and sustainable industrialization, and foster innovation”), SDG 12 (“Ensure sustainable

Page 20 of 84

consumption and production patterns”), and SDG 17 (“Strengthen the means of implementation and

revitalize the Global Partnership for Sustainable Development”).

Finding 3: In-depth interviews confirmed that all six targeted countries prioritized innovation and

digitalization. However, the circular economy concept was not fully reflected in national or regional

priorities, largely because this was a new notion for Central Asian countries. Meanwhile, the

evaluation found that the Project’s scope was entirely relevant to UNECE sub-programme 4

(“Economic Cooperation and Integration”), which contributed to the substantive work being

completed under the SPECA framework and was partially aligned with sub-programme 1

(“Environment”).

Finding 4: The scope of the Project was aligned with the strategic reviews conducted by the UNECE

for the member States - namely the Innovation for Sustainable Development Reviews (I4SDRs) and

the Environmental Performance Reviews (EPRs).

Finding 5: Key stakeholders highlighted the exceptional position held by the UNECE in securing high-

level political support, cross-regional engagement and collaboration, and the provision of multisector

technical and analytical expertise.

Finding 6: The evaluation confirmed that the Project’s results framework did not apply gender, human

rights, disability, and climate-change-sensitive indicators by design. Moreover, the Project documents

demonstrated uneven reference to gender, human rights, disability, and climate change dimensions;

some referred to the climate change agenda, linking it to the Project’s thematic areas but neglected

to refer to the gender, human rights, and disability dimensions.

Finding 7: Despite lacking gender equality, human rights, disability, and climate change-sensitive

indicators, the UNECE incorporated climate change and the gender mainstreaming agenda in the

Project’s activities. The side event to the 16th session of the UNECE Committee on Innovation,

Competitiveness and Public-Private Partnerships promoted women’s entrepreneurship in transition

economies for sustainable development and the Work Plan for the SPECA WG on ITSD approved at

the 4th Session of the SPECA WG on ITSD referred to the climate change dimension.

Key Evaluation Question 1: To what extent did the Project respond to the priorities and needs of

the participating countries?

29. The Project assisted in putting into practice the SPECA Innovation Strategy for Sustainable Development11. It is important to note that the preparation of that strategy was originally proposed at the 12th session of the SPECA Governing Council in December 2017 in Dushanbe (Tajikistan) and was prepared by the UNECE Secretariat in cooperation with the SPECA participating States. It was then approved at the 14th Session of the SPECA Governing Council in November 2019. The above- mentioned strategy aimed to strengthen the capacity of the SPECA participating States in formulating and implementing innovation policies for sustainable development and advancing regional cooperation in this regard.

30. The strategy also emphasized that the SPECA participating States should develop and agree on an Action Plan to implement the SPECA Innovation Strategy for Sustainable Development.

31. The desk research validated the fact that the Project supported the development of a subregional gap analysis covering the targeted SPECA participating States to inform the first Action

11 SPECA_Innovation_Strategy_English.pdf (unece.org).

Page 21 of 84

Plan for the SPECA Innovation Strategy for Sustainable Development. The Project also helped to develop the Action Plan for the strategy with relevant performance indicators.

32. All key informants interviewed over the course of this evaluation confirmed that the Project activities were fully relevant for their countries’ needs. The evaluation also noted that, while Afghanistan was included in the list of targeted countries during the design phase of the Project, the country’s engagement was no longer feasible after the Taliban came to power in 2021.

33. The Project introduced diverse activities to fulfill its objectives such as the sessions of the SPECA Working Group (WG) on Innovation and Technology for Sustainable Development (ITSD), capacity building meetings and regional workshops, side events, study tours, Expert Group meetings, and the session of the SPECA Governing Council (Table 6).

Table 6: A Snapshot of the Project Activities

Type of the Project Activity

Activity Title

Sessions of the SPECA WG on ITSD

- 1st session of the SPECA WG on ITSD (30 July 2020). - 2nd session of the SPECA WG on ITSD (20-21 October 2021). - 3rd session of the SPECA WG on ITSD (20 July 2022). - 4th session of the SPECA WG on ITSD (18-19 October 2023).

Session of the SPECA Governing Council

16th session of the SPECA Governing Council: Approval of the Action Plan of the SPECA Innovation Strategy for Sustainable Development (19 November 2021).

Expert Group Meeting

Expert Group Meeting on Innovation and Technology for Sustainable Development: Proposal for a SPECA Network of Business Incubators and Accelerators for Sustainable Development (NBIASD) (19-20 July 2022).

Capacity building meetings and workshops

- Science, Technology and Innovation (STI) gap assessment of SPECA countries (26 November 2020)

- 1st meeting of the Task Force on the Roadmap for the Development of Innovation Support Infrastructure in Kyrgyz Republic until 2025 (9 June 2021).

- 2nd meeting of the Task Force on the Roadmap for the Development of the Innovation Ecosystem of Kyrgyz Republic until 2025 (8 September 2021).

- Challenges and opportunities for supporting innovative, high-growth enterprises in the SPECA countries (23 December 2021).

- Capacity building for SPECA policymakers on supporting innovative high- growth enterprises in the SPECA subregion (15 February and 17 February 2022).

- New approaches to innovation policy in the transition economies of the SPECA subregion (19 May 2022).

- Fostering innovative entrepreneurship through educational curricula in the SPECA subregion (19 September 2022).

- Effective management and development of business incubators and accelerators (27-29 September 2022).

- Development of business incubators and start-up accelerators: training for university management (21 February 2023).

- Working with venture funds and business angels: UNECE capacity building for staff of business incubators and accelerators in the SPECA subregion (16 March 2023).

- Workshop for Chief Innovation Officers on innovation for the circular economy (2 May 2023).

- Enhancing venture capital in Azerbaijan (July-August 2023).

Page 22 of 84

- Innovation for the Circular Economy: Bridging Start-Ups and Corporations (18 October 2023).

Side events ➢ Side event to the 70th session of the Economic Commission for Europe: Fostering circular solutions through innovation (3 April 2023).

➢ Side event to the 16th session of the UNECE Committee on Innovation, Competitiveness and Public-Private Partnerships: Promoting women’s entrepreneurship in transition economies for sustainable development (1 June 2023).

Study tours ➢ Study tour to Georgia for innovation policymakers from the SPECA subregion (10-11 July 2023).

➢ Innovation for the Circular Economy: Empowering Start-ups to Achieve More with Less: UNECE Study Tour for SPECA-participating States (20-21 December 2023).

Source: Innovation and Technology for Sustainable Development | UNECE.

34. All interviewed stakeholders and most of the mini-survey respondents confirmed that the Project was highly relevant to the needs of their countries. Some interviewees also referred to specific legal acts and/or national policy documents that explicitly prioritized innovation in the country's development agenda. The table below (Table 7) presents the relevant documents highlighting the importance of innovation within national frameworks. Table 7: National Policy Documents and Legal Acts Related to Promoting Innovation

Country Relevant Legal Act/Policy Document

Reference to the Innovations and Circular Economy

The Republic of Azerbaijan

Azerbaijan 2030: National Priorities for Socio-economic Development (2021).

Refers to five national socio-economic priorities for 2022-2030: (1) a sustainably growing competitive economy; (2) a dynamic, inclusive, and socially just society; (3) competitive human capital and space for modern innovations; (4) a large-scale return to the “territories liberated from occupation”; and (5) clean environment and a "green growth" country.

Decree of the President of the Republic of Azerbaijan as of 10 January 2019.

Mandated the Presidential Administration to develop a national innovation strategy and related action plan.

"National Strategy for the Development of Information Society in the Republic of Azerbaijan for 2014-2020".

Focused on the establishment of an information society, through the use of ICT by citizens, community, the private sector, and government agencies.

The Law on Science (adopted on 14 June 2016).

Encourages, inter alia, entrepreneurship and innovation among scientists to contribute towards competitive scientific-technological development in the country.

The Republic of Kazakhstan

Entrepreneur Code of the Republic of Kazakhstan (dated 29 October 2015).

Increasing the overall innovation activities in the Republic of Kazakhstan, including the promotion of the development of high-tech and knowledge-intensive industries.

National Development Plan of the Republic of Kazakhstan through 2025.

Emphasizes building a diversified and innovative economy, critical for growth and competitiveness in the global digital economy.

Kyrgyz Republic

National Strategy of Development of the Kyrgyz Republic for 2018-2040.

Sets out that Kyrgyz Republic will have a competitive economy that is focused on the application of innovative and environmentally friendly nature-saving technologies, as well as an economy that is diversified,

Page 23 of 84

balanced, and inclusive, with a favorable investment environment. States that Kyrgyz Republic will accelerate innovative socio-economic development and will strive to advance where the country has the greatest competitive advantages in a globally competitive and open economy. The Kyrgyz government's efforts are aimed at significantly improving the investment climate throughout the country in order to turn the country into an "investment oasis" that will be an attractive place for investment, which has a competitive advantage compared to other countries in the region and the Eurasian Economic Union.

Strategy for Sustainable Industrial Development of the Kyrgyz Republic 2019- 2023.

Focused on innovation, and outlined that the transition to an innovative path of development and the formation of progressive structural changes would require the adoption of urgent measures.

Decree on the approval of the strategy of sustainable development of industry of the Kyrgyz Republic for 2019-2023.

Refers to the development of information and telecommunication employment necessary for industrial development, as well as the introduction of new technologies into the industry.

Concept for the Scientific and Innovative Development of the Kyrgyz Republic until 2022.

Stipulates that the state’s policy for the creation of innovation systems should be implemented in a manner which results in a favorable economic and legal framework that promotes innovative activity, and the building of infrastructure for innovative systems and creating a state-backed system for the commercialization of new intellectual property.

The Republic of Tajikistan

National Development Strategy of the Republic of Tajikistan for the Period up to 2030.

The following three basic principles of future development form the basis for the country’s development strategy until 2030: (1) prevention or prediction (reduction) of vulnerability in future development; (2) industrialization or more efficient use of national resources; and (3) innovation or development through the integration of new developments into the country's social and economic mainstreams.

The Innovation Development Programme of the Republic of Tajikistan for 2011-2020.

The programme sets out to stimulate scientific, technical and innovative activity, form a regulatory legal framework for innovation, exploit the country's scientific potential in innovative processes, effectively use scientific and technological developments and inventions, and create and develop innovative infrastructures.

The Law on Innovation Activity (dated 16 April 2012).

The law outlined innovation activities and established the broad parameters of the state’s innovation policy.

The Republic of Turkmenistan

Concept for the Development of the Digital Economy in Turkmenistan for 2019-2025 (dated 2018).

Aimed to increase the efficiency of the functioning of all sectors of the economy and the public sector through the use of information technology.

Page 24 of 84

Concept for the Development of the Digital Education System for 2019- 2025 (dated September 2017).

Sets out to create an information-based educational environment and the effective provision of sufficient technical equipment.

The Law of Turkmenistan “On innovation activity” (dated August 2014).

Covered the legal, economic, and organizational relationships between/among the subjects of innovation and the factors arising from its implementation.

The Republic of Uzbekistan

Action Strategy on Five Priority Areas of Development of the Republic of Uzbekistan in 2017–2021

Pledged to stimulate research and innovation activities, create effective implementation mechanisms to apply scientific and innovative achievements, and establish scientific and research laboratories and high-level technology centers, as well as technology parks at universities.

Development Strategy 2022- 2026

In 2019, the Agency for Educational Institutions of the Republic of Uzbekistan, established by presidential decree under the Cabinet of Ministers, broke ground on the first four of 14 purpose-built Presidential Schools12 with the mission of delivering world-class education to the most gifted and talented students in the country. This represented the first step in the country’s aspirational vision of developing future leaders in sciences, engineering, social spheres and politics, who would contribute to the innovation economy.

Strategy of Innovative Development for 2019-2021.

Focused on the development of human capital as the main factor determining the country’s competitiveness in the world arena and its innovative progress.

Data: Evaluation Dataset (2024).

35. Over the course of the Project’s implementation, the targeted countries continued with the implementation of structural reforms to advance innovation-supportive ecosystems. For example, pursuant to the Decree of the President of the Republic of Azerbaijan, “On some measures to improve governance in the field of digitalization, innovation, high technologies and communications in the Republic of Azerbaijan” (dated 11 October 2021), the Government of Azerbaijan established the Innovation and Digital Development Agency.

36. Similarly, on 26 January 2023, the President of the Republic of Kazakhstan issued a decree to establish the National Council for Science and Technology. The Council was mandated to improve the management of the national scientific system and elaborate recommendations for setting priorities for the development of state policy in the field of science, and the scientific and technical activities of the country. Furthermore, in January 2024, the Government of Kazakhstan adopted the Law on Science and Technology Policy, which set the goal to double GDP by 2029 through, inter alia, the commercialization of scientific and technical activities. Key Evaluation Question 2: To what extent were the project activities consistent with global and

regional priorities and aligned with the SDGs?

12 The main purpose of the Presidential schools is to identify and educate gifted children through the use of

advanced technologies in the educational process, as well as to support and encourage gifted youth.

Page 25 of 84

37. Desk research validated that the Project activities were designed to contribute to the achievement of several SDGs, namely SDG 8, SDG 9, SDG 12, and SDG 17 (Table 8), as defined in the Project Design Document (hereafter “Prodoc”). Table 8: The Contribution of the Project Activities to the SDGs

SDG SDG Target

SDG 8: Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all.

Target 8.2: Achieve higher levels of economic productivity through diversification, technological upgrading and innovation, including through a focus on high-value added and labour-intensive sectors. Target 8.3: Promote development-oriented policies that support productive activities, decent job creation, entrepreneurship, creativity and innovation, and encourage the formalization and growth of micro-, small- and medium- sized enterprises, including through access to financial services.

SDG 9: Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation.

Target 9.4: By 2030, upgrade infrastructure and retrofit industries to make them sustainable, with increased resource-use efficiency and greater adoption of clean and environmentally sound technologies and industrial processes, with all countries taking action in accordance with their respective capabilities. Target 9.5: Enhance scientific research, upgrade the technological capabilities of industrial sectors in all countries, in particular developing countries, including, by 2030, encouraging innovation and substantially increasing the number of research and development workers per 1 million people and public and private research and development spending. Target 9.b: Support domestic technology development, research and innovation in developing countries, including by ensuring a conducive policy environment for, inter alia, industrial diversification and value addition to commodities.

SDG 12: Ensure sustainable consumption and production patterns.

Target 12.a: Support developing countries to strengthen their scientific and technological capacity to move towards more sustainable patterns of consumption and production.

SDG 17: Strengthen the means of implementation and revitalize the Global Partnership for Sustainable Development.

Target 17.6: Enhance North-South, South-South and triangular regional and international cooperation on and access to science, technology and innovation and enhance knowledge sharing on mutually agreed terms, including through improved coordination among existing mechanisms, in particular at the United Nations level, and through a global technology facilitation mechanism. Target 17.7: Promote the development, transfer, dissemination and diffusion of environmentally sound technologies to developing countries on favourable terms, including on concessional and preferential terms, as mutually agreed. Target 17.8: Fully operationalize the technology bank and science, technology and innovation capacity building mechanism for least developed countries by 2017 and enhance the use of enabling technology, in particular information and communications technology.

Source: The Project Design Document, UNECE (2020).

Page 26 of 84

38. At the Regional Workshop on Innovation and Technology Application for Sustainable

Development held in June 2019, the participating parties were informed about the rationale behind

developing the SPECA Innovation Strategy for Sustainable Development. One of the invited experts

highlighted that, by 2018, the SPECA participating States already faced significant challenges in

achieving the SDGs and lagged behind countries in high-income regions such as Europe (Table 9).

Overall, according to recommendations issued at the 2017 SPECA Economic Forum on “Innovation for

the SDGs in the SPECA Region,” the 12th Session of the SPECA Governing Council (organized in

December 2017) underscored the importance of elaborating an innovation strategy for the SPECA

region to support national governments in achieving the SDGs.

Table 9: Status of the SPECA participating States in Achieving the SDGs (2018 data)

Source: “SPECA Innovation Strategy for Sustainable Development,” Rumen Dobrinsky European Alliance for

Innovation (2019).13

39. Furthermore, about 71 percent (17 out of 24) of the mini-survey respondents confirmed that

the Project activities were consistent to a great extent with global and regional priorities and aligned

with the SDGs (Figure 2).

Figure 2: Feedback from Mini-survey Respondents on the Project Alignment with SDGs and Regional Priorities

Data: The Evaluation Dataset (2024).

13

https://www.unescap.org/sites/default/files/3.%20SPECA%20Innovation%20Strategy%20for%20Sustainable%

20Development.pdf.

17 (0%)

7 (0%)

(0%)

0

5

10

15

20

25

0

5

10

15

20

Were consistent to a great extent Partially consistent Not consistent at all

Page 27 of 84

40. In-person interviews validated that while the targeted countries prioritized innovations and a digitalization agenda, the concept of a circular economy (as distinct from the concept of the green economy) was still new for the Central Asian countries and was not necessarily aligned with national or regional priorities. This was reported in the policy document produced under the Project: “Implementing circular economy practices and policies in SPECA countries is still at an early stage, and many challenges need to be addressed, among which are lack of awareness and knowledge, limited access to funding and resources, infrastructure and logistics challenges, as well as cultural and regulatory barriers.”14

41. In-depth interviews and desk research revealed that the beneficiary countries15 confirmed making efforts to achieve SDG targets, including SDG 9, which sets out to build resilient infrastructure, promote sustainable industrialization, and foster innovation. Moreover, all beneficiary countries except Afghanistan completed their Voluntary National Review (VNR), 16 thereby prioritizing the achievement of the SDGs (Table 10). Table 10: Status of the VNRs in the Targeted Countries

No Beneficiary Country VNR Completion Date

1 Republic of Azerbaijan 201717, 201918 and 202119

2 Republic of Kazakhstan 201920 and 202221

3 Kyrgyz Republic 202022

4 Republic of Tajikistan 201723 and 202324

5 Republic of Turkmenistan 201925 and 202326

6 Republic of Uzbekistan 202027 and 202328 Data: The Evaluation Dataset (2024).

42. Azerbaijan was the first among the post-Soviet beneficiary countries to submit two VNR reports by 2019. In addition, the Government of Azerbaijan launched the National Information Portal on SDGs,29 serving as an interactive dashboard that collects consolidated data on the SDGs from selected government institutions and monitors progress made toward the SDGs in real-time. 43. The President of the Republic of Kazakhstan prioritized the SDGs at the Voice of Global South Summit and called on all countries to focus their efforts on achieving the SDGs30. He also pledged to

14 https://unece.org/sites/default/files/2023-10/6.1.%20%20Road%20Map%20NBIASD-

Circular%20STEP_0.pdf. 15 Noteworthy, the evaluation limitations were affected by the political changes in Afghanistan. 16 https://www.canada.ca/en/employment-social-development/programs/agenda-2030/voluntary-national-review.html; VNRs

are reports that countries submit to the United Nations to report on the actions taken and progress made to advance the

implementation of the United Nations 2030 Agenda for Sustainable Development and the 17 SDGSs.

17 Voluntary National Review 2017, Azerbaijan | High-Level Political Forum. 18 Voluntary National Review 2019, Azerbaijan | High-Level Political Forum. 19 Voluntary National Review 2021, Azerbaijan | High-Level Political Forum. 20 Voluntary National Review 2019, Kazakhstan | High-Level Political Forum. 21 Voluntary National Review 2022, Kazakhstan | High-Level Political Forum. 22 Voluntary National Review 2020, Kyrgyz Republic | High-Level Political Forum. 23 Voluntary National Review 2017, Tajikistan | High-Level Political Forum. 24 Voluntary National Reviews 2023, Tajikistan | High-Level Political Forum. 25 Voluntary National Review 2019, Turkmenistan | High-Level Political Forum. 26 Voluntary National Reviews 2023, Turkmenistan | High-Level Political Forum. 27 Voluntary National Review 2020, Uzbekistan | High-Level Political Forum. 28 Voluntary National Reviews 2023, Uzbekistan | High-Level Political Forum. 29 Home - DIM - National Information Portal for Sustainable Development (azstat.gov.az). 30 Kazakh President Tokayev Addresses SDGs, Food and Water Security at Voice of Global South Summit - The

Astana Times.

Page 28 of 84

prioritize the development of innovation and achievement of the SDGs. In addition, in 2019, the country presented for the first time its VNR31 at the UN High-Level Political Forum in New York, USA. In addition, the second VNR for Kazakhstan was conducted in 202232. 44. In 2020, Kyrgyz Republic conducted a VNR on the implementation of the SDGs in the country. Meanwhile, the SDGs were included in several national policies and reflected in the National Development Strategy (2018–2040) as well as in the Government of Kyrgyzstan’s “Unity, Trust, Creation” (2018–2022) programme.

45. In 2016, Tajikistan developed and adopted the National Development Strategy to 2030 (NDS- 2030), defining the main directions regarding achievement of the SDGs in Tajikistan. Moreover, Tajikistan also conducted VNRs in 201733 and 202334. Elsewhere, in 2023, Turkmenistan was part of a VNR on the progress of the implementation of the Global Agenda for Sustainable Development 202335. 46. Uzbekistan has confirmed its commitment to the achievement of the 2030 Agenda, and its 2017-2021 National Action Strategy served as a pathway towards the SDG’s implementation. According to the secondary data “To fully implement the SDGs, Uzbekistan is actively seeking investment and technology, at the same time strengthening closer ties with Central Asian countries.”36 Uzbekistan also carried out two VNRs, the first in 202037 and the second in 202338.

47. Furthermore, regarding global and regional priorities, several interviewees referred to the Global Innovation Index (GII)39, an annual ranking of countries' capacity for and success in innovation. It is published by the World Intellectual Property Organization (WIPO). For ease of reference, the GII scores for some of the Project’s beneficiary countries are presented below (Figure 3).

Figure 3: Global Innovation Index (GII) for the Project’s Beneficiary Countries (2019-2023)40

Source: GII Reports for 2019-2023, WIPO.

31 Voluntary National Review 2019, Kazakhstan | High-Level Political Forum. 32 Voluntary National Review 2022, Kazakhstan | High-Level Political Forum. 33 Voluntary National Review 2017, Tajikistan | High-Level Political Forum. 34 Voluntary National Reviews 2023, Tajikistan | High-Level Political Forum. 35 VNR-2023 Turkmenistan Report EN.pdf (un.org) 36 Uzbekistan's National Sustainable Development Goals: Progress and Challenges in Achieving the SDGs |

United Nations Development Programme (undp.org). 37 Voluntary National Review 2020, Uzbekistan | High-Level Political Forum. 38 Voluntary National Reviews 2023, Uzbekistan | High-Level Political Forum. 39 GII is useful for policymakers, businesses, and others to assess innovation progress based on innovation metrics

across 132 economies. It includes themes beyond rankings, i.e., funding for innovation.

40 In the context of GII, the lowest rate (i.e., “1”) signifies a stronger innovation environment.

84 82 80 93 89 79 77 79 83 81

90 94 98 94 106 100 109 103 104 111

0

93 86 82 82

2019 2020 2021 2022 2023

Azerbaijan Kazakhstan Kyrgyzstan Tajikistan Uzbekistan

Page 29 of 84

48. The GII for 2019 emphasized that “the subregion of Central Asia is noteworthy for starting to prioritize innovation activities and related policies in a sustained manner”41 and that while no GII data were available for Uzbekistan in 2019, the country was at that time making continuous progress in data collection to be included later in the GII rankings. Moreover, Uzbekistan has committed to improving its GII position and strives to enter the world’s top 50 by 2030. Meanwhile, no GII data were available for Turkmenistan and Afghanistan42 for the selected period (2019-2023).

49. The evaluation also analyzed the Terms of Reference (ToR) for the sessions of the SPECA WG on ITSD (a total of four sessions were held)43 conducted under the Project’s framework. An in-depth analysis of the ToR proved that the SPECA WG on ITSD was mandated, among other things, to:

➢ […Provide a platform for supporting progress towards achieving the Sustainable Development Goals with a focus on the SDGs directly related to innovation and technology development (including, but not limited to SDG 9 on industry, innovation and infrastructure; SDG 8 on decent work and economic growth; SDG 13 on combatting climate change and its impacts; and SDG 17 on partnerships).

➢ Improve awareness of SPECA countries on the innovation and technology development related SDGs, provide information and share best practice experiences with regard to the implementation of these SDGs, taking into account interlinkages between the SDGs…]44

50. The ToR of the SPECA WG on ITSD also outlined the WG’s composition. Each SPECA country45 had to nominate national representatives meeting the following criteria: “[…senior officials involved in the formulation and implementation or be aware of national policies and programmes, which affect national policies related to innovation and technology development in their respective countries and able to effectively follow up the recommendations and decisions of the WG on ITSD…]. Further desk research and in-depth interviews validated that the sessions of the SPECA WG on ITSD were attended by senior-level state sector officials (e.g. directors, deputy directors, board members, senior advisors, senior managers, etc.) from the SPECA participating States. Key Evaluation Question 3: How relevant were the project activities vis-à-vis the programme of

work of the UNECE? What value has UNECE’s efforts added in this area?

51. Desk research and in-person interviews confirmed that the scope of the Project was determined by the mandate of the SPECA. The evaluation also acknowledged that the UNECE subprogramme 4, “Economic Cooperation and Integration,” implemented by the Economic Cooperation and Trade Division, provided a considerable contribution to the substantive work completed under the SPECA framework. That subprogramme aims to support UNECE member States in designing and implementing policies, processes, and initiatives to strengthen good governance practices and build innovative, competitive, and inclusive societies progressing towards the achievement of the SDGs in the areas of innovation policy and public-private partnership (PPP). The figure below (Figure 4) delineates the relevance of the Project’s activities with regard to the overall scope and structure of the UNECE.

41 Global Innovation Index 2019, WIPO, 2019.

42 Officially, the Islamic Emirate of Afghanistan. 43 1st session of the SPECA WG on ITSD (30 July 2020); 2nd session of the SPECA WG on ITSD (20-21

October 2021); 3rd Session of the SPECA WG on ITSD (20 July 2022); 4th session of the SPECA WG on ITSD

(18-19 October 2023). 44 UNECE (unescap.org). 45 SPECA participating States: Afghanistan, Azerbaijan, Kazakhstan, Kyrgyz Republic, Tajikistan, Turkmenistan, and

Uzbekistan.

Page 30 of 84

Figure 4: Conceptual Roadmap of the Project and its Link to the UNECE’s Mandate

A multilateral platform that facilitates

greater economic integration &

cooperation among its member States and

promotes sustainable development and

economic prosperity.

Economic Cooperation

and

Trade Division

Contribution

Sessions of the SPECA WG on ITSD

16th Session of the SPECA Governing Council

Expert Group Meeting

Capacity building meetings & workshops

Side events & Study tours

Source: The Evaluation Dataset, 2024.

52. Four sessions of the SPECA WG on ITSD were conducted during the Project. Notably, the SPECA WG on ITSD is a subsidiary body within the governing structure of the SPECA, created on the basis of the decision of the SPECA Governing Council in November 2019. The sessions of the SPECA WG on ITSD served different objectives including:

➢ Delivering presentations on the developments in the area of innovation and technology for sustainable development in the SPECA participating States.

➢ Giving presentations on the status of digital connectivity and e-resilience as a foundation of infrastructure during the COVID-19 pandemic and related Project implementation progress.

➢ Discussing the SPECA participating States' efforts to harness innovation and enhance regional cooperation in the area of innovation and supporting implementation of the SPECA Innovation Strategy for Sustainable Development, approved by the SPECA Governing Council in November 2019.

➢ Endorsing the implementation of the Action Plan of the SPECA Innovation Strategy for Sustainable Development and getting the SPECA participating States to comment on and contribute to the activities of the UNECE and the UN ESCAP in support of the Innovation Strategy and the Action Plan.

➢ Presenting the main findings and recommendations of the following UNECE handbooks: “Business Incubators for Sustainable Development in the SPECA Subregion” (2021) 46 ;

46 Business Incubators for Sustainable Development in the SPECA Sub-region | UNECE.

UNECE sub-programme 4:

“Economic Cooperation and

Integration”

The United Nations Special

Programme for the Economies of

Central Asia (SPECA)

UNDA Project 2023N:

“Strengthening Innovation Policies for

SPECA Countries in Support of the 2030

Agenda for Sustainable Development”

Page 31 of 84

“Supporting Innovative High-Growth Enterprises in the SPECA Subregion” (2022)47; and “New Innovation Policy in the SPECA Subregion” (2023)48.

➢ Issuing presentations on the upcoming activities of the SPECA Network of Business Incubators and Accelerators for Sustainable Development targeted at promoting a circular economy.

53. Under the Project’s framework, in November 2019, the UNECE organized the 16th Session of the SPECA Governing Council that adopted 13 decisions, including the decision on the “Tashkent Statement.” The latter was introduced to promote sustainable transport and trade, using the UN’s legal norms, standards, best practice recommendations, and other tools to support sustainable development and a circular economy in the region. 49 In addition, the SPECA Governing Council approved the Action Plan of the SPECA Innovation Strategy for Sustainable Development.

54. An Expert Group Meeting on Innovation and Technology for Sustainable Development was organized immediately after the 3rd session of the SPECA WG on ITSD (in 2022) to discuss the implementation of the SPECA Work Plan on ITSD 2022-2023 in coherence with the Action Plan 2022- 2026 of the Asia-Pacific Information Superhighway. The Expert Group Meeting also served the purpose of reviewing the UNECE tools supporting the implementation of the SPECA Innovation Strategy for Sustainable Development.

55. Furthermore, according to key stakeholders, the scope of the Project was aligned with other strategic activities of the UNECE such as the Innovation for Sustainable Development Reviews (I4SDRs)50 and Environmental Performance Reviews (EPRs)51 conducted by the UNECE for its SPECA participating States as per their requests. The table below (Table 11) presents a snapshot of the latest I4SDRs and EPRs conducted for SPECA participating States.

Table 11: I4SDRs and EPRs requested by SPECA participating States

SPECA Country I4SDRs EPR

Status Year Status Year

The Republic of Azerbaijan Not conducted N/A Conducted 2022

The Republic of Kazakhstan Conducted 2012 Conducted 2019

Kyrgyz Republic Conducted 2019 Conducted 2022

The Republic of Tajikistan Conducted 2015 Conducted 2017

The Republic of Turkmenistan Not conducted N/A Conducted 2012

The Republic of Uzbekistan Conducted 2022 Conducted 2020 Source: The Evaluation Dataset, 2024.

56. Moreover, according to the key stakeholders, the EPR team from the UNECE Environmental Policy Division (subprogramme 1: “Environment”) cooperated with the Economic Cooperation and Trade Division (subprogramme 4: “Economic Cooperation and Integration”) to integrate EPR

47 Supporting Innovative High-Growth Enterprises in the SPECA sub-region, UNECE Policy Handbook |

UNECE. 48 New Innovation Policy for transition economies in the SPECA subregion | UNECE. 49 https://unece.org/sites/default/files/2022-

01/Decisions%20of%20the%2016th%20session%20of%20the%20SPECA%20Governing%20Council_ENG.pd

f. 50 The I4SDRs focus on evaluating a country's national and regional innovation system in the context of

sustainable development. 51 The UNECE conducts EPRs to assess how well member countries are doing in terms of environmental

protection.

Page 32 of 84

recommendations into the I4SDR recommendations52 as both serve the purpose of supporting the member States in achieving and monitoring review-relevant SDGs.

57. In addition, with regard to the UNECE’s efforts to add value to the Project’s implementation, key stakeholders highlighted the following main strategic contributions:

➢ Participation of key UNECE personnel and experts in, and their contextual contribution to, the Project’s activities.

➢ The UNECE assumed a specific niche in the SPECA participating States by supporting policy and analytical work, and facilitating information sharing and capacity building in particular thematic areas (e.g., the circular economy) not addressed by other international organizations in the targeted countries.

➢ The UNECE provided a collaborative platform engaging national and international stakeholders and experts, enabling them to address national and regional agendas related to the development of innovation ecosystems.

58. The evaluation also validated that the UNECE, in addition to providing thematic expertise, secured high-level political support for the Project. For example, UNECE and UN ESCAP senior-level staff (namely, the Deputy Executive Secretary, SPECA Coordinator at the UNECE and the Deputy Executive Secretary of the UN ESCAP and SPECA Coordinator at the UN ESCAP) took part in the 12th session of the SPECA Governing Council along with high-level representatives of Afghanistan, Azerbaijan, Kazakhstan, Kyrgyz Republic, Tajikistan, Turkmenistan, and Uzbekistan (the Project’s beneficiary countries). The 12th session of the SPECA Governing Council was followed by the 2017 SPECA Economic Forum on “Innovation for the SDGs in the SPECA Region” (held in Dushanbe on 5-6 December 2017) during which the Governing Council highlighted the importance of developing an innovation strategy for the SPECA region to achieve the SDGs.

Key Evaluation Question 4: To what extent were gender, human rights, disability perspectives and

climate change considerations integrated into the design and implementation of the Project? How

can these perspectives be better included in future projects design and implementation?

59. In-person interviews and online mini-survey results revealed varying opinions about the extent to which gender equality and mainstreaming, human rights, disability inclusion, and climate change dimensions were integrated into the Project. Many respondents of the online mini-survey expressed a belief that all of the above-mentioned dimensions were integrated to a great extent. For example, 54.5 percent (12 out of 22) were certain that gender equality and mainstreaming aspects were presented and integrated to a great extent, and 45.5 percent (10 out of 22) claimed they were integrated to a moderate extent. Very few respondents believed that all of the dimensions cited in the question were not reflected in the Project’s design and implementation (Figure 5).

52I.e., the recommendations related to innovative policies in environmental management and environmental

policies.

Page 33 of 84

Figure 5: Opinions of Mini-survey Respondents on Gender Equality and Mainstreaming, Human

Rights, Disability Perspectives, and Climate Change Considerations

Source: The Evaluation Dataset (2024).

60. The majority of the key stakeholders confirmed that in the future there would be a need to strengthen gender equality and mainstreaming, human rights, disability perspectives, and climate change dimensions in the Project’s scope. In addition, a few key stakeholders stated that while the UNECE had advocated a balanced approach to selecting speakers and participants, there was still a discrepancy between the gender-inclusive intentions and the actual distribution of responsibilities in the partnering national institutions in Central Asian countries, which resulted in an uneven gender representation at certain events. With regard to this, some key stakeholders claimed that engaging in dialogue with national partners was crucial to encourage more active participation of female leaders. Some also asserted that additional support, resources, and training would be required to empower women and build a culture of gender equality at decision-making levels. Key stakeholders also mentioned that the beneficiary countries were encouraged to nominate female participants to attend the Project’s events. 61. The evaluation acknowledged the receipt of very diverse feedback from the in-depth interviews regarding the extent to which the Project had integrated gender equality and mainstreaming, human rights, disability, and climate change measurements. Many interviewees were certain that the Project integrated gender equality and mainstreaming and climate change dimensions but was lacking with regard to human rights and disability.

62. The evaluation noted that the project results framework did not apply gender, human rights, disability, and climate change-sensitive indicators. It did not allow the collection of comprehensive data to monitor and track progress in the implementation of the above-mentioned dimensions. Nevertheless, the Project team organized a side event with a specific focus on the gender mainstreaming agenda, and a side event53 to the 16th session of the UNECE Committee on Innovation,

53 Flyer – Side event “Promoting women’s entrepreneurship in transition economies for sustainable

development” | UNECE.

Page 34 of 84

Competitiveness and Public-Private Partnerships promoted women’s entrepreneurship in transition economies for sustainable development.

63. Furthermore, desk research validated that the documents/reports produced during the Project referred to some of these dimensions but lacked references and/or recommendations related to others. For example, the Work Plan for the SPECA WG on ITSD approved at the 4th Session of the SPECA WG on ITSD incorporated several activities to be conducted between October 2023 and December 2025. One of these activities was focused on the climate change agenda, with the following title: “Capacity development to reduce and mitigate disaster risks in endorheic (inland) water basins in Central Asia.” Meanwhile, the policy document produced under the “Development of Synergies Between the NBIASD and Circular STEP, UNECE’s Stakeholder Engagement Network for Circular Economy Road Map” project outlined the link between the circular economy and climate change agenda, as well as highlighting the priorities of the targeted countries in addressing climate change.

5.2 Effectiveness

Finding 8: The evaluation validated that the Project’s design and implementation, including the

sessions and the composition of the SPECA WG on the ITSD and expert groups, effectively addressed

the needs of the beneficiary countries.

Finding 9: The evaluation found that the Project’s objectives and results were achieved at the outcome and output levels. At the same time, the evaluation observed overlaps between the outcome level indicator IA1.1 (“An action plan for the SPECA innovation strategy with performance indicators aligned with the SDGs is developed and agreed with the SPECA participating States and adopted by the SPECA Governing Council”) with the outcome level indicator OP1.4 (“Produce a final draft of the SPECA Innovation Strategy Action Plan and secure its formal approval by the SPECA Governing Council, with the attendance of the national focal points and experts”). Finding 10: The evaluation validated the thematic coherence of the Project activities with the interventions (which vary per country) funded by other international donors (including UN agencies), addressing the development needs of the targeted countries. Finding 11: The Project experienced some external challenges (i.e., the COVID-19 pandemic and political changes in the host and beneficiary countries). The Project team applied an agile management approach to overcome such obstacles and to deliver the planned results.

Key Evaluation Question 5: To what extent were the project design and set-up effective for

meeting the needs of the beneficiary countries?

64. To address this evaluation question, the evaluation reviewed the organizational and decision- making practices applied under the Project. Pursuant to the Prodoc, both were clearly defined among the partnering institutions and partners. Thus, the UNECE Secretariat was fully responsible for the Project’s administration and implementation. In the meantime, UN-ESCAP played the role of a committed equal partner leading or co-leading several activities and/or providing a venue for selected activities. The evaluation confirmed the above-mentioned arrangements over the course of the Project’s implementation. 65. Furthermore, according to the Prodoc, the UNECE engaged with other UN organizations through the Inter-Agency Task Team on Science, Technology, and Innovation (TTSTI). The evaluation

Page 35 of 84

found no evidence of the meetings of the Inter-Agency TTSTI being conducted under the Project framework. It did, however, confirm that meetings had taken place between the SPECA WG on ITSD and the Inter-Agency TTSTI. Elsewhere, the evaluation noted other strategic meetings having been conducted under the Project such as those of the SPECA WG on ITSD and another of the SPECA Governing Council.

66. According to key stakeholders, the existing intergovernmental structures such as the SPECA Governing Council and the SPECA WG on ITSD oversaw the progress made in implementing the SPECA Innovation Strategy for Sustainable Development. In addition, desk research validated their crucial role in addressing the needs of the beneficiary countries: “The SPECA Governing Council is the supreme management body which provides overall policy guidance to the work carried out within the SPECA framework, oversees the work of the SPECA Project Working Groups, takes stock of progress achieved, identifies priorities for the future, and approves the SPECA’s work plans. It is constituted of senior policymakers at the level of vice prime minister or minister from the SPECA countries, the UNECE, and the UN ESCAP executive secretaries.”54

67. Key stakeholders also reported that the SPECA WG on ITSD served as the main mechanism to ensure that the Project activities were aligned with the needs of the Project's targeted countries. For ease of reference, the SPECA WG on ITSD is a subsidiary body within the governing structure of the UN (SPECA) created by the decision of the SPECA Governing Council in November 201955. As reported, the SPECA WB on ITSD consists of “the senior policymakers, researchers, decision-makers, and implementers from the SPECA countries responsible for developing guidance and finding practical solutions to problems identified by the countries.”56

68. The evaluation also noted the holding of other strategic meetings such as: ➢ Task Force Meeting on the Roadmap for the Development of the Innovation Ecosystem of

Kyrgyz Republic until 2025. ➢ Expert Group Meeting on Innovation and Technology for Sustainable

Development: Proposal for a SPECA Network of Business Incubators and Accelerators for Sustainable Development (NBIASD).

➢ A side event around the 70th session of the UNECE. 69. The evaluation validated that the events listed above were attended by representatives of key state agencies from the targeted countries to ensure that the consolidated decisions aligned with the national priorities, and to endorse follow-up activities. 70. Finally, all key stakeholders, that were consulted over the course of this evaluation, confirmed that the Project design and setup were efficient when it came to addressing the needs of the targeted countries.

Key Evaluation Question 6: To what extent were the project objectives and expected results

achieved?

71. To address this key question, the evaluation reviewed the Project’s results framework to assess the progress made against output and outcome level indicators.

54 Governing Council | UNECE. 55 Innovation and Technology for Sustainable Development | UNECE. 56 SPECA Working Group on Innovation and Technology for Sustainable Development | ESCAP (unescap.org).

Page 36 of 84

Outcome-level Achievements

72. The desk research validated the full achievement of all three outcome-level indicators by the time of the Project’s completion (Table 12). The baseline for the indicators IA1.1 and IA1.2 was reported to be zero as there was neither an action plan in place for the SPECA Innovation Strategy for Sustainable Development nor any joint initiatives conducted on innovation for sustainable development involving two or more SPECA participating States by the time of the Project’s commencement. Table 12: Progress Made Against Outcome-level Indicators

Intended Outcomes (OCs) Indicator of achievement (IA) at the start of the project

Indicator of achievement (IA) at the end of the project

OC1: Strengthened co- operation on innovation to promote sustainable development and deliver agenda 2030 in the SPECA subregion.

IA1.1: An Action Plan for the SPECA innovation strategy with performance indicators aligned with the SDGs is developed and agreed with SPECA countries and adopted by the SPECA Governing Council.

Fully Achieved

IA1.2: At least two joint initiatives on innovation for sustainable development involving two or more SPECA countries developed and endorsed by the beneficiary.

Fully Achieved

OC2: Enhanced capacity of national policymakers and stakeholders to design and carry out effective innovation policy and institutional reform.

IA2.1: At least 3 of the 7 SPECA countries designed at least one initiative to improve support for innovation for sustainable development.

Fully Achieved

Source: Evaluation Dataset and Final Report for the 12th Tranche of the Development Account, UNECE (May

2024).

73. By the time of the Project’s completion, the UNECE reported the finalization of the Action Plan for the SPECA Innovation Strategy for Sustainable Development. Initially, the draft action plan for implementing the SPECA Innovation Strategy for Sustainable Development 57 was prepared for consideration at the 2nd Session of the SPECA WG on ITSD that took place on 20-21 October 2021. Later, the final version of the action plan was approved at the 16th session of the SPECA Governing Council conducted in Tashkent (Uzbekistan) on 19 November 2021. 74. With regard to IA1.2, the UNECE reported the existence of two joint initiatives:

i. The SPECA Network of Business Incubators and Accelerators for Sustainable Development, endorsed by the beneficiary countries.

ii. A proposal to establish the Digital Solutions Centre for Sustainable Development (DSCSD) for the SPECA participating States and Mongolia58.

75. Moreover, according to the key stakeholder, “The project’s activities contributed to the

development of an initiative group that launched a series of events to improve the innovation

57 2 Action Plan_SPECA Innovation Strategy for Sustainable Development_Draft_ENG.pdf (unece.org). 58 https://www.unescap.org/sites/default/d8files/event-documents/D5a-RoadmapDSCSD-E.pdf,

Page 37 of 84

ecosystem and sustainable development goals.” The stakeholder also reported on the creation of the

Kazakstan Association of the Universities Business Incubators and Accelerators (KAUBIA,

www.kaubia.kz). Overall, 22 organizations are members of the KAUBIA: 19 universities, two venture

capital funds, and one information technology hub. According to the stakeholder, “the Association

disseminates the knowledge acquired during the UNECE training to middle managers and interacts

with the government to create initiatives for the development of innovation.”

The SPECA Network of Business Incubators and Accelerators for Sustainable Development endorsed by

the beneficiary countries.

76. At the 3rd session (on 20 July 2022) of the SPECA WG on ITSD a proposal was presented to set up the SPECA Network of Business Incubators and Accelerators for Sustainable Development.59 This network was designed to provide a platform for the beneficiary countries to exchange solutions and best practices supporting innovative entrepreneurship through the development of business incubators (BIs) and business accelerators (BAs) for sustainable development in the SPECA countries. In this regard, the anticipated activities of the network included workshops, roundtables, trainings, study tours, and other information exchange initiatives covering broad areas (i.e., the transition to a circular economy, green entrepreneurship, and the commercialization of innovation and technology). In addition, the UNECE produced several policy papers and conducted trainings on the policy topics covered; all were instrumental for the network with respect to building the capacity of BIs and BAs. Specific materials included: ➢ A policy handbook - “Business Incubators for Sustainable Development in the SPECA Subregion”

(2021) (English60 and Russian61). ➢ A policy handbook - “Supporting Innovative High-growth Enterprises in the SPECA Subregion”

(2023) (English62). ➢ A policy paper - “New Innovation Policy in the Transition Economies of the SPECA Subregion”

(2024) (English63).

77. The network reported delivering online training sessions on the above-mentioned policy agendas: ➢ A webinar - "Promoting Innovative Entrepreneurship through Educational Programs in the

SPECA Subregion" (19 September 2022)64. ➢ Online training - "Effective Operation and Development of a Business Incubator and Start-up

Accelerator" (27-29 September 2022)65.

Establishment of the Digital Solutions Centre for Sustainable Development (DSCSD) for SPECA

participating States and Mongolia.

78. In October 2023, at the 4th session of the SPECA WG on ITSD, the participating parties were informed about the initiative of the Government of Kazakhstan, through the UN ESCAP resolution

59 https://www.unescap.org/sites/default/d8files/event-documents/D8a-ConceptNote-SPECA-

NetworkBusinessIncubatorsAccelerators-E.pdf. 60 https://unece.org/sites/default/files/2023-

10/3.%20Business%20incubators%20for%20sustainable%20development%20in%20SPECA-2021-ENG_0.pdf. 61 https://unece.org/sites/default/files/2023-10/3R%20SPECA-2021_RU%20Corr_0.pdf. 62https://unece.org/sites/default/files/2023-03/2227804_E_ECE_CECI_33_WEB_144dpi.pdf. 63https://drupal-main-staging.unece.org/sites/default/files/2024-01/NIPTE-SPECA-2023-EN-

WEB%20SIGNED.pdf. 64 https://unece.org/sites/default/files/2024-

02/UNECE_report_webinar_SPECA_Network__19092022_ENG_0.pdf. 65 https://unece.org/info/SPECA/events/371190.

Page 38 of 84

79/1066 and resolution 80/167 to establish the DSCSD. In May 2024, the Government of Kazakhstan issued a concept note for the development of the DSCSD entitled “UN ESCAP Digital Solutions Centre for Sustainable Development.”68 The DSCSD was expected to meet the following two main purposes: ➢ Accelerating digital transformation towards a sustainable, inclusive, and resilient digital

economy and society; and ➢ Strengthening coordination and cooperation among multiple stakeholders in the digital

transformation process.

79. The Government of Kazakhstan committed to providing premises for the DSCSD, and in the DSCSD concept note69 it presented a collaboration model between the DSCSD and other relevant institutions as follows:

➢ The UN ESCAP Asian and Pacific Centre for Transfer of Technology (APCTT) in the area of digital technology transfer.

➢ The UN ESCAP Asian and Pacific Training Centre for Information and Communication Technology for Development (APCICT) to deliver training sessions for the representatives of beneficiary countries.

➢ The UN ESCAP Asian and Pacific Centre for Development of Disaster Information Management (APDIM) in the field of digital solutions for disaster management.

80. With regard to the outcome-level indicator IA2.1, the Project reported that three SPECA participating States (Azerbaijan, Kyrgyz Republic, and Uzbekistan) out of seven had designed national initiatives to support innovation for sustainable development.

➢ Azerbaijan: The UNECE supported the development of the innovation ecosystem in Azerbaijan by conducting three online consultations 70 (July-August 2023) with national policymakers on enhancing venture capital in Azerbaijan.

➢ Kyrgyz Republic: Under the Project framework, the UNECE supported the State Agency for Intellectual Property and Innovation of Kyrgyz Republic (Kyrgyzpatent) in creating the Roadmap for the Development of the Innovation Ecosystem of Kyrgyz Republic 71 and establishing an innovation center in Bishkek as well as BIs at 11 universities across the country. In this regard, the UNECE hosted the following two virtual capacity building activities:

o Capacity building: 1st Meeting of the Task Force on the Roadmap for the Development of Innovation Support Infrastructure in Kyrgyz Republic until 2025 (9 June 2021); and

o Capacity building: 2nd Meeting of the Task Force on the Roadmap for the Development of the Innovation Ecosystem of Kyrgyz Republic until 2025 (8 September 2021).

66 https://www.unescap.org/sites/default/d8files/event-documents/ESCAP_RES_79_10_E_0.pdf. 67

https://documents.un.org/doc/undoc/gen/240/021/6e/pdf/2400216e.pdf?token=q0LJDmiM00bme5lp1x&fe=true. 68

https://acpr.unescap.org/specialsessions/documents/Explanatory_Note_DSC_for_SD_Kazakhstan_Ver_7May20

24.pdf. 69

https://acpr.unescap.org/specialsessions/documents/Explanatory_Note_DSC_for_SD_Kazakhstan_Ver_7May20

24.pdf. 70 https://unece.org/sites/default/files/2024-

02/Questions%20and%20Answers%20-%20Enhancing%20Venture%20Capital%20in%20Azerbaijan.pdf. 71 Roadmap_ Development of the Innovation Ecosystem in Kyrgyzstan_RUS.pdf (unece.org).

Page 39 of 84

➢ Uzbekistan: The UNECE supported the preparation of the Innovation for Sustainable Development Review of Uzbekistan.72 Furthermore, the country established the Agency for Innovative Development, under the newly restructured Ministry of Higher Education, Technology and Innovation, and adopted the new National Science, Technology and Innovation Policy Strategy for 2022-2030.

Output-level Achievements

81. The evaluation acknowledged that while the results framework did not refer to output-level indicators, it did allow for the measurement of outputs in the “output description” section. The evaluation confirmed the achievement of all eight outputs reported in the results framework (Table 13).

Table 13: Progress Made Against Outputs

Output number

Output description Output Status

OP1.1 Develop a subregional gap analysis covering the 7 SPECA countries to inform the first Action Plan for the SPECA Innovation Strategy.

Achieved

OP1.2 Organize a subregional workshop to present and validate the gap analysis.

Achieved

OP1.3 Organize a subregional workshop to develop an Action Plan and its performance indicators and develop joint activities between countries on innovation for sustainable development.

Achieved

OP1.4 Produce a final draft of the SPECA Innovation Strategy Action Plan and secure its formal approval by the SPECA Governing Council, with attendance of the national focal points and experts (a session within the framework of the SPECA Economic Forum and the SPECA Governing Council).

Achieved

OP1.5 Organize a subregional workshop to measure the implementation progress of the Action Plan based on its key performance indicators.

Achieved

OP2.1. Develop training materials on good practices on innovation policies and institutions for sustainable development, including technology transfer and science and technology parks (based on activities with ESCAP and IATT on science, technology, and innovation under A2.1).

Achieved

OP2.2 Within the framework of a national capacity building agreement with 3 SPECA countries, organize 6 national workshops (2 for each country) on a specific topic to address key issues at the national level identified by the gap analysis as part of national consultations carried out under OP1.1, as reflected in the SPECA Innovation Strategy Action Plan. In parallel with these events, UNECE and ESCAP will provide targeted support to three countries on three specific issues.

Achieved

OP 2.3 Development and launch of the SPECA Network of Business Incubators and Accelerators for Sustainable Development, which was established to specifically assist innovative entrepreneurship support institutions in the SPECA subregion.

Achieved

Source: The Evaluation Dataset (2024) and Final Report for the 12th Tranche of the Development Account,

UNECE (2024).

72 unece.org/sites/default/files/2022-06/9789211172966_I4SDR_UZBEKISTAN_2022_web_full%2Bcover.pdf.

Page 40 of 84

82. More specifically, under OP1.1, the Project conducted gap analyses/assessments for the targeted countries as follows:

➢ Science, Technology, and Innovations (STI) Gap Analysis of Afghanistan.73 ➢ Science, Technology, and Innovation (STI) Gap Assessment of Azerbaijan.74 ➢ Science, Technology, and Innovation (STI) Gap Assessment of Kazakhstan.75 ➢ Science, Technology, and Innovation (STI) Gap Analysis of Kyrgyz Republic.76 ➢ Science, Technology, and Innovation (STI) Gap Analysis of Uzbekistan.77 ➢ Science, Technology, and Innovation (STI) Gap Analysis of Tajikistan (English 78 and Russian79

versions). ➢ Science, Technology, and Innovation (STI) Gap Assessment of Turkmenistan.80

83. In addition, under OP1.1, the UNECE reported producing a background paper entitled “Prospects for SPECA Regional Cooperation on Innovation for Sustainable Development.”81

84. Under OP1.2, in November 2020, the UNECE delivered an online workshop entitled “Science, Technology and Innovation (STI) Gap Assessment of the SPECA Countries: Paving the Way for Action Under the SPECA Innovation Strategy for Sustainable Development”82 to present and validate the findings of the STI Gap Assessment papers produced for the SPECA participating States.

85. Under OP1.3, the UNECE organized a subregional workshop on innovation for sustainable development (2nd Session of the SPECA WG on ITSD).83 The UNECE presented a draft action plan for implementing the SPECA Innovation Strategy for Sustainable Development, and the WG on ITSD approved the submission of the draft action plan for consideration at the next session of the SPECA Governing Council.

86. Under OP1.4, the UNECE was expected to produce a final draft Action Plan for the SPECA Innovation Strategy for Sustainable Development and secure its formal approval from the SPECA Governing Council. The evaluation noticed a certain overlap between this output and outcome-level indicator IA1.1 (“An action plan for the SPECA Innovation Strategy for Sustainable Development with performance indicators aligning with the SDGs is developed and agreed with the SPECA participating States and adopted by the SPECA Governing Council”). At the same time, the evaluation validated the

73 https://unece.org/eci/documents/2021/03/reports/science-technology-and-innovation-sti-gap-analysis-

afghanistan. 74 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Azerbaijan_Report_Yulia%20Alieva.pdf. 75 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Kazakhstan_Report_Elena%20Shevchenko.pdf. 76 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Kyrgyzstan_Report_%20Aziz%20Soltobaev.pdf . 77 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Uzbekistan_Report_Nodira%20Kurbanbaeva.pdf . 78 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Tajikistan_Report_%20Bahodur%20Mengliev_0.pdf . 79 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Tajikistan_Report_%20Bahodur%20Mengliev_RUS.pdf. 80 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Turkmenistan%20Report_Yuriy%20Aronskyi.pdf . 81 https://unece.org/sites/default/files/2021-

10/UNECE_%20Background%20paper_SPECA_%20potential%20areas%20for%20cooperation%20on%20ISD

_2020_ENG.pdf . 82 SPECA_Webinar_ Report_ 26 Nov.pdf (unece.org) 83 https://unece.org/sites/default/files/2020-12/SPECA_Webinar_%20Report_%2026%20Nov.pdf.

Page 41 of 84

production of the Action Plan in two languages English84 and Russian.85 The Action Plan was approved at the 16th Session of SPECA Governing Council: “The Council approves the Action Plan for the SPECA Innovation Strategy for Sustainable Development. It expresses its gratitude to the UNECE and UN ESCAP secretariats and the SPECA participating countries for the development of this document.”86

87. Under OP1.5, the UNECE organized a subregional workshop (4th Session of the SPECA WG on ITSD) to measure the implementation progress of the Action Plan developed for the SPECA participating States. In this regard, the UNECE also produced an assessment report entitled “Assessment Report on the Implementation of the SPECA Innovation Strategy for Sustainable Development and its Action Plan.”87 At the 4th Session of the SPECA WG on ITSD, the WG also expressed its appreciation regarding the nomination of national focal points (by the SPECA participating States) to support the implementation of the Action Plan for the SPECA Innovation Strategy for Sustainable Development.

88. Under OP2.1, the UNECE produced/organized several handbooks (serving as a basis for follow- up trainings) and training materials/events, as follows:

➢ Online workshop - “Fostering Innovative Entrepreneurship through Educational Curricula in the SPECA Subregion”88 (19 September 2022).

➢ Online training for SPECA countries - “Effective Management and Development of Business Incubators and Accelerators in the SPECA Subregion”89 (27-29 September 2022).

➢ Online training for SPECA countries - “Development of Business Incubators and Start-up Accelerators: Training for University Management”90 (21 February 2023).

➢ Online training for the staff of BIs and BAs in the SPECA subregion - “Working with Venture Funds and Business Angels”91 (16 March 2023).

➢ A policy handbook - “Handbook on Business Incubators for Sustainable Development in the SPECA Subregion.”92

➢ A policy handbook - “Supporting Innovative High-growth Enterprises in the SPECA Subregion.”93

➢ A policy paper - “New Innovation Policy for Transition Economies in the SPECA Subregion.”94

84 https://unece.org/sites/default/files/2021-

11/SPECA%20Innovation%20Strategy%20for%20Sustainable%20Development_Action%20Plan_Draft%20for

%20GC_ENG_FINAL_0.pdf. 85 https://unece.org/sites/default/files/2021-

11/SPECA%20Innovation%20Strategy%20for%20Sustainable%20Development_Action%20Plan_Draft%20for

%20GC_RUS_FINAL.pdf. 86 https://unece.org/sites/default/files/2022-

01/Decisions%20of%20the%2016th%20session%20of%20the%20SPECA%20Governing%20Council_ENG.pd

f. 87 https://unece.org/sites/default/files/2023-11/Assessment%20Report_full%20first%20draft.pdf. 88 https://unece.org/speca/events/webinar-fostering-innovative-entrepreneurship-through-educational-curricula-

speca-sub. 89 Online training for SPECA countries " Effective management and development of business incubators and

accelerators" | UNECE. 90 Development of business incubators and start-up accelerators: training for university management (Развитие

бизнес-инкубаторов и стартап акселераторов – тренинг для менеджмента ВУЗов) | UNECE. 91 Working with venture funds and business angels: UNECE practical online training for staff of business

incubators and accelerators in the SPECA sub-region | UNECE. 92 https://unece.org/sites/default/files/2021-

10/Business%20incubators%20for%20sustainable%20development%20in%20SPECA-2021-ENG.pdf. 93 Supporting Innovative High-Growth Enterprises in the SPECA sub-region, UNECE Policy Handbook |

UNECE. 94 https://unece.org/sites/default/files/2024-01/NIPTE-SPECA-2023-EN-WEB%20SIGNED.pdf.

Page 42 of 84

89. Under OP2.2, the UNECE committed to “…organize 6 national workshops (2 for each country) on a specific topic to address key issues at the national level...” In the final report for the Project, the UNECE reported delivering six capacity building workshops and/or consultations: two in the Kyrgyz Republic, one in Uzbekistan, and three in Azerbaijan (Table 14). The UNECE arranged a workshop in Uzbekistan jointly with the National Office for Innovation Implementation and Technology Transfer for chief innovation officers from over 100 state-owned enterprises. The workshop aimed to enhance the efforts of the Uzbek government to foster a culture of innovation in the beneficiary state-owned enterprises. Meanwhile, the workshop in the Kyrgyz Republic was conducted as part of the 2nd Meeting of the Task Force on the Roadmap for the Development of the Innovation Ecosystem of the Kyrgyz Republic until 2025. Table 14: Project Activities Under OP 2.2

Country Event Title Implementation Date

Azerbaijan Three online consultations with local policymakers on enhancing venture capital in Azerbaijan95

July-August 2023

Kyrgyz Republic

Two capacity building activities to support development of a Roadmap for the Development of Innovation Support Infrastructure in the Kyrgyz Republic until 2025.96,97

September 2021

Uzbekistan National capacity building activities to help implement the recommendations of the UNECE Innovation for Sustainable Review of Uzbekistan98. These included UNECE capacity building workshop for Chief Innovation Officers on innovation for the circular economy99 and UNECE B2B capacity building “Innovation for the Circular Economy: Bridging Start-Ups and Corporations”.100

May 2023

Source: The Evaluation Dataset (2024).

90. Under OP2.3, the Project was expected to develop and launch the SPECA Network of Business Incubators and Accelerators for Sustainable Development (NBIASD). The evaluation confirmed the achievement of this output. First and foremost, the proposal to develop the SPECA NBIASD was presented at the Expert Group Meeting on Innovation and Technology for Sustainable Development in July 2022. At the 4th session of the SPECA WG on ITSD, the parties were updated on the progress of the creation of the SPECA NBIASD. In 2022-2023, the NBIASD organized several training sessions for BIs (Table 15). Table 15: Project Activities Under OP 2.3

Event Title Implementation Modality

Implementation Date

“Fostering Innovative Entrepreneurship through Educational Curricula in the SPECA Subregion”101

Online 19 September 2022

95 Capacity building: Enhancing venture capital in Azerbaijan, July-August 2023 | UNECE. 96 https://unece.org/info/SPECA/events/357216. 97 https://unece.org/info/SPECA/events/359822. 98 UNECE Innovation for Sustainable Development Review of Uzbekistan | UNECE. 99 https://unece.org/info/events/event/378783. 100 https://unece.org/speca/events/unece-b2b-conference-innovation-circular-economy-bridging-start-ups-and-

corporations. 101 Webinar : Fostering innovative entrepreneurship through educational curricula in the SPECA sub-region |

UNECE.

Page 43 of 84

“Effective Management and Development of Business Incubators and Accelerators"102

Online 27 - 29 September 2022

“Development of Business Incubators and Start-up Accelerators: Training for University Management”103

Online 21 February 2023

“Working with Venture Funds and Business Angels: UNECE Practical Online Training for Staff of Business Incubators and Accelerators in the SPECA Subregion”104

Online 16 March 2023

“Fostering Circular Solutions through Innovation”105 Online 3 April 2023 Source: The Evaluation Dataset (2024) and Final Report for the 12th Tranche of the Development Account,

UNECE (2024).

Key Evaluation Question 7: To what extent were the project activities coherent and harmonized

with those of other partners operating within the same context, particularly those of other UN

system entities?

91. Desk research identified other international donors engaged with the targeted countries to support innovation and the circular economy agenda (Table 16). The evaluation observed that the United Nations Development Programme (UNDP) supported the Accelerator Labs initiative, consisting of 91 labs in 115 countries, including Azerbaijan, Kyrgyz Republic, and Uzbekistan, to accelerate learning to achieve the SDGs by 2030. The Accelerator Lab network, a joint venture of the Qatar Fund for Development and the Federal Ministry for Economic Cooperation and Development of Germany, serves as an incubator tapping into local innovations to accelerate progress towards realizing the 2030 Agenda. The evaluation also identified targeted country-focused interventions supporting innovative approaches. However, the desk research confirmed that the strategic focus of international donor- funded programs and projects varied per country, as they were adjusted to fit the national needs and realities.

92. The evaluation also noted the UN Global Compact business incubator and accelerators initiative aimed to increase the capacity and awareness of business in support of achieving the SDGs.

The Republic of Azerbaijan

93. The United Nations Industrial Development Organization (UNIDO), with a financial contribution from the Government of Slovenia, has been working with the Government of Azerbaijan to strengthen the country’s innovation ecosystem and unlock its potential for digital transformation. In 2019, the Ministry of Transport, Communications and High Technologies of the Republic of Azerbaijan requested the UNIDO’s support in acquiring knowledge and experience and sharing the best practices in promoting the country’s innovation ecosystem and facilitating the competitiveness of innovative enterprises.

94. The European Commission (EC), through UNDP, supported the development and growth of civil society in the country. It did this by promoting a social entrepreneurship culture and building an enabling ecosystem and capacities for social entrepreneurship and social innovation through the “Developing Innovation-driven and Sustainable Civil Society in Azerbaijan” project.

The Republic of Kazakhstan

102 Online training for SPECA countries " Effective management and development of business incubators and

accelerators" | UNECE. 103 Development of business incubators and start-up accelerators: training for university management (Развитие

бизнес-инкубаторов и стартап акселераторов – тренинг для менеджмента ВУЗов) | UNECE. 104 Working with venture funds and business angels: UNECE practical online training for staff of business

incubators and accelerators in the SPECA sub-region | UNECE. 105 Fostering Circular Solutions through Innovation | UNECE.

Page 44 of 84

95. The Global Environment Facility (GEF) provided funding for the UNIDO to implement the “Global Cleantech Innovation Programme in Kazakhstan - Promoting cleantech innovation and entrepreneurship in small- and medium-sized enterprises for green jobs in Kazakhstan” project. It was aimed at supporting clean energy technology innovations and entrepreneurship in the country through the development of a cleantech innovation platform and accelerator programme.

96. Another notable intervention was the “Fostering Productive Innovation Project for Kazakhstan” supported by the World Bank. It incorporated the following four main thematic components:

➢ Development of a knowledge base for innovation aimed at promoting high-quality, nationally relevant research and development as well as advanced human capital activities through the provision of junior researcher group grants, senior scientist group grants, and PhD research and training grants.

➢ Innovation consortia through promoting collaboration among existing scientific research institutes and design bureaus, as well as scientific and engineering laboratories in Kazakhstan.

➢ Consolidation of the technology commercialization cycle through complementing the existing financial instruments and solutions suitable for different stages of start-up company development.

➢ Strengthening coordination of the national innovation system and enhancing the capacity of the existing institutional structures through complementing the existing financial instruments and solutions that fit the various stages of start-up company development.

Kyrgyz Republic and The Republic of Tajikistan

97. The Foreign, Commonwealth and Development Office (FCDO) allocated GBP 20,000,000 to implement the “Enterprise and Innovation Programme (EIP)” in Tajikistan and Kyrgyz Republic. The EIP has been implemented by a consortium led by DAI Global Ltd which included the University of Central Asia, VentureHive, IMKON, BizExpert, and International Alert. Under the EIP framework, DAI Global has reported establishing four hybrid “business innovation centres” (BICs) (three in Kyrgyz Republic, and one in Tajikistan), through which 40 business support programmes have been designed and delivered (26 in Kyrgyz Republic, and 14 in Tajikistan).

98. In January 2024, the EU transferred around EUR 3 million to the Cabinet of Ministers of the Kyrgyz Republic to support digital transformation. Specifically, this assistance is expected to promote the work of the Innovation Center of Digital Competencies under the High Technology Park, and promote the “Sanarip Aimak”/“Digital Aimak” system in all regions of the country. It should ensure the provision of electronic services to the population and enhance the country’s cybersecurity.

The Republic of Turkmenistan

99. UNDP is actively engaged with the Government of Turkmenistan in supporting the implementation of national priorities in the area of digital development outlined in the “Concept for the Development of the Digital Economy in Turkmenistan for 2019-2025” and the “State Programme for the Development of the Digital Economy in Turkmenistan for 2021-2025.” UNDP has also provided support through the “Assistance in the implementation of a pilot electronic data interchange system in Turkmenistan” project. It mainly addresses the issue of developing and deploying a pilot electronic data interchange system in Turkmenistan. The Republic of Uzbekistan

100. Uzbekistan benefits from various initiatives funded by donors including the United Nations Educational, Scientific and Cultural Organization (UNESCO), the Islamic Development Bank (IsDB), the

Page 45 of 84

Republic of Korea, and the UNIDO. In 2022, the Government of Uzbekistan, UNESCO, and the IsDB presented the new National Science, Technology and Innovation Policy of Uzbekistan (2022-2030), which was developed within the framework of a joint IsDB-UNESCO initiative entitled “Strengthening the inclusive Science, Technology and Innovation (STI) system in Uzbekistan.” For ease of reference, UNESCO systematically provides technical assistance and advisory, and capacity building activities to foster the development and governance of STI, and strengthen national and regional STI systems. 101. Furthermore, the Government of Korea sponsored the “UNIDO ITPO Korea Advisory Programme.” It aimed to address the issue of attracting foreign investment partners and accessing innovative technologies to become competitive in the global market. The programme contributed to the achievement of SDG 8 (“Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all”), SDG 9 (“Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation”), and SDG 17 (“Strengthen the means of implementation and revitalize the global partnership for sustainable development”).

Page 46 of 84

Table 16: International Donor-funded Initiatives to Develop the Innovation Ecosystem in the Targeted Countries

Beneficiary Country

International Donor

Intervention Title Implementatio n dates

Budget Intervention Goal/Objectives

The Republic of Azerbaijan

UNIDO and the Government of Slovenia

Development of an innovation ecosystem and support infrastructure, including a Digital Education and Innovation Centre in Azerbaijan.106

January 2020 - June 2023

EUR 199,725.50

The project objective is innovation ecosystem building, including fostering an environment for Start-Ups and Scale-Ups and skill upgrading, as well as raising awareness of the opportunities and challenges of the fourth industrial revolution (4IR) for pursuing Inclusive and Sustainable Industrial Development (ISID) in Azerbaijan.107

EU/UNDP “Developing Innovation- driven and Sustainable Civil Society in Azerbaijan”

November 2020 - November 2023

US$ 4,129,704

The project objective is to contribute to building a conducive environment for a vibrant, sustainable and innovation-driven civil society in Azerbaijan.

The Republic of Kazakhstan

GEF/UNIDO

“Global Cleantech Innovation Programme in Kazakhstan - Promoting cleantech innovation and entrepreneurship in small and medium-sized enterprises for green jobs in Kazakhstan”

January 2020 - December 2023

US$ 1,817,862

The project aimed to accelerate cleantech innovation and entrepreneurship by SMEs and start-ups, and to strengthen the cleantech innovation and entrepreneurship ecosystem of Kazakhstan.

The World Bank

“Fostering Productive Innovation Project for Kazakhstan”

December 2014 - October 2024

US$ 81.20 million

The project objective is to promote high-quality, nationally relevant research and commercialization of technologies.

Kyrgyz Republic

FCDO/DAI Central Asia Enterprise and Innovation Programme (EIP)

January 2018 - December 2024

£20,000,00 0

The programme aims at providing technical assistance to promote a stronger, diversified and more inclusive private sector in Kyrgyz Republic and Tajikistan. The Republic

of Tajikistan

106 https://open.unido.org/projects/AZ/projects/?_ga=2.93484062.229922146.1716975936-1759072182.1712852663. 107 Microsoft Word - UNIDO - ProDoc Azerbaijan Final_101219.docx.

Page 47 of 84

The Republic of Turkmenista n

UNDP

“Assistance in the implementation of a pilot electronic data interchange system in Turkmenistan”

March 2023 - December 2025

US$ 2,488,861

The project goal is to strengthen the institutional capacity of the Agency through the introduction of digital solutions in the work of the Agency in the provision of public services to citizens and assistance in the implementation of a pilot electronic data interchange system108.

The Republic of Uzbekistan

IsDB-UNESCO

Strengthening the inclusive Science, Technology and Innovation (STI) system in Uzbekistan.

Data not available online

Data not available online

Project goal is to enhance Uzbekistan's capacity to leverage science, technology, and innovation as drivers of sustainable development and inclusive growth.

Republic of Korea/UNIDO

UNIDO ITPO Korea advisory programme

January 2023 - December 2025

$613,335 This project aimed to address the issue of attracting foreign investment partners and in accessing innovative technologies to be competitive in the global market.

Source: The Evaluation Dataset (2024).

108 Assistance in the implementation of a pilot electronic data interchange system in Turkmenistan | UNDP Transparency Portal.

Key Evaluation Question 8: What were the challenges/obstacles (including COVID-19) to achieving the

expected results? How successfully did the Project overcome these?

102. Based on the in-person interviews and desk research, the evaluation identified several challenges that affected the implementation of the Project activities. It is important to note that all such challenges were external, and thus beyond the control of the Project team. First and foremost, due to travel-related restrictions associated with the COVID-19 pandemic, the Project team had to cancel some in-person activities in 2020-2022, and shift its capacity building activities to an online mode in 2022-2023.

103. Key stakeholders also mentioned staff turnover at the national institutions from beneficiary SPECA participating States (main counterparts) as a hindrance. Changes in this regard delayed the nomination of focal points, which was thus beyond the Project team’s control. Some key stakeholders also mentioned the structural changes in Azerbaijan (namely, the establishment of the Innovation and Digital Development Agency) and Uzbekistan (namely, the establishment of the new Ministry of Innovative Development). Others also cited regular changes to the delegates attending the sessions of the SPECA WG on ITSD.

104. Furthermore, due to the fluctuating political situation in Afghanistan, the country’s engagement in the Project’s activities became unfeasible as of 2021. Further political challenges arose because of the worsening security situation in Israel by the end of 2023 (Q4 2023). According to the Work Plan, the Golda Meir Mashav-Carmel International Training Center in Haifa (Israel) was expected to host a training session for beneficiaries of the Project in November 2023. However, with that being unfeasible, the UNECE replaced this event with a study tour in Georgia that was conducted in December 2023.

5.3 Efficiency

Finding 12: The quantitative and qualitative analysis of the technical and financial aspects of the Project

confirmed that the modified budget allowed for all the results to be achieved as planned at the outcome

and output levels.

Finding 13: The Project was completed with a six-month delay mainly caused by external factors. The Project team changed the delivery mode when appropriate to secure a more effective implementation of activities. Finding 14: The evaluation validated the occurrence of a shift in resource allocation after the COVID-19 pandemic, which resulted in additional operational opportunities under the Project framework (i.e., organizing regional webinars and producing extra policy handbooks). Furthermore, the budget analysis showed a significant increase in spending on budget items related to external contractual services and consultant and expert fees. Finding 15: The project staffing was limited to part-time engagement of the UNECE P-Level and G-level staff.

Page 49 of 84

Key Evaluation Question 9: Were the resources adequate for achieving the results?

105. Analysis of the actual expenditures of the Project against the budget allocated thereto at the design stage revealed that the budget utilization rate at the Project’s completion was 86.2 percent of the revised budget. More detailed analysis of the actual expenditure per budget item revealed significant changes in actual vs. planned expenditures for some budget items such as “Consultants and experts,” “Travel of staff,” “Contractual services,” and “Workshops/study tours (grants and contributions).” Indeed, the actual expenditure for the budget item “Workshops/study tours” was 29.8 percent of the planned amount, and there was a significant increase in actual expenditures compared to what was budgeted for in “Consultants and experts” (124.7 percent) and “Contractual services” (278.5 percent) (Table 17). Table 17: The Project Budget in USD (Planned vs. Actual)

Budget Item The budget

allocated at

the design

The revised

budget

Actual

expenditures

Funds

consumed (%

of revised

budget)

Other staff cost - General

Temporary Assistance (GTA)

25,000 24,000 23,959.00 99.83%

Consultants and experts 221,316 276,478 275,928.00 99.80%

Travel of staff 81,000 12,283 11,145.00 90.74%

Contractual services 23,500 65,598 65,446.00 99.77%

General operating expenses 16,500 6,475 5,545.00 85.64%

Supplies and materials 0 0 0 0.00%

Furniture and equipment 0 147 147.00 100.00%

Workshops/study tours (grants

and contributions)

116,000 50,003 34,623.00 69.24%

Total (total) 483,316 434,984109 416,793.00 86.24%

Source: The Project Financial Report (2024).

106. The above-mentioned budgetary changes were caused by the modalities of the Project having to

be altered due to COVID-19-related restrictions. In particular, travel, face-to-face workshops, and study

tours were replaced with virtual capacity building activities. This had affected the hosting fees, including

the remuneration for contracted consultants.

Key Evaluation Question 10: Were the results achieved on time and were all activities organized

efficiently?

107. The initial completion date for the Project was June 2023. However, it was extended (through a

no-cost extension) until December 2023. The desk research and in-person interviews confirmed that the

delay in the Project’s completion was caused by external factors (e.g., the COVID-19 pandemic and the

109 The UN Department of Economic and Social Affairs (UN DESA) reduced the project budget to $434,984 during the

course of the project (in 2022).

Page 50 of 84

challenging political environment in Israel) that affected the actual delivery and/or mode of the capacity

building activities. Yet, the Project team applied an agile management approach to adjust the Work Plan

to the changing circumstances.

108. All stakeholders that were consulted over the course of this evaluation confirmed their

satisfaction with the efficiency of the Project activities (Figure 6). Several key stakeholders mentioned

that these were well planned as they included an initial analysis of the existing situation/processes to

identify strategic gaps/solutions which served as a foundation for follow-up activities. At the same time,

some key stakeholders referred to the preliminary work conducted before the Project’s commencement

that was logically integrated into the Project’s design and implementation.

109. Overall, about 57 percent of the respondents (13 out of 23) confirmed that the Project activities

met their expectations to a great extent, and 43 percent (10 out of 23) claimed that the activities met

their expectations.

Figure 6: Feedback from Mini-survey Respondents on their Satisfaction with Project Events

Source: Evaluation Dataset (2024).

110. Many key stakeholders also mentioned that while online capacity building events were useful in

terms of providing information on principles and practices, face-to-face meetings and events allowed for

continuous information and knowledge sharing in an informal environment (after the completion of the

event). In this regard, many beneficiaries referred to continuing open dialogue and brainstorming with

the host country representatives and the invited experts and guests, leading to more effective idea

generation through the sharing of diverse perspectives.

Key Evaluation Question 11: To what extent were the resources used economically and how could the

use of resources be improved?

111. Analysis of the Project spending per budget item proved that the share for some budget items significantly increased after the COVID-19 pandemic broke out. In particular, the budget allocated to “Contractual services” and “Consultants and Experts” came to constitute 15.7 percent and 66.2 percent of the total budget, respectively (Figure 7).

0 (0%)

0 (0%)

13 (57%)

10 (43%)

0 2 4 6 8 10 12 14 16

Was worse than expected

Difficult to answer

Met my expectations to a great extent

Was about what I expected

Page 51 of 84

Figure 7: Budget Item share to the Total Budget (Planned vs. Actual)

Source: Evaluation Dataset (2024).

112. While the COVID-19 pandemic caused some delays and drove the need for some adjustments, the Project team efficiently reallocated resources (e.g., by reducing travel costs) to provide an alternative to the approved work plan, and carried out additional online subregional events that were not included in the original Work Plan. In addition, the UNECE published three new policy handbooks, which were not part of the approved Action Plan for the SPECA Innovation Strategy for Sustainable Development, as follows:

➢ UNECE Policy Handbook: Business Incubators for Sustainable Development in the SPECA Subregion.110

➢ UNECE Policy Handbook: Supporting Innovative High-growth Enterprises in the SPECA Subregion.111

➢ UNECE Policy Handbook: New Innovation Policy for Transition Economies in the SPECA Subregion.112

113. The evaluation acknowledged a high level of satisfaction among the beneficiaries of the Project

with regard to the quality, importance, and relevance of the Project activities, which were mainly carried

out after the COVID-19 pandemic. At the same time, key stakeholders highlighted the importance of

putting more effort, and allocating more of the budget, toward organizing face-to-face meetings and

events focusing on lessons learned, such as by introducing case studies of specific reforms in partnering

countries and sharing experiences of specific challenges faced and practical solutions applied.

114. The evaluation also noted the Project staffing cost share in relation to the total budget of the

Project constituted 5.2 percent at the design stage and 5.7 percent by completion. The UNECE part-time

assigned one P-level staff to administer the Project.113

110 unece.org/sites/default/files/2021-10/Business incubators for sustainable development in SPECA-2021-ENG.pdf. 111 https://unece.org/sites/default/files/2023-03/2227804_E_ECE_CECI_33_WEB_144dpi.pdf. 112 https://unece.org/sites/default/files/2024-01/NIPTE-SPECA-2023-EN-WEB%20SIGNED.pdf. 113 There was a part-time engagement of G-level staff as well.

5,2%

45,8%

16,8%

4,9%

3,4%

0,0%

0,0%

24,0%

5,7%

66,2%

2,7%

15,7%

1,3%

0,0%

0,0%

8,3%

0,0% 20,0% 40,0% 60,0% 80,0%

Other staff cost - General Temporary Assistance (GTA)

Consultants and experts

Travel of staff

Contractual services

General operating expenses

Supplies and materials

Furniture and equipment

Workshops/study tours (grants and contributions)

Percent of the total expenditures Percent of the total budget allocated

Page 52 of 84

5.4 Sustainability

Finding 15: The Project document incorporated a strategy to sustain the results and applied those

directly related to the Project’s implementation. Yet, the Project reports and in-person interviews

demonstrated a lack of formal commitment on behalf of the donor community to build on the Project

results.

Finding 16: The evaluation confirmed the interest of the beneficiary countries and a sense of ownership with regard to the Project results (i.e., the Action Plan for the SPECA Innovation Strategy and SPECA Network of Business Incubators and Accelerators for Sustainable Development). At the same time, the evaluation acknowledged that several strategic challenges directly affected the sustainability, scale-up, and replication of the Project results, including the lack of financial resources, political influences from neighboring countries, changes in political agenda, staff turnover, and the shortage of a structured knowledge transfer system in beneficiary countries. Finding 17: The UNECE and SPECA participating States accepted there is a need to sustain economic trends

and investment in the SPECA region and agreed to set up the SPECA Multi-Partner Trust Fund (MPTF)

under the UN’s management. The evaluation also acknowledged the proactive actions of the Secretariat

to raise funds and continue supporting SPECA participating States in strengthening their innovation

policies and putting more focus on gender equality and the empowerment of women and girls.

Key Evaluation Question 12: What measures were adopted to ensure that project outcomes would

continue after the Project ended and to what extent have these measures addressed the existing risks

for sustainability?

115. The evaluation acknowledged that the Prodoc incorporated an exit strategy to sustain the achieved results after the Project’s completion. Its exit strategy included but was not limited to:

➢ Making an essential contribution to secure the agreement on an action plan. ➢ Piloting selected activities in line with the action plan and national priorities. ➢ Starting to engage with all countries to explore areas of existing or potential interest. ➢ Engaging systematically with potential donors and lenders to help countries to formulate new

project ideas that would likely attract funding. ➢ Strengthening long-standing regional coordination mechanisms set up under the SPECA WG on

ITSD that served as both a steering entity and a network of policymakers from the subregion.

116. The evaluation verified that measurements were taken for all of the above categories except for systematic engagement with potential donors and lenders. Meanwhile, no evidence was provided to demonstrate that the UNECE had helped the targeted countries to formulate new project ideas to attract donors, with the exception of the DSCSD and KAUBIA114 concepts developed as a result of the UNECE115 and UN ESCAP support.

114 Created by the initiative group that was formed under the Project. 115 This was an initiative of the Government of Kazakhstan, through the UN ESCAP resolution 79/10 and resolution 80/1 to

establish the DSCSD.

Page 53 of 84

117. The evaluation also acknowledged the participatory nature of the Project’s design and implementation. First and foremost, the Project’s initiation and design were fully aligned with the priorities of the targeted countries, as expressed and approved at the 12th session of the SPECA Governing Council in December 2017 in Dushanbe (Tajikistan) and at the 14th Session of the SPECA Governing Council in November 2019.

118. Moreover, the evaluation confirmed that all documents produced over the course of the Project were publicly available in English and Russian. All stakeholders reached out to in the evaluation confirmed that the thematic and policy documents produced under the Project were highly appreciated. A few of them highlighted a need at the local level to transfer knowledge of the conceptual aspects of the produced document further by using plainer language.

Key Evaluation Question 13: To what extent do the partners and beneficiaries ‘own’ the outcomes of

the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or

institutionalized?

119. The evaluation also took note of the implementation modalities of the Project (i.e., the sessions of the WG on ITSD) that secured the approval and interest of the participating parties, thereby securing their ownership. At the same time, in-person interviews also revealed that on some occasions the changes in focal points affected the agendas of the beneficiary countries and the consistency of the discussions and consultations to a certain extent.

120. All national stakeholders consulted during this evaluation expressed high appreciation for the achieved results. Many also highlighted that while there is a political will to replicate and scale-up the achieved results, that might not be feasible due to financial constraints.

121. Some stakeholders also mentioned that more engagement from the third sector (i.e., representatives of private companies, associations, and non-government institutions) in the activities of similar interventions would strengthen the “buy-in” at national level and increase the chance of a follow- up improvement at country level as capacity building investment in the third sector would spur proactive cooperation and initiatives to improve the innovative ecosystem.

122. Overall, the key stakeholders reported the following challenges/risks to sustaining, replicating, and scaling-up the Project results:

➢ Availability of financial resources as some countries struggle to acquire the necessary funds to preserve the results achieved.

➢ The continuous necessity to build the capacity of human resources. ➢ Maintaining the level of involvement and participation of stakeholders and partners without a

coordination body and funds allocated for these purposes. ➢ Lack of a structured knowledge transfer mechanism to ensure that the knowledge and

methodologies are transferred locally. ➢ Lack of resources (financial and human) to maintain systematic monitoring and evaluation of the

Project results. ➢ Changes in the political or economic climate of a country or region that may affect the

sustainability of Project results.

Page 54 of 84

➢ Sociocultural factors possibly causing certain resistance or inconsistency of social and cultural norms and values that may in turn influence the implementation and sustainability of Project initiatives.

➢ Political influence from neighboring countries that are introducing their own initiatives and structures to replace those of the UNECE.

➢ The need for systematic technical support, advisory services, and updates to maintain the functionality of the systems and technologies created within the Project.

123. At the same time, it is important to highlight that at the SPECA Week (which was beyond the scope of the Project) that took place in Baku (Azerbaijan) in November 2023, the participating countries underlined the importance of collaboration in the SPECA and supported the establishment of the SPECA Multi-Partner Trust Fund (SPECA MPTF) under the UN’s management. Around the same time, the UNECE issued the ToR regulating the functional modalities of the SPECA MPTF, stating: “The SPECA MPTF is a UN- managed pooled fund mechanism established to operationalize stronger cooperation and integration in the SPECA region through programmatic interventions. The scope, theory of change, and governance mechanisms of the SPECA MPTF are specified in its Terms of Reference. The Fund will be governed by the SPECA UN Trust Fund Steering Committee (Azerbaijan). The co-chair(s) of the Steering Committee of the SPECA Trust Fund will present an update on the SPECA Trust Fund’s work to the decision-making bodies of the SPECA upon request, including annual reports, to the SPECA Governing Council.” 116

124. In addition, the evaluation verified the proactive fundraising of the UNECE team to address the requests and needs of the SPECA participating States. After the completion of the Project, UNECE drafted a new project proposal to be submitted to UN DESA117 to continue supporting SPECA participating States in strengthening their innovation policies through analytical work and the production of policy publications, capacity building activities, and facilitation of the digital transformation of innovative enterprises. This new project also considers addressing the challenges related to the gender divide and the exclusion of vulnerable groups from digital transformation.

116 SPECA ToR_EN.pdf (unece.org). 117 Under the UNDA 18th Tranche (2026-2029).

Page 55 of 84

6. Conclusions and Recommendations

125. The evaluation reached the following conclusions:

➢ The Project's participatory design and implementation modalities allowed for its complete alignment with the national and global development agendas. At the same time, the Project modalities were not heavily focused on the inclusion of the third sector (i.e., private sector, associations, and relevant non-state actors). Moreover, the operational, structural, and conceptual arrangement of the Project safeguarded its direct contribution to the UNECE’s overall mandate, and its work programme related to economic cooperation and integration and the environment. It also enabled the UNECE to leverage the best cross-sector expertise and resources to deliver the results in the most efficient manner possible.

➢ The absence of indicators to measure progress across gender, human rights, disability, and climate change dimensions resulted in an information gap and low awareness among beneficiary countries regarding the correlation between the innovation agenda and the above- mentioned dimensions. This might have lessened the impact of the Project deliverables on the capacity of the participating countries to prioritize advances in the relevant cross-sectoral processes and structures that would eventually contribute to the achievement of the relevant SDGs (i.e., SDG 5118, SDG 10119, and SDG 13120).

➢ While the Project fully achieved the planned results at the outcome and output levels121. It provided a platform for policy-level discussions within the framework of the SPECA Working Group on Innovation and Technology for Sustainable Development and the 16th session of the SPECA Governing Council and served as an information hub for SPECA participating States. AT eth same time, the overlap between the outcome level and output level indicators renders a distinctive qualitative assessment of the reported results impossible.

➢ The scope and objective of the Project exclusively focused on providing harmonized and strategic cross-country and cross-regional assistance across policy and individual capacity building levels.

➢ The Project team successfully applied agile management practices to secure an iterative and incremental implementation of the Project, focusing on flexibility, collaboration, and the satisfaction of the participating countries and institutions.

➢ ➢ The Project's human resource’s structure needed readjustment to secure advanced

backstopping (at the administration level) and continuous engagement of external thematic experts and service providers. Likewise, the implementation modalities of the Project highlighted the need for better-balanced face-to-face and online capacity building activities.

118 SDG 5: “Gender Equality.” 119 SDG 10: “Reduced Inequality.” 120 SDG 13: “Climate Action.” 121 Outcome level indicator IA1.1 (“An action plan for the SPECA innovation strategy with performance indicators

aligned with the SDGs is developed and agreed with SPECA countries and adopted by the SPECA Governing

Council”); Output level indicator OP1.4 (“Produce a final draft of the SPECA Innovation Strategy Action Plan and

secure its formal approval by the SPECA Governing Council, with the attendance of the national focal points and

experts (a session within the framework of the SPECA Economic Forum and the SPECA Governing Council)”).

Page 56 of 84

➢ The sustainability, replication, and scale-up of the Project’s results cannot be guaranteed unless the UNECE and SPECA participating States proactively address the external risks and challenges associated with financial limitations through the sustainability strategy to support the SPECA participating States across all relevant projects.

126. Based on its findings and conclusions, the evaluation issues the following four recommendations for supporting the SPECA participating States:

a. The evaluation recommends continuing to apply participatory and agile management modalities in future project design and implementation. It will allow the project teams to get real-time insights into project progress and potential issues, identify risks, and mitigate them early on, resulting in increased efficiency and effectiveness of the projects

b. The evaluation strongly advises engaging the private sector, associations, and non-state actors in project activities to strengthen in-country and regional collaboration across sectors. This would also secure a better balance of political support and a sense of ownership with industry- specific insights, data-driven evidence (providing the data crucial for policy decision-making), and in-country lobbying for policy changes.

c. The evaluation also heavily recommends revising the formulation of outcome and output level indicators to avoid overlap and ensure that complex concepts are broken down into measurable terms at the outcome level. This could be done in consultation with the relevant stakeholders to evaluate the validity and reliability of the indicators selected. Furthermore, the evaluation recommends incorporating gender, human rights, disability, and climate change indicators into the results framework to enable continuous reporting on progress made. Notably, some indicators could be developed at the output level (i.e. disability or gender indicators), and others can be elaborated based on outcome-level measurements (i.e. climate change indicators). However, it will be up to the project team to decide on the measurement layer (outcome or output) of the indicators selected, depending on the context of the upcoming projects and the potential availability and reliability of the data pertinent to the given indicator.

d. With regard to the implementation modalities of the capacity building activities, the evaluation recommends the development of online webinars with face-to-face meetings. In this regard, the UNECE is advised to arrange online national capacity building and awareness-raising events for a broader audience. However, face-to-face regional meetings and site visits would be more relevant for the sort of capacity building activities that would potentially lead to cross-sector strategic partnerships and provide first-hand experience of the practical implications and challenges of the approaches applied in the host countries.

e. The evaluation recommends reconsidering project staffing patterns to ensure smooth implementation and efficient continuity of the projects’ activities in case of emergencies. While the UNECE staff will remain engaged on a part-time basis, their level of engagement can be increased depending on the lifetime of the project. In this regard, the evaluation recommends including the budget lines associated with the salaries of the support and professional staff in the new project proposal. The availability of the qualified workforce assigned to the specific project is particularly vital for the efficient and smooth implementation of the project activities. For ease of reference, the budget share of both analytical and capacity building activities of the Project under evaluation constituted 86.6 % of the total budget).

Page 57 of 84

7. Lessons learned and good practices

127. In terms of the lesson learned, it is crucial to continue using participatory and agile management

approaches to foster stakeholder buy-in, increase adaptability, and improve decision-making. By involving

stakeholders at every step and adapting to changing circumstances, UNECE was able to navigate

challenges, such as the unexpected impact of the COVID-19 pandemic, and achieve the Project goals more

effectively. For example, by quickly transitioning to virtual meetings and online collaboration tools, the

Project team was able to maintain momentum and keep stakeholders engaged. It’s clear that participatory

and agile approaches are essential for future projects, ensuring that UNECE stays responsive, innovative,

and aligned with the needs of the member States. Overall, UNECE can maximize the benefits of

participatory and agile methodologies by incorporating techniques like design thinking and co-creation

workshops. These approaches can help us generate innovative ideas, solve complex problems, and build

strong relationships with key stakeholders.

128. Another important lesson learned is that in order to design effective indicators, it is crucial to

focus on clarity, relevance, feasibility, timeliness, flexibility, balance, and stakeholder involvement. By

ensuring that indicators are specific, measurable, achievable, relevant, and time-bound, and by aligning

them with project goals, UNECE can track progress, measure outcomes, and make informed decisions.

Additionally, considering the practicality of data collection, the balance of quantitative and qualitative

data, and stakeholder input is essential for designing a robust indicator framework.

129. Furthermore, it is important to avoid overlap between outcome and output indicators. This can

lead to confusion and hinder effective monitoring and evaluation. By carefully defining the hierarchy of

indicators and ensuring that they measure distinct levels of achievement, we can improve the quality and

usefulness of our data.

Page 58 of 84

Annexes

Annex 1 Evaluation ToR

Annex 2 Evaluation Matrix

Annex 3 Data Collection Tools

Annex 4 List of Stakeholders Interviewed

Annex 5 List of Documents Reviewed

Annex 6 Management Response and Recommendation Action Plan

Page 59 of 84

Annex 1: Evaluation ToR

TERMS OF REFERENCE

UNDA2023N: Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for

Sustainable Development.

I. Purpose

The purpose of this evaluation is to assess the extent to which the objectives of the UNECE project 2023N

“Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable

Development” were achieved.

The evaluation will assess the relevance, effectiveness, efficiency and sustainability of the project in

supporting the seven SPECA participating States in their efforts to spur innovation as a central driver of

the increasingly urgent transition from a low productivity and resource-intensive model of economic

development to knowledge-based and more sustainable economic growth, in line with the SDGs.

The evaluation will also assess any impacts the project may have had on progressing human rights, gender

equality, disability inclusion, climate change and disaster risk reduction in the context of this engagement.

The evaluation will finally look at the activities repurposed to address the impact of the COVID-19 crisis,

and assess, where relevant, UNECE’s COVID-19 early response through this project.

II. Background

The project aimed to support the seven SPECA participating States in their efforts to spur innovation as a

central driver of the increasingly urgent transition from a low productivity and resource-intensive model

of economic development to knowledge-based and more sustainable economic growth, in line with the

SDGs. It builds on a clear mandate: the SPECA participating States, recognizing this challenge, formally

decided at the 12th session of the SPECA Governing Council in 2017 to “Develop an innovation strategy

to promote sustainable development in the SPECA region” – a process that, with UNECE and ESCAP

support, has led to a final draft that will be submitted for approval at the 2019 SPECA governing council.

In this context, the project aimed to support and create sustainable further momentum around the first

steps towards putting this strategy into practice. Specifically, the objective was to strengthen institutional

capacities to harness innovation as a driver of sustainable development and regional integration.

The first target outcome was strengthened cooperation on innovation among the SPECA participating

States. Central to this outcome was to work with the SPECA participating States to put together a concrete

first action plan under the strategy, including several joint initiatives. This involved a detailed gap analysis

covering all seven countries to inform the development of the draft action plan, which was discussed in

detail at a subregional workshop before finalization and submission for approval. Towards the end of the

implementation period, a subregional workshop served to measure implementation progress based on

the indicators in the action plan. In addition to the national initiatives, two initiatives involving cooperation

among the SPECA participating States were established: (i) SPECA Network of Business Incubators and

Accelerators for Sustainable Development (NBIASD); (ii) Progress towards creation of a subregional Digital

Solutions Centre for Central Asia in Kazakhstan.

Page 60 of 84

The second outcome on enhanced capacity to design and carry out effective innovation policy and

institutional reform, aimed to address some of the leading constraints in putting the strategy into

practice among the SPECA participating States. This involved both a series of subregional seminars on

selected issues of common concern, as well as in-depth trainings for at least three countries.

Importantly, this project foresaw substantial flexibility to respond to country requests and priorities and

to build on existing or impending momentum.

ESCAP, as the co-lead in providing Secretariat support to the SPECA Working Group on Innovation and

Technology for Sustainable Development, played a role in most activities. There was an aim to engage the

donor community throughout, with the clear purpose of finding projects that contribute to the strategy

and the action plan that qualify for donor funding – sustaining the momentum beyond the closure of this

project.

III. Evaluation objectives, scope and questions

The evaluation will be guided by the objectives and indicators of achievement established in the results

framework of the project document. The evaluation will be conducted in Q4 of 2023. It will cover the full

implementation of the project, from January 2020 to December 2023 in the SPECA participating States of

the UNECE region, Azerbaijan, Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan and Uzbekistan.

The evaluation criteria are relevance, effectiveness, efficiency and sustainability.

Relevance 1. To what extent did the project respond to the priorities and needs of participating countries? 2. To what extent were the project activities consistent with global and regional priorities and

aligned with the SDGs? 3. How relevant were the project activities vis-à-vis the programme of work of the UNECE? What

value has UNECE’s efforts added in this area? 4. To what extent were gender, human rights, disability perspectives and climate change

considerations integrated into the design and implementation of the project? How can these perspectives be better included in future projects design and implementation?

Effectiveness 5. To what extent were the project design and set-up effective for meeting the needs of the

beneficiary countries? 6. To what extent were the project objectives and expected results achieved? 7. To what extent were the project activities coherent and harmonized with those of other partners

operating within the same context, particularly those of other UN system entities? 8. What were the challenges/obstacles (including COVID-19) to achieving the expected results? How

successfully did the project overcome these?

Efficiency 9. Were the resources adequate for achieving the results? 10. Were the results achieved on time and were all activities organized efficiently? 11. To what extent were the resources used economically and how could the use of resources be

improved?

Page 61 of 84

12. Sustainability 13. What measures were adopted to ensure that project outcomes would continue after the project

ended and to what extent have these measures addressed the existing risks for sustainability? 14. To what extent do the partners and beneficiaries ‘own’ the outcomes of the project? How is the

stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

IV. Evaluation approach and methodology

The evaluation will be conducted in accordance with: the ECE Evaluation Policy122; the Administrative

instruction guiding Evaluation in the UN Secretariat123; and the United Nations Evaluation Group (UNEG)

Norms and Standards for Evaluation 124 . Human rights and gender equality considerations will be

integrated at all stages of the evaluation125: (i) in the evaluation scope and questions; (ii) in the methods,

tools and data analysis techniques; (iii) in the findings, conclusions and recommendations of the final

report. The evaluator will explicitly explain how human rights, gender, disability, SDGs, and climate change

considerations will be taken into account during the evaluation.

The evaluator is required to use a mixed-method approach, including qualitative as well as quantitative

data gathering and analysis as the basis for a triangulation exercise of all available data to draw

conclusions and findings. The evaluator shall conduct online surveys and interview a wide range of diverse

stakeholders.

The evaluation should be conducted based on the following mixed methods to triangulate information:

1. A desk review of all relevant documents, including the project document and information on project

activities (monitoring data); materials developed in support of the activities (agendas, plans,

participant lists, background documents, donor reports and publications); proposed programme

budgets covering the evaluation period; project reports to the donor.

2. Online survey of key stakeholders and beneficiaries: the survey will be developed by the consultant

on her/his preferred platform.

3. Interviews (in-person and/or by telephone/video): the evaluator shall interview a wide range of

diverse stakeholders and beneficiaries including policymakers, representatives of the government,

international organisations, academia, civil society as applicable.

4. Observation of workshops and meetings, including the SPECA Working Group on Innovation and

Technology for Sustainable Development in Tashkent on 18-19 October 2023.

The evaluator will further elaborate on the evaluation methodology in the Inception Report that will

among others include the electronic survey questions, interview guide, and whether any of the six

countries will be selected for an in-depth assessment. The evaluation report will be written in English, will

consist of approximately 30 pages and will include an executive summary (max. 2 pages) describing the

evaluation methodology, key findings, conclusions and recommendations. The evaluator will also

produce an Evaluation Brief summarizing key evaluation findings, lessons learned and recommendations,

including through images and infographics.

122 UNECE Evaluation policy 123 ST/AI/2021/3 124 UNEG 2016 Norms and Standards for Evaluation 125 In line with UNEG Guidance contained in Integrating Human Rights and Gender Equality in Evaluations

Page 62 of 84

V. Evaluation schedule126

August 2023 ToR finalized

August 2023 Evaluator selected

September 2023 Contract signed. Evaluator starts the desk review

October 2023 Evaluator submits inception report including survey design

October 2023 Launch of data gathering, including survey and interviews

November 2023 Evaluator submits draft evaluation report and evaluation brief

November 2023 Evaluator submits final evaluation report and evaluation brief

VI. Resources and Management of the evaluation

An independent consultant will be engaged to conduct the evaluation under the management of the

Programme Management Unit (PMU). Payment will be made upon satisfactory delivery of work.

The PMU will manage the evaluation and will be involved in the following steps: Selection of the evaluator;

Preparation and clearance of the Terms of Reference; Provision of guidance to the Project Manager and

evaluator as needed on the evaluation design and methodology; Clearance of the final report after quality

assurance of the draft report.

The Project Manager, in consultation with the Division Director, will be involved in the following steps:

Provide all documentation needed for desk review, contact details, support and guidance to the

evaluation consultant as needed throughout the timeline of the evaluation; Advise the evaluator on the

recipients for the electronic survey and for follow-up interviews; Process and manage the consultancy

contract of the evaluator, along the key milestones agreed with PMU.

VII. Intended use / Next steps

The results of the evaluation will be used in the planning and implementation of future activities of the

UNECE Economic Cooperation and Integration Subprogramme. Findings of this evaluation will be used

when possible to:

• Improve direct project’s follow up actions, implementation of products by project beneficiaries

and dissemination of the knowledge created through the project.

• Assess the gaps and further needs of countries in the area of this project.

• Formulate tailored capacity building projects to strengthen the national capacity in enhancing

innovation.

The results of the evaluation will be reported to the inter-governmental Team of Specialists on Innovation

and Competitiveness Policies and the Committee on Innovation, Competitiveness and Public-Private

Partnerships, as well as with the Executive Committee if required.

Following the issuance of the final report, the Project Manager will develop a Management Response for

addressing the recommendations made by the evaluator. The final evaluation report, the management

response and the progress on implementation of recommendations will be publicly available on the

UNECE website.

126 Final timetable to be agreed following engagement of the evaluator

Page 63 of 84

VIII. Criteria for evaluators

The evaluator should have:

1. An advanced university degree or equivalent background in relevant disciplines.

2. Knowledge of and experience in sustainable economic development and/or innovation policy.

3. Relevant professional experience in design and management of evaluation processes with multiple

stakeholders, survey design and implementation, project planning, monitoring and management,

gender mainstreaming and human-rights due diligence.

4. Demonstrated methodological knowledge of evaluations, including quantitative and qualitative data

collection and analysis for end-of-cycle project evaluations, including demonstrated experience in

conducting questionnaires and interviews.

5. Fluency in written and spoken English and Russian.

Evaluators should declare any conflict of interest to UNECE before embarking on an evaluation project,

and at any point where such conflict occurs.

Annex 2: Evaluation Matrix

Evaluation Question Data collection methods

Data source

Relevance

To what extent did the Project respond to the priorities and needs of participating countries?

Desk research, online survey and KIIs

UNECE & UNESCAP reports (secondary data) and Project reports primary data). KIIs with the UNECE & UNESCAP staff: the project team and UNECE regional advisors. Survey among the participants (from the targeted countries) of the capacity-building activities. KIIs with the invited experts (key participants of SPECA network, international and national experts and speakers, and WG delegates.).

To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

Desk research and KIIs UNECE & UNESCAP reports (secondary data) and Project reports primary data). KIIs with the UNECE & UNESCAP staff. KIIs with the invited experts (key participants of SPECA network, international and national experts and speakers, and WG delegates.)..

How relevant were the project activities vis-à-vis the programme of work of the UNECE? What value has UNECE’s efforts added in this area?

Desk research and KIIs UNECE reports (secondary data) and Project reports primary data). KIIs with the UNECE project staff and regional advisors.

To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project? How can these perspectives be better included in future projects design and implementation?

Desk research, online survey and KIIs

UNECE & UNESCAP reports (secondary data) and Project reports (primary data). KIIs with the UNECE & UNESCAP staff. KIIs with the invited experts (key participants of SPECA network, international and national experts and speakers, and WG delegates.). Survey among the participants (from the targeted countries) of the capacity-building activities.

Effectiveness:

To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries?

Desk research, online survey and KIIs

The Project reports (primary data). Survey among the participants (from the targeted countries) of the capacity-building activities. KIIs with the UNECE & UNESCAP project staff.

To what extent were the project objectives and expected results achieved?

Desk research and KIIs The Project reports (primary data). KIIs with the UNECE & UNESCAP project staff.

To what extent were the project activities coherent and harmonized with those of other partners operating within the same context, particularly those of other UN system entities?

Desk research and KIIs The Project reports (primary data). KIIs with the UNECE & UNESCAP staff (including the project team).

Page 65 of 84

What were the challenges/obstacles (including COVID-19) to achieving the expected results? How successfully did the Project overcome these?

Desk research, online survey and KIIs

The Project reports (primary data). KIIs with the UNECE & UNESCAP staff (including the project team). KIIs with the invited experts (key participants of SPECA network, international and national experts and speakers, and WG delegates.). Survey among the participants (from the targeted countries) of the capacity-building activities.

Efficiency

Were the resources adequate for achieving the results? Desk research and KIIs The Project reports (primary data). KIIs with the UNECE & UNESCAP project team.

Were the results achieved on time and were all activities organized efficiently?

Desk research, online survey and KIIs

The Project reports (primary data). KIIs with the UNECE & UNESCAP project team

To what extent were the resources used economically and how could the use of resources be improved?

Desk research and KIIs The Project reports (primary data). KIIs with the UNECE & UNESCAP project team.

Sustainability

What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

Desk research, online survey and KIIs

The Project’s reports. KIIs with the UNECE & UNESCAP staff, and WG delegates. Survey among the participants (from the targeted countries) of the capacity-building activities.

To what extent do the partners and beneficiaries ‘own’ the outcomes of the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

Desk research, online survey and KIIs

The Project’s reports. KIIs with the UNECE & UNESCAP staff, and WG delegates. Survey among the participants (from the targeted countries) of the capacity-building activities.

Annex 3: Data Gathering Tools

Interview Protocol UNECE Project team

1. Name of Interviewee(s)

2. Organization

3. Position

4. Location

5. Date of Interview

Relevance

Q1 To what extent did the Project respond to the priorities and needs of participating countries at the design stage and/or implementation stage?

A1

Q2 Have you ever engaged UNECE Regional Advisor or United Nations Country Teams at the project design and implementation phases?

A2

Q3 To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

A3

Q4 Could you please the approaches taken to ensure that the Project responded to the priorities and needs of participating countries, regional and global priorities and SDGs?

A4

Q5 How relevant were the project activities vis-à-vis the programme of work of the UNECE?

A5

Q6 What value has UNECE’s efforts added in this area?

A6

Q7 To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project?

A7

Page 67 of 84

Q8 How can these perspectives be better included in future projects design and implementation?

A8

Effectiveness

Q9 To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries?

A9

Q10 To what extent were the project objectives and expected results achieved?

A10

Q11 What were the challenges/obstacles, internal or external, (including COVID-19) to achieving the expected results? How successfully did the Project overcome these?

A11

Q12 To what extent were the project activities coherent and harmonized with those of other partners operating within the same context, particularly those of other UN system entities? What kind of strategies/ approaches had been applied in this regard?

A12

Efficiency

Q13 Were the resources (HR, financial, etc.) adequate for achieving the results?

A13

Q14 Were the results achieved on time and were all activities organized efficiently?

A14

Q15 To what extent were the resources used economically and how could the use of resources be improved?

A15

Sustainability

Q16 What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

A16

Q17 To your opinion, to what extent do the partners and beneficiaries ‘own’ the outcomes of the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

Page 68 of 84

A17

Q18 What would you recommend to sustain, replicate and scale up the results of the project?

A18

Interview Protocol UNECE Regional Advisors

1. Name of Interviewee(s)

2. Organization

3. Position

4. Location

5. Date of Interview

Relevance

Q1 To what extent are you familiar with the goals and objectives of the project “Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable Development”?

A1

Q2 To what extent were you engaged in the project design and/or implementation?

A2

Q3 To what extent did the Project respond to the priorities and needs of participating countries127 at the design stage and/or implementation stage?

A3

Q4 To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

A4

Q5 Have you ever discussed the project objectives with the United Nations Country Teams at the project design phase?

A5

127 The Regional Advisor will be interviewed about the countries of their respective geographic coverage.

Page 69 of 84

Q6 How relevant were the project activities vis-à-vis the programme of work of the UNECE? What value has UNECE’s efforts added in this area?

A6

Q7 To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project? How can these perspectives be better included in future projects design and implementation?

A7

Effectiveness

Q8 To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries128?

A8

Effectiveness

Q9 What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

A9

Sustainability

Q10 What would you recommend to sustain, replicate and scale up the results of the project?

A10

Interview Protocol UNESCAP staff

1. Name of Interviewee(s)

2. Organization

3. Position

4. Location

5. Date of Interview

Relevance

Q1 To what extent are you familiar with the goals and objectives of the project “Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable Development”

A1

128 The Regional Advisor will be interviewed about the countries of their respective geographic coverage.

Page 70 of 84

Q2 To what extent were you engaged in the project design and/or implementation?

A2

Q2 To what extent did the Project respond to the priorities and needs of participating countries129 at the design stage and/or implementation stage?

A2

Q3 To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

A3

Q4 How relevant were the project activities vis-à-vis the programme of work of the UNECE? What value has UNECE’s efforts added in this area?130

A4

Q5 To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project? How can these perspectives be better included in future projects design and implementation?

A5

Effectiveness

Q6 To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries (Afghanistan in particular)?

A6

Q7 To what extent were the project objectives and expected results achieved?

A7

Q8 To what extent were the project activities coherent and harmonized with those of other partners operating within the same context, particularly those of other UN system entities?

A8

Q9 What were the challenges/obstacles (including COVID-19) to achieving the expected results? How successfully did the Project overcome these?

129 The Regional Advisor will be interviewed about the countries of their respective geographic coverage. 130 The questions will be addressed to the UNECE regional Advisors only.

Page 71 of 84

A9

Efficiency

Q10 Were the resources adequate for achieving the results?

A10

Q11 Were the results achieved on time and were all activities organized efficiently?

A11

Q12 To what extent were the resources used economically and how could the use of resources be improved?

A12

Sustainability

Q13 What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

A13

Q14 To what extent do the partners and beneficiaries ‘own’ the outcomes of the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

A14

Q15 What would you recommend to sustain, replicate and scale up the results of the project?

A15

A

No Interview Protocol Key participants of SPECA network, international and

national experts and speakers, Invited experts, WG

delegates.

1. Name of Interviewee(s)

2. Organization

3. Position

4. Location

5. Date of Interview

Relevance

Page 72 of 84

Q1 To what extent are you familiar with the goals and objectives of the project “Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable Development”

A1

Q2 To what extent did the Project respond to the priorities and needs of participating countries131 at the design stage and/or implementation stage?

A2

Q3 To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

A3

Q4 To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project? How can these perspectives be better included in future projects design and implementation?

A4

Effectiveness

Q5 To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries (Afghanistan in particular)?

A5

Q6 What were the challenges/obstacles (including COVID-19) to achieving the expected results? How successfully did the Project overcome these?

A6

Efficiency

Q7 Were the results achieved on time and were all activities organized efficiently?

A7

Sustainability

Q8 What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

A8

Q9 To what extent do the partners and beneficiaries ‘own’ the outcomes of the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

131 Afghanistan, Azerbaijan, Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan, and Uzbekistan.

Page 73 of 84

A9

Q10 What would you recommend to sustain, replicate and scale up the results of the project?

A10

Online Survey

Consent and Confidentiality Statement The UNECE cordially invites you to participate in the independent evaluation of the project “Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable Development” (the UNECE Project) conducted in 2020-2023. With this regard, we are sending you the online questionnaire to fill in. The information received through this questionnaire will be treated confidentially with no reference to the names of the respondents. Completing the survey will only take 15 minutes of your time. This survey will be available from April 15, 2024, through April 29, 2024. The UNECE would like to thank you in advance for your support and input.

1. Where do you work (please check the one that applies)

☐ Government - UNECE Member State

☐ Government - Non UNECE Member State

☐ UN Agency

☐ Non-Governmental Organization

☐ Independent Expert

☐ Academia

☐ Other (please specify): …….

2. Country (Please specify):

3. Gender:

Page 74 of 84

4. Please specify your participation in the Project.

☐ National consultant

☐ International consultant

☐ Participant of the event

☐ Working Group delegate

☐ Other

If other, please specify.

5. In case you took part in the capacity-building activities under this Project, please select all that apply:

Option Activity Date and venue

☐ 1st Session of the SPECA Working Group on Innovation and Technology for Sustainable Development.

30 July 2020, virtual modality (Country chair - Kazakhstan).

Subregional Workshop: Science, Technology and Innovation (STI) gap assessment of SPECA countries.

26 November 2020, virtual modality.

☐ Capacity building: 1st meeting of the Task Force on the Roadmap for the Development of Innovation Support Infrastructure in Kyrgyzstan until 2025.

9 June 2021, virtual modality.

☐ Capacity building: 2nd meeting of the Task Force on the Roadmap for the Development of the Innovation Ecosystem of Kyrgyzstan until 2025.

8 September 2021, virtual modality.

☐ 2nd Session of the SPECA Working Group on Innovation and Technology for Sustainable Development and Subregional workshop: Action plan of the SPECA Innovation Strategy for Sustainable Development.

20-21 October 2021, Hybrid mode: online platform, (Bishkek, Kyrgyzstan).

☐ 16th session of the SPECA Governing Council: Approval of the Action Plan of the SPECA Innovation Strategy for Sustainable Development.

19 November 2021, (Hybrid session) Tashkent, Uzbekistan.

☐ Capacity building: Challenges and opportunities for supporting innovative, high-growth enterprises in the SPECA countries.

23 December 2021, Virtual mode: online platform (Geneva, Switzerland).

☐ Capacity building for SPECA policymakers on supporting innovative high-growth enterprises in the SPECA sub-region.

15 February and 17 February 2022, Virtual mode: online platform (Geneva, Switzerland).

☐ Capacity building: “New approaches to innovation policy in the transition economies of the SPECA sub-region.”

19 May 2022, Virtual mode: online platform (Geneva, Switzerland).

Page 75 of 84

☐ Expert Group Meeting on Innovation and Technology for Sustainable Development: Proposal for a SPECA Network of Business Incubators and Accelerators for Sustainable Development (NBIASD).

19-20 July 2022, Hybrid mode (Almaty, Kazakhstan)

☐ 3rd Session of the SPECA Working Group on Innovation and Technology for Sustainable Development: Approval of creation of a SPECA NBIASD with national focal points.

20 July 2022, Hybrid mode (Almaty, Kazakhstan)

☐ Capacity building: Fostering innovative entrepreneurship through educational curricula in the SPECA sub-region.

19 September 2022, Virtual mode: online platform, (Geneva, Switzerland)

☐ Capacity building for SPECA countries "Effective management and development of business incubators and accelerators."

27-29 September 2022, Virtual mode: online platform, (Geneva, Switzerland).

☐ Capacity building for SPECA countries “Development of business incubators and start-up accelerators: training for university management.”

21 February 2023, Virtual mode: online platform, (Geneva, Switzerland).

☐ Working with venture funds and business angels: UNECE capacity building for staff of business incubators and accelerators in the SPECA sub-region.

16 March 2023, Virtual mode: online platform, (Geneva, Switzerland).

☐ Side event to the 70th session of the Economic Commission for Europe: Fostering circular solutions through innovation.

3 April 2023 (online, Geneva).

☐ UNECE capacity building workshop for Chief Innovation Officers on innovation for the circular economy.

2 May 2023 (Tashkent, Uzbekistan).

☐ Side event to the 16th session of the UNECE Committee on Innovation, Competitiveness and Public-Private Partnerships: Promoting women’s entrepreneurship in transition economies for sustainable development.

1 June 2023 (Geneva, Switzerland).

☐ Study tour to Georgia for innovation policymakers from the SPECA sub-region.

10-11 July 2023 (Tbilisi Georgia).

☐ Capacity building: Enhancing venture capital in Azerbaijan.

July-August 2023, Virtual mode: online platform.

☐ UNECE B2B capacity building “Innovation for the Circular Economy: Bridging Start-Ups and Corporations”).

18 October 2023 (Tashkent Uzbekistan

☐ 4th Session of the SPECA Working Group on Innovation and Technology for Sustainable Development and Subregional workshop: Progress Update on the Action Plan of the SPECA Strategy on Innovation for Sustainable Development.

18-19 October 2023 (Tashkent Uzbekistan)

☐ Innovation for the Circular Economy: Empowering Start-ups to Achieve More with Less: UNECE Study Tour for SPECA Participating States.

20-21 December 2023 (Tbilisi, Georgia)

6. To what extent did the project activities respond to the priorities and needs of your and/or targeted country (ies)?

☐ Responded to a great extent

☐ Partially responded

☐ Did not respond at all

☐ I do not know/Cannot answer

Page 76 of 84

Please specify your answer.

7. To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

☐ Were consistent to a great extent

☐ Partially consistent

☐ Not consistent at all

☐ I do not know/have no answer Please specify your answer…

8. To what extent were the project activities you attended relevant to the needs of your and/or targeted country

(ies)?

☐ Was relevant very much

☐ The activity was more or less relevant to my country needs

☐ Was not relevant at all to my country's needs

☐ Difficult to answer Please specify your answer…

9. To what extent were the project activities you attended met your expectations?

☐ It met my expectations to a great extent

☐ It was about what I expected

☐ It was worse than expected

☐ Difficult to answer

Please specify your answer…

10. To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project?

Page 77 of 84

Measurement Gender equality & mainstreaming

Human rights Disability perspectives

Climate change issues

To a great extent

To a moderate extent

Not integrated at all

Can’t answer

11. What needs to be done to better integrate gender, human rights, disability and climate change perspectives in future projects design and implementation?

12. Please specify any challenges to sustaining the project outcomes after completion of the Project.

13. Please specify the likelihood of sustaining, scaling up, replicating, or institutionalizing the stakeholders’ engagement after the project completion. 14. What would you recommend to sustain, replicate and scale up the results of the project? 15. What would you recommend as a potential follow-up activity/project?

Thank you for your input!

Page 78 of 84

Annex 4: List of Stakeholders Interviewed

No Name Position Organisation

1 Ms. Aisuluu

Mustapakulova Head of Innovations Division Kyrgyzpatent

2 Ms. Anastasia Pankova Consultant UNECE

3 Ms. Basak Demir Co-Founder DCube Circular Economy and

Sustainability

4 Mr. Christopher Athey Economic Affairs Officer UNECE

5 Ms. Ekaterina Guznova International Expert UNECE

6 Ms. Elif Kizildeli Associate Economic Affairs

Officer UNECE

7 Ms. Elizabeth Tuerk Director of Economic

Cooperation and Trade Division UNECE

8 Mr. Farid Huseynov

Project Manager of Startups

Support and Acceleration

Department

Innovation and Digital

Development Agency of

Azerbaijan

9 Mr. Oleg Dzioubinski Regional Advisor on Sustainable

Energy UNECE

10 Ms. Sarangoo

Radnaaragchaa

Regional Advisor on

Environment UNECE

10 Mr. Serdar Ishangulyyev Head of Youth Start-ups Union of Economists of

Turkmenistan

11 Ms. Yelena Shevchenko Head of Projects

Innovation Cluster at Nazarbaev

University of the Republic of

Kazakhstan

12 Mr. Yerbolat Orazbekuly First vice-president

Association of Business Incubators

and Accelerators based at

universities (Kazakhstan)

Page 79 of 84

Annex 5: List of Documents Reviewed

1. The UNDA 2023N Project Annual Progress Report for 2020, UNECE;

2. The UNDA 2023N Project Annual Progress Report for 2021, UNECE;

3. The UNDA 2023N Project Annual Progress Report for 2022, UNECE;

4. The 2023N Project Document “Strengthening innovation policies for SPECA countries in support of the

2030 Agenda for Sustainable Development,” UNECE, 2020;

5. Draft decisions of the 4th session of the SPECA WG on ITSD, UNECE, 2023;

6. Final Report for the 12th Tranche Of The Development Account “Strengthening innovation policies for

SPECA countries in support of the 2030 Agenda for Sustainable Development,” UNECE, March 2024 ;

7. “Draft Action Plan for implementing the SPECA Innovation Strategy for Sustainable Development

Prepared for consideration by the 16th Session of the SPECA Governing Council,” UNECE, 19 November 2021,

Tashkent, Uzbekistan;

8. “SPECA Innovation Strategy for Sustainable Development,” 14th Session of the Speca Governing

Council (Ashgabat, Turkmenistan) UNECE, 21 November 2019,

unece.org/fileadmin/DAM/SPECA/documents/gc/session14/SPECA_Innovation_Strategy_English.pdf ;

9. Draft discussion paper on establishing a “Digital Solutions Centre for Sustainable Development”:

options for operational modalities, UN ESCAP, 14 July 2022,

https://www.unescap.org/sites/default/d8files/event-documents/D5a-RoadmapDSCSD-E.pdf;

10. Assessment Report on the Implementation of the SPECA Innovation Strategy for Sustainable

Development and its Action Plan, Mr. Rumen Dobrinsky and Ms. Lyudmyla Tautiyeva, Assessment Report_full

first draft.pdf (unece.org);

11. “SPECA Innovation Strategy for Sustainable Development,” Rumen Dobrinsky European Alliance for

Innovation (2019),

https://www.unescap.org/sites/default/files/3.%20SPECA%20Innovation%20Strategy%20for%20Sustainable%

20Development.pdf;

12. “Azerbaijan 2030: National Priorities for Socio-economic Development,” The Government of the

Republic of Azerbaijan, 2021, Azerbaijan 2030: National Priorities for Socio-economic Development – Policies -

IEA;

13. Development of Synergies Between the NBIASD and Circular STEP, UNECE’s Stakeholder Engagement

Network for Circular Economy Road Map, UNECE, 2023, https://unece.org/sites/default/files/2023-

10/6.1.%20%20Road%20Map%20NBIASD-Circular%20STEP_0.pdf;

14. Global Innovation Index Reports for 2019-2023, WIPO;

15. Terms Of Reference of the SPECA Working Group on Innovation and Technology for Sustainable

Development, UNECE, 2019,

https://www.unescap.org/sites/default/files/ToR_SPECA_WG_on_ITSD_English.pdf;

16. “Business Incubators for Sustainable Development in the SPECA Subregion,” UNECE, 2021, Business

Incubators for Sustainable Development in the SPECA Sub-region | UNECE.

17. “Supporting Innovative High-Growth Enterprises in the SPECA Subregion,” UNECE, 2022, Supporting

Innovative High-Growth Enterprises in the SPECA sub-region, UNECE Policy Handbook | UNECE;

18. “New Innovation Policy in the SPECA Subregion,” UNECE, 2023, New Innovation Policy for transition

economies in the SPECA subregion | UNECE;

Page 80 of 84

19. DECISIONS of the 16th Session of the SPECA Governing Council, UNECE and UNESCAP, 19 November

2021, https://unece.org/sites/default/files/2022-

01/Decisions%20of%20the%2016th%20session%20of%20the%20SPECA%20Governing%20Council_ENG.pdf;

20. Draft discussion paper on establishing a “Digital Solutions Centre for Sustainable Development”:

options for operational modalities, UNESCAP, 2023, https://www.unescap.org/sites/default/d8files/event-

documents/D5a-RoadmapDSCSD-E.pdf;

21. Concept note SPECA Network of Business Incubators and Accelerators for Sustainable Development,

D8a-ConceptNote-SPECA-NetworkBusinessIncubatorsAccelerators-E.pdf (unescap.org);

22. “Business incubators for sustainable development in the SPECA subregion UNECE Policy Handbook,”

UNECE, 2021, 3. Business incubators for sustainable development in SPECA-2021-ENG_0.pdf (unece.org);

23. Бизнес-инкубаторы для устойчивого развития в субрегионе СПЕКА, UNECE, 2021, 3R SPECA-

2021_RU Corr_0.pdf (unece.org);

24. “Supporting Innovative High-Growth Enterprises in the SPECA Sub-Region: UNECE Policy Handbook,”

UNECE, 2023, 2227804_E_ECE_CECI_33_WEB_144dpi.pdf (unece.org);

25. « New Innovation Policy for transition economies in the SPECA subregion,” UNECE, 2024, NIPTE-SPECA-

2023-EN-WEB SIGNED.pdf (unece.org);

26. Recommendations for Enhancing Venture Capital in Azerbaijan, UNECE, 2024, Questions and Answers -

Enhancing Venture Capital in Azerbaijan.pdf (unece.org);

27. “Concept, UN ESCAP Digital Solutions Centre for Sustainable Development,” Explanatory Note to the

United Nations Economic and Social Commission for Asia and the Pacific (ESCAP) Sponsored by the

Government of Kazakhstan, 2024, Explanatory_Note_DSC_for_SD_Kazakhstan_Ver_7May2024.pdf

(unescap.org);

28. “Innovation for Sustainable Development: Review of Uzbekistan,” UNECE, 2022,

unece.org/sites/default/files/2022-06/9789211172966_I4SDR_UZBEKISTAN_2022_web_full%2Bcover.pdf;

29. Science, Technology, and Innovations (STI) Gap Analysis of Afghanistan., UNECE, 2021,

https://unece.org/eci/documents/2021/03/reports/science-technology-and-innovation-sti-gap-analysis-

afghanistan;

30. Science, Technology, and Innovation (STI) Gap Assessment of Azerbaijan, UNECE, 2021,

https://unece.org/sites/default/files/2021-03/STI%20gap%20analysis_Azerbaijan_Report_Yulia%20Alieva.pdf;

31. Science, Technology, and Innovation (STI) Gap Assessment of Kazakhstan, UNECE, 2021,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Kazakhstan_Report_Elena%20Shevchenko.pdf;

32. Science, Technology, and Innovation (STI) Gap Analysis of Kyrgyzstan, UNECE, 2021,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Kyrgyzstan_Report_%20Aziz%20Soltobaev.pdf;

33. Science, Technology, and Innovation (STI) Gap Analysis of Uzbekistan, UNECE, 2021,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Uzbekistan_Report_Nodira%20Kurbanbaeva.pdf;

34. Science, Technology And Innovation Gap Analysis Of Tajikistan, UNECE, 2020,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Tajikistan_Report_%20Bahodur%20Mengliev_0.pdf;

35. Анализ пробелов в сфере науки, технологий и инноваций (НТИ) в ТАДЖИКИСТАНЕ , UNECE, 2020, 1 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Tajikistan_Report_%20Bahodur%20Mengliev_RUS.pdf;

Page 81 of 84

36. Science, Technology and Innovation (STI) Gap Assessment of Turkmenistan, UNECE, 2020,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Turkmenistan%20Report_Yuriy%20Aronskyi.pdf;

37. Development Strategy for New Uzbekistan for 2022-2026, Appendix #1 to the Decree of the President of the Republic of Uzbekistan #PF-60 of January “28”, 2022, development_strategy_of_new_uzbekistan_for_202.pdf (ecesbf.uz).

Annex 6: Management Response and Recommendation Action Plan

Management Response to the Terminal Evaluation of the Development Account 13th Tranche Project

Project title and ID: UNDA Project 2023N “Strengthening Innovation Policies for SPECA Countries in Support of the 2030 Agenda for

Sustainable Development.”

Entity name: UNECE

Overall Remarks by Management

Recommendation Action Plan

Recommendation Management response1 Planned action Responsible unit(s) Target

completion

date

1. The evaluation recommends continuing to apply

participatory and agile management modalities in future

project design and implementation. It will allow the

project teams to get real-time insights into project

progress and potential issues, identify risks, and mitigate

them early on, resulting in increased efficiency and

effectiveness of the projects.

2. The evaluation strongly advises engaging the

private sector, associations, and non-state actors in

project activities to strengthen in-country and regional

collaboration across sectors. This would also secure a

better balance of political support and a sense of

ownership with industry-specific insights, data-driven

1 Accepted, partially accepted, or rejected.

Page 83 of 84

evidence (providing the data crucial for policy decision-

making), and in-country lobbying for policy changes.

3. The evaluation also heavily recommends revising

the formulation of outcome and output level indicators to

avoid overlap and ensure that complex concepts are

broken down into measurable terms at the outcome

level. This could be done in consultation with the relevant

stakeholders to evaluate the validity and reliability of the

indicators selected. Furthermore, the evaluation

recommends incorporating gender, human rights,

disability, and climate change indicators into the results

framework to enable continuous reporting on progress

made. Notably, some indicators could be developed at the

output level (i.e. disability or gender indicators), and

others can be elaborated based on outcome-level

measurements (i.e. climate change indicators). However,

it will be up to the project team to decide on the

measurement layer (outcome or output) of the indicators

selected, depending on the context of the upcoming

projects and the potential availability and reliability of the

data pertinent to the given indicator.

4. With regard to the implementation modalities of

the capacity building activities, the evaluation

recommends the development of online webinars with

face-to-face meetings. In this regard, the UNECE is advised

to arrange online national capacity building and

awareness-raising events for a broader audience.

However, face-to-face regional meetings and site visits

would be more relevant for the sort of capacity building

activities that would potentially lead to cross-sector

strategic partnerships and provide first-hand experience

of the practical implications and challenges of the

approaches applied in the host countries.

5. The evaluation recommends reconsidering project staffing patterns to ensure smooth

Page 84 of 84

implementation and efficient continuity of the projects’ activities in case of emergencies. While the UNECE staff will remain engaged on a part-time basis, their level of engagement can be increased depending on the lifetime of the project. In this regard, the evaluation recommends including the budget lines associated with the salaries of the support and professional staff in the new project proposal. The availability of the qualified workforce assigned to the specific project is particularly vital for the efficient and smooth implementation of the project activities. For ease of reference, the budget share of both analytical and capacity building activities of the Project under evaluation constituted 86.6 % of the total budget).

Name Title Signature Date

1

TERMS OF REFERENCE

UNDA2023N: Strengthening innovation policies for SPECA countries in support of the

2030 Agenda for Sustainable Development.

I. Purpose

The purpose of this evaluation is to assess the extent to which the objectives of the UNECE project

2023N “Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for

Sustainable Development” were achieved.

The evaluation will assess the relevance, effectiveness, efficiency and sustainability of the project in

supporting seven SPECA countries in their efforts to spur innovation as a central driver of the

increasingly urgent transition from a low productivity and resource-intensive model of economic

development to knowledge-based and more sustainable economic growth, in line with the SDGs.

The evaluation will also assess any impacts the project may have had on progressing human rights,

gender equality, disability inclusion, climate change and disaster risk reduction in the context of this

engagement. The evaluation will finally look at the activities repurposed to address the impact of the

COVID-19 crisis, and assess, where relevant, UNECE’s COVID-19 early response through this project.

II. Background

The project aimed to support seven SPECA countries in their efforts to spur innovation as a central

driver of the increasingly urgent transition from a low productivity and resource-intensive model of

economic development to knowledge-based and more sustainable economic growth, in line with the

SDGs. It builds on a clear mandate: SPECA countries, recognizing this challenge, formally decided at

the 12th session of the SPECA Governing Council in 2017 to “Develop an innovation strategy to

promote sustainable development in the SPECA region” – a process that, with UNECE and ESCAP

support, has led to a final draft that will be submitted for approval at the 2019 SPECA governing

council.

In this context, the project aimed to support and create sustainable further momentum around the first

steps towards putting this strategy into practice. Specifically, the objective was to strengthen

institutional capacities to harness innovation as a driver of sustainable development and regional

integration.

The first target outcome was strengthened cooperation on innovation among the SPECA countries.

Central to this outcome was to work with SPECA countries to put together a concrete first action plan

under the strategy, including several joint initiatives. This involved a detailed gap analysis covering

all seven countries to inform the development of the draft action plan, which was discussed in detail at

a sub-regional workshop before finalization and submission for approval. Towards the end of the

implementation period, a sub-regional workshop served to measure implementation progress based on

the indicators in the action plan. In addition to the national initiatives, two initiatives involving

cooperation among the SPECA countries were established: (i) SPECA Network of Business

Incubators and Accelerators for Sustainable Development (NBIASD); (ii) Progress towards creation

of a subregional Digital Solutions Centre for Central Asia in Kazakhstan.

The second outcome on enhanced capacity to design and carry out effective innovation policy and

institutional reform, aimed to address some of the leading constraints in putting the strategy into

practice among SPECA countries. This involved both a series of sub-regional seminars on selected

issues of common concern, as well as in-depth trainings for at least three countries. Importantly, this

project foresaw substantial flexibility to respond to country requests and priorities and to build on

existing or impending momentum.

2

ESCAP, as the co-lead in providing Secretariat support to the SPECA Working Group on Innovation

and Technology for Sustainable Development, played a role in most activities. There was an aim to

engage the donor community throughout, with the clear purpose of finding projects that contribute to

the strategy and the action plan that qualify for donor funding – sustaining the momentum beyond the

closure of this project.

III. Evaluation objectives, scope and questions

The evaluation will be guided by the objectives and indicators of achievement established in the results

framework of the project document. The evaluation will be conducted in Q4 of 2023. It will cover the

full implementation of the project, from January 2020 to December 2023 in the SPECA participating

countries of the UNECE region, Azerbaijan, Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan and

Uzbekistan.

The evaluation criteria are relevance, effectiveness, efficiency and sustainability.

Relevance

1. To what extent did the project respond to the priorities and needs of participating countries?

2. To what extent were the project activities consistent with global and regional priorities and

aligned with the SDGs?

3. How relevant were the project activities vis-à-vis the programme of work of the UNECE? What

value has UNECE’s efforts added in this area?

4. To what extent were gender, human rights, disability perspectives and climate change

considerations integrated into the design and implementation of the project? How can these

perspectives be better included in future projects design and implementation?

Effectiveness

5. To what extent were the project design and set-up effective for meeting the needs of the

beneficiary countries?

6. To what extent were the project objectives and expected results achieved?

7. To what extent were the project activities coherent and harmonized with those of other partners

operating within the same context, particularly those of other UN system entities?

8. What were the challenges/obstacles (including COVID-19) to achieving the expected results?

How successfully did the project overcome these?

Efficiency

9. Were the resources adequate for achieving the results?

10. Were the results achieved on time and were all activities organized efficiently?

11. To what extent were the resources used economically and how could the use of resources be

improved?

12. Sustainability

13. What measures were adopted to ensure that project outcomes would continue after the project

ended and to what extent have these measures addressed the existing risks for sustainability?

14. To what extent do the partners and beneficiaries ‘own’ the outcomes of the project? How is the

stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

IV. Evaluation approach and methodology

The evaluation will be conducted in accordance with: the ECE Evaluation Policy1; the Administrative

instruction guiding Evaluation in the UN Secretariat 2 ; and the United Nations Evaluation Group

1 UNECE Evaluation policy 2 ST/AI/2021/3

3

(UNEG) Norms and Standards for Evaluation3. Human rights and gender equality considerations will

be integrated at all stages of the evaluation4: (i) in the evaluation scope and questions; (ii) in the

methods, tools and data analysis techniques; (iii) in the findings, conclusions and recommendations of

the final report. The evaluator will explicitly explain how human rights, gender, disability, SDGs, and

climate change considerations will be taken into account during the evaluation.

The evaluator is required to use a mixed-method approach, including qualitative as well as quantitative

data gathering and analysis as the basis for a triangulation exercise of all available data to draw

conclusions and findings. The evaluator shall conduct online surveys and interview a wide range of

diverse stakeholders.

The evaluation should be conducted based on the following mixed methods to triangulate information:

1. A desk review of all relevant documents, including the project document and information on project

activities (monitoring data); materials developed in support of the activities (agendas, plans,

participant lists, background documents, donor reports and publications); proposed programme

budgets covering the evaluation period; project reports to the donor.

2. Online survey of key stakeholders and beneficiaries: the survey will be developed by the consultant

on her/his preferred platform.

3. Interviews (in-person and/or by telephone/video): the evaluator shall interview a wide range of

diverse stakeholders and beneficiaries including policymakers, representatives of the government,

international organisations, academia, civil society as applicable.

4. Observation of workshops and meetings, including recording of meetings as appropriate.

The evaluator will further elaborate on the evaluation methodology in the Inception Report that will

among others include the electronic survey questions, interview guide, and whether any of the six

countries will be selected for an in-depth assessment. The evaluation report will be written in English,

will consist of approximately 30 pages and will include an executive summary (max. 2 pages)

describing the evaluation methodology, key findings, conclusions and recommendations. The evaluator

will also produce an Evaluation Brief summarizing key evaluation findings, lessons learned and

recommendations, including through images and infographics.

V. Evaluation schedule5

September 2023 ToR finalized

September 2023 Evaluator selected

October 2023 Contract signed. Evaluator starts the desk review

November 2023 Evaluator submits inception report including survey design

and interview guide

November 2023 Launch of data gathering, including survey and interviews

December 2023 Evaluator submits draft evaluation report and evaluation brief

December 2023 Evaluator submits final evaluation report and evaluation brief

VI. Resources and Management of the evaluation

An independent consultant will be engaged to conduct the evaluation under the management of the

Programme Management Unit (PMU). Payment will be made upon satisfactory delivery of work.

The PMU will manage the evaluation and will be involved in the following steps: Selection of the

evaluator; Preparation and clearance of the Terms of Reference; Provision of guidance to the Project

3 UNEG 2016 Norms and Standards for Evaluation 4 In line with UNEG Guidance contained in Integrating Human Rights and Gender Equality in Evaluations 5 Final timetable to be agreed following engagement of the evaluator

4

Manager and evaluator as needed on the evaluation design and methodology; Clearance of the final

report after quality assurance of the draft report.

The Project Manager, in consultation with the Division Director, will be involved in the following steps:

Provide all documentation needed for desk review, contact details, support and guidance to the

evaluation consultant as needed throughout the timeline of the evaluation; Advise the evaluator on the

recipients for the electronic survey and for follow-up interviews; Process and manage the consultancy

contract of the evaluator, along the key milestones agreed with PMU.

VII. Intended use / Next steps

The results of the evaluation will be used in the planning and implementation of future activities of the

UNECE Economic Cooperation and Integration Subprogramme. Findings of this evaluation will be

used when possible to:

• Improve direct project’s follow up actions, implementation of products by project beneficiaries

and dissemination of the knowledge created through the project.

• Assess the gaps and further needs of countries in the area of this project.

• Formulate tailored capacity building projects to strengthen the national capacity in enhancing

innovation.

The results of the evaluation will be reported to the inter-governmental Team of Specialists on

Innovation and Competitiveness Policies and the Committee on Innovation, Competitiveness and

Public-Private Partnerships, as well as with the Executive Committee if required.

Following the issuance of the final report, the Project Manager will develop a Management Response

for addressing the recommendations made by the evaluator. The final evaluation report, the

management response and the progress on implementation of recommendations will be publicly

available on the UNECE website.

VIII. Criteria for evaluators

The evaluator should have:

1. An advanced university degree or equivalent background in relevant disciplines.

2. Knowledge of and experience in sustainable economic development and/or innovation policy.

3. Relevant professional experience in design and management of evaluation processes with multiple

stakeholders, survey design and implementation, project planning, monitoring and management,

gender mainstreaming and human-rights due diligence.

4. Demonstrated methodological knowledge of evaluations, including quantitative and qualitative

data collection and analysis for end-of-cycle project evaluations, including demonstrated

experience in conducting questionnaires and interviews.

5. Fluency in written and spoken English and Russian.

Evaluators should declare any conflict of interest to UNECE before embarking on an evaluation project,

and at any point where such conflict occurs.

1

United Nations Development Account

Terminal Evaluation of project 2023N “Strengthening Innovation Policies for SPECA Countries in Support

of the 2030”

(2020-2023)

Report completed on: July 2024

Evaluation conducted by: Nelly Dolidze

2

Evaluator: Ms. Nelly Dolidze

Evaluation Manager: Mr. Christopher Athey, Economic Affairs Officer, Innovative Policies

Development Section

The United Nations Economic Commission for Europe (UNECE), Palais des Nations, 8-14,

avenue de la Paix, CH-1211 Geneva 10, Switzerland

The evaluator would like to express her gratitude to the member States, the United Nations

Economic Commission for Europe (UNECE) representatives, non-state institutions, and the

subject-matter experts who provided inputs to the online survey and during an intensive

interview process.

The evaluator also wishes to express her most profound appreciation to all those who

supported the evaluation and made it possible to complete this assignment. Furthermore, the

evaluator would also like to warmly acknowledge the crucial role of the Innovative Policies

Development Section and Programme Management Unit staff at the UNECE for their invaluable

support in this evaluation.

Moreover, many thanks also go to Ms. Elisabeth Türk (Director of the Economic Cooperation

and Trade Division (ECTD)), Mr. Christopher Athey (Economic Affairs Officer), Mr. Nicolas Dath-

Baron (Officer in Charge of the Program Management Unit), Mr. Oscar Fast (Associate

Economic Affairs Officer), and any others to have invested their efforts to provide valuable

comments and advice.

This report was commissioned by the UNECE The findings, conclusions and recommendations of this

report are those of the external evaluator and do not necessarily reflect the views of the UNECE.

3

Table of Contents

List of Acronyms and Abbreviations ....................................................................................................... 5

Tables and Figures ..................................................................................................................................... 6

Executive summary ................................................................................................................................... 7

1. Introduction .......................................................................................................................................... 11

2. Description of the Project .................................................................................................................. 11

2.1 Background.............................................................................................................................. 11

2.2 Project objectives and expected results ............................................................................. 11

2.3 Project strategies and key activities ................................................................................... 11

2.4 Target countries and beneficiaries ...................................................................................... 14

2.5 Key partners and other key stakeholders ........................................................................... 14

2.6 Resources ................................................................................................................................ 15

2.7 Link to the Sustainable Development Goals (SDGs) ......................................................... 15

2.8.1 Innovative elements ............................................................................................................... 15

3. Evaluation objectives, scope and questions ................................................................................... 15

3.1 Purpose and objectives ......................................................................................................... 15

3.2.1 Evaluation scope, criteria and questions ............................................................................ 16

4. Methodology ........................................................................................................................................ 17

4.1 Evaluation Limitations ........................................................................................................... 19

5. Findings ................................................................................................................................................. 19

5.1 Relevance ................................................................................................................................. 19

5.2 Effectiveness ........................................................................................................................... 34

5.3 Efficiency ................................................................................................................................. 48

5.4 Sustainability .......................................................................................................................... 52

6. Conclusions and Recommendations ................................................................................................ 55

4

7. Lessons learned and good practices ................................................................................................ 57

Annexes ..................................................................................................................................................... 58

Annex 1: Evaluation ToR ......................................................................................................................... 59

Annex 2: Evaluation Matrix ..................................................................................................................... 64

Annex 3: Data Gathering Tools .............................................................................................................. 66

Annex 4: List of Stakeholders Interviewed .......................................................................................... 78

Annex 5: List of Documents Reviewed ................................................................................................. 79

Annex 6: Management Response and Recommendation Action Plan ............................................ 82

5

List of Acronyms and Abbreviations

AIIB Asian Infrastructure Investment Bank

BA Business Accelerator

BI Business Incubator

DAC Development Assistance Committee

DSCSD Digital Solutions Centre for Sustainable Development

EA Expected Accomplishments

EC The European Commission

EPR Environment Performance Review

FCDO Foreign, Commonwealth and Development Office

GII Global Innovation Index

ICT Information-Communication Technologies

I4SDR Innovation for Sustainable Development Review

IsDB Islamic Development Bank

ITU International Telecommunications Union

KII Key Informant Interview

MPTF Multi-Partner Trust Fund

NBIASD Network of Business Incubators and Accelerators for Sustainable

Development

OECD Organisation for Economic Co-operation and Development

SDG Sustainable Development Goal

SPECA The United Nations Special Programme for the Economies of Central Asia

STI Science, Technology and Innovation

ToC Theory of Change

ToR Terms of Reference

TTSTI Task Team on Science, Technology and Innovation

UN United Nations

UNECE United Nations Economic Commission of Europe

UNEG United Nations Evaluation Group

UNESCAP United Nations Economic Commission for Asia and the Pacific

UNESCO United Nations Educational, Scientific and Cultural Organization

UNDA United Nations Development Account

UNDP United Nations Development Programme

UNIDO United Nations Industrial Development Organization

UNITAR United Nations Institute for Training and Research

EPR Environment Performance Review

WG on ITSD Working Group on Innovation and Technology for Sustainable Development

WIPO World Intellectual Property Organization of the United Nations

WS Workstream

VNR Voluntary National Review

6

Tables and Figures

Table 1 The Project Structure

Table 2 The Project Activities

Table 3 Approved Budget of the Project

Table 4 Key Evaluation Questions

Table 5 Cluster of Stakeholders Contacted

Table 6 A Snapshot of the Project Activities

Table 7 National Policy Documents and Legal Acts Related to Promoting Innovations

Table 8 The Contribution of the Project Activities to the SDGs

Table 9 Status of the SPECA participating States in Achieving the SDGs (2018 data)

Table 10 Status of the VNRs in the Targeted Countries

Table 11 The I4SDRs and EPRs Carried out in the SPECA participating States

Table 12 Progress Made Against Outcome-level Indicators

Table 13 Progress Made Against Outputs

Table 14 Project Activities Under OP 2.2

Table 15 Project Activities Under OP 2.3

Table 16 International Donor-funded Initiatives to Develop the Innovation Ecosystem in the

Targeted Countries

Table 17 The Project Budget in USD (Planned vs. Actual)

Figure 1 Profile of the Online Mini-survey Respondents

Figure 2 Feedback from Mini-survey Respondents on the Project Alignment with SDGs and

Regional Priorities

Figure 3 Global Innovation Index (GII) for the Project’s Beneficiary Countries (2019-2023)

Figure 4 Conceptual Roadmap of the Project and its Link to the UNECE’s Mandate

Figure 5 Opinions of Mini-survey Respondents on Gender Equality & Mainstreaming, Human

Rights, Disability Perspectives, and Climate Change Considerations

Figure 6 Feedback from Mini-Survey Respondents on their Satisfaction with Project Events

Figure 7 Budget Idem share to the Total Budget (Planned vs. Actual)

7

Executive summary

1. The evaluation of the United Nations Development Account (UNDA) Project 2023N

“Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable

Development” (hereafter called “the Project”) was carried out by an external evaluator in February-June

2024.

2. The Project was launched in January 2020 and concluded in December 2023. It was financed

through the UNDA and had a total budget of USD 483,316. By design, the Project targeted six (6) member

States of both the UN Economic Commission for Europe (UNECE) and the UN Economic and Social

Commission for Asia and the Pacific (ESCAP), and one member State of UN ESCAP: Afghanistan1, the

Republic of Azerbaijan, the Republic of Kazakhstan, the Kyrgyz Republic, the Republic of Tajikistan,

the Republic of Turkmenistan, and the Republic of Uzbekistan.

3. Pursuant to the Terms of Reference (ToR) of the evaluation (ANNEX 1), the evaluation was

conducted in accordance with the Development Assistance Committee’s (DAC) criteria introduced by the

Organisation for Economic Co-operation and Development (OECD) for evaluating the development

projects and programs: relevance, effectiveness, efficiency, and sustainability. UNECE is the primary user

of this evaluation.

4. Overall, the evaluation covered the full duration of the Project and resulted in the following key

findings and conclusions:

➢ The Project's participatory design and implementation modalities, including the sessions and the

composition of the SPECA WG on the ITSD and expert groups, allowed for its complete alignment

with the national and global development agendas. It is noteworthy that six out of the seven

beneficiary countries prioritized innovation and digitalization. At the same time, the project

design and implementation modalities were not heavily focused on the inclusion of the third

sector (i.e., private sector, associations, and relevant non-state actors).

➢ The evaluation also validated that the Project was fully aligned with global and regional priorities

and four SDGs 2 . Moreover, the Project activities were thematically coherent with and

complementary to the interventions (which vary per country) funded by other international

donors (including UN agencies).

1 Afghanistan a member of SPECA but not of the UNECE - participated in subregional activities supported by UN

ESCAP, of which Afghanistan is a member. 2 I.e., SDG 8 (“Promote sustained, inclusive, and sustainable economic growth, full and productive employment and

decent work for all”), SDG 9 (“Build resilient infrastructure, promote inclusive and sustainable industrialization, and

foster innovation”), SDG 12 (“Ensure sustainable consumption and production patterns”), and SDG 17 (“Strengthen

the means of implementation and revitalize the Global Partnership for Sustainable Development”).

8

➢ The operational, structural, and conceptual arrangement of the Project safeguarded its direct

contribution to the UNECE’s overall mandate, and its work programme related to economic

cooperation and integration as well as the environment. The key stakeholders highlighted the

exceptional position held by the UNECE in securing high-level political support, cross-regional

engagement and collaboration, and the provision of multisector technical and analytical expertise.

➢ The evaluation found that the Project’s objectives and results were achieved at the outcome and

output levels. Noteworthily, the most significant results/outcomes of the Project were the

development and adoption of the Action Plan for the SPECA Innovation Strategy, the development

of the SPECA Network of Business Incubators and Accelerators for Sustainable Development, and

the Project’s contribution to the development of the Kazakhstan Association of the Universities

Business Incubators and Accelerators (KAUBIA, www.kaubia.kz). The stakeholders reported on

the capacity building events conducted within the framework of the Associations mentioned

above. However, the overlap between the outcome level and output level indicators3 renders a

distinctive qualitative assessment of the reported results impossible.

➢ The Project provided a platform for policy-level discussions within the framework of the SPECA

Working Group on Innovation and Technology for Sustainable Development and the 16th session

of the SPECA Governing Council. It facilitated science, technology, and innovation gap analysis for

all the targeted countries. It also supported the production of a background paper related to the

prospects for SPECA regional cooperation on innovation. UNECE posted all the reports online to

make them easily accessible to the public. The Project also contributed to building the capacity of

the targeted countries through face-to-face and online workshops, side events, and training

sessions.

➢ The Project results framework was aligned with the relevant development priorities of the

targeted countries but lacked gender, human rights, disability, and climate-change-sensitive

indicators by design. The absence of indicators to measure progress across gender, human rights,

disability, and climate change dimensions resulted in an information gap and low awareness

among beneficiary countries regarding the correlation between the innovation agenda and these

dimensions. This might have lessened the impact of the Project deliverables on the capacity of

the participating countries to prioritize progress in the relevant cross-sectoral processes and

structures that would eventually contribute to the achievement of the relevant SDGs (i.e., SDG 54,

SDG 105, and SDG 136).

3 Outcome level indicator IA1.1 (“An action plan for the SPECA innovation strategy with performance indicators

aligned with the SDGs is developed and agreed with SPECA countries and adopted by the SPECA Governing

Council”); Output level indicator OP1.4 (“Produce a final draft of the SPECA Innovation Strategy Action Plan and

secure its formal approval by the SPECA Governing Council, with the attendance of the national focal points and

experts (a session within the framework of the SPECA Economic Forum and the SPECA Governing Council”). 4 SDG 5: “Gender Equality.” 5 SDG 10: “Reduced Inequality.” 6 SDG 13: “Climate Action.”

9

➢ Despite lacking gender equality, human rights, disability, and climate change-sensitive indicators,

the UNECE incorporated climate change and gender mainstreaming agenda in the Project’s

activities. The side event to the 16th session of the UNECE Committee on Innovation,

Competitiveness and Public-Private Partnerships promoted women’s entrepreneurship in

transition economies for sustainable development, and the Work Plan for the SPECA WG on ITSD

approved at the 4th Session of the SPECA WG on ITSD referred to climate change dimension.

➢ The Project experienced some external challenges (e.g., the COVID-19 pandemic and political

changes in the host and beneficiary countries) that delayed the project implementation. The

Project team successfully applied an agile management approach to secure an iterative and

incremental implementation of the Project, focusing on flexibility, collaboration, and the

satisfaction of the participating countries and institutions. This also included the budget

modifications that allowed for all the results to be achieved as planned at the outcome and output

levels and switching to an online capacity building mode. Therefore, the budget analysis showed

a significant increase in spending on budget items related to external contractual services and

expert fees.

➢ The Project backstopping and administration were limited to one P-level staff, and a full-time

administrative support staff was lacking. According to key stakeholders, the Project's human

resource structure needed readjustment to secure advanced backstopping (at the administration

level) and continuous engagement of external thematic experts and service providers.

➢ The evaluation confirmed the interest of the beneficiary countries and a sense of ownership

regarding the Project results. At the same time, the evaluation acknowledged that several

strategic challenges directly affected the sustainability, scale-up, and replication of the Project

results, including the lack of financial resources, political influences from neighbouring countries,

changes in the political agenda, staff turnover, and the shortage of a structured knowledge

transfer system in the beneficiary countries. The evaluation concluded that the sustainability,

replication, and scale-up of the Project’s results cannot be guaranteed unless the UNECE and

SPECA participating States proactively address the external risks and challenges associated with

financial limitations through the sustainability strategy (i.e., SPECA Multi-Partner Trust Fund

(MPTF)) to support SPECA participating States across all relevant projects.

➢ The evaluation also acknowledged the proactive actions of the Secretariat to raise funds and

continue supporting SPECA participating States in strengthening their innovation policies. Its new

project proposal (pending donor approval) also incorporated inclusive digital transformation and

leaving no one behind agenda with a focus on gender equality and the empowerment of women

and girls.

5. Based on the above findings, the evaluation resulted in the following five recommendations:

10

➢ The evaluation recommends continuing to apply participatory and agile management

modalities in future project design and implementation to allow the project teams to get real-

time insights into project progress and potential issues.

➢ The evaluation strongly advises continuing engaging the private sector, associations, and non-

state actors in project activities to strengthen in-country and regional collaboration across

sectors and secure a better balance of political support and a sense of ownership with industry-

specific insights.

➢ The evaluation also heavily recommends revising the formulation of outcome and output level

indicators of future projects to avoid overlap and ensure that complex concepts are broken

down into measurable components at the outcome level. Furthermore, the evaluation

recommends incorporating gender, human rights, disability, and climate change indicators into

the results framework to enable continuous reporting on progress made.

➢ With regard to the implementation modalities of the capacity building activities, the evaluation

recommends the development of online webinars (for general capacity building activities at the

national level) with face-to-face meetings (those addressing regional, strategic and more

advanced knowledge sharing). However, face-to-face regional meetings and site visits would be

more relevant for the sort of capacity building activities that would potentially lead to cross-sector

strategic partnerships and provide first-hand experience of the practical implications and

challenges of the approaches applied in the host countries.

➢ The evaluation recommends reconsidering project staffing patterns to align their level of

engagement to the lifetime of the project. The Secretariat needs to include administrative support

and professional-level staff in the project proposals to ensure the smooth continuity of new

project activities. This is particularly crucial, taking into account the level of effort required for

quality assurance of the developed concept papers and assessment notes and for organizing

cross-country and regional face-to-face capacity building activities, workshops, and side events.

Page 11 of 84

1. Introduction

6. The current document presents the evaluation report for the UNDA’s “Strengthening

Innovation Policies for SPECA Countries in Support of the 2030 Agenda for Sustainable Development”

project (hereinafter, “the Project”).

2. Description of the Project

2.1 Background

7. The Project was initially launched in January 2020, following a decision taken in 2017 at the

12th session of the Governing Council of the United Nations Special Programme for the Economies of

Central Asia (SPECA) to develop an innovation strategy to promote sustainable development in the

SPECA region. Later, the innovation strategy was submitted for approval to the 2019 SPECA Governing

Council, aiming to “…achieve a sustained improvement in the national capacity and capabilities of the

SPECA countries to formulate and implement innovation policies for sustainable development and to

raise the level and quality of regional cooperation in the implementation of innovations that target

and support sustainable development.”7 The Project was expected to be concluded in December 2023.

It was financed through the UNDA and had a total budget of USD 483,316.

2.2 Project objectives and expected results

8. The Project was set out to support and create further sustainable momentum in the course of

taking the first steps toward putting the above-mentioned strategy into practice. In addition, it

intended to support the seven SPECA participating States in their efforts to spur innovation as a central

driver of the increasingly urgent transition from a resource-intensive model of economic development

characterized by low productivity to a knowledge-based model delivering more sustainable economic

growth, in line with the Sustainable Development Goals (SDGs).

9. The Project encompassed the following two main objectives:

➢ To support and create further sustainable momentum while putting into practice an

innovation strategy to promote sustainable development in the SPECA region, as decided at

the 12th session of the SPECA Governing Council in 2017; and

➢ To strengthen the institutional capacities of the landlocked SPECA countries to harness

innovation as a driver of sustainable development and regional integration.

2.3 Project strategies and key activities

10. The Project was implemented in line with two main workstreams (WSs) corresponding to the

two expected outcomes and seven outputs (Table 1).

7 SPECA_Innovation_Strategy_English.pdf (unece.org).

Page 12 of 84

Table 1: The Project Structure

Workstreams (WS)

Outcomes Outputs Indicator of Achievement (IA)

WS 1

Strengthened co-operation on innovation to promote sustainable development and deliver the Agenda 2030 in the SPECA subregion.

Output 1.1: Develop a subregional gap analysis covering the 7 SPECA countries to inform the first Action Plan for the SPECA Innovation Strategy.

IA 1.1: An Action Plan for the SPECA innovation strategy with performance indicators aligned with the SDGs is developed and agreed with SPECA countries and adopted by the SPECA Governing Council.

IA 1.2: At least two joint initiatives on innovation for sustainable development involving two or more SPECA countries developed and endorsed by the beneficiary countries.

Output 1.2: Organize a subregional workshop to present and validate the gap analysis.

Output 1.3: Organize a subregional workshop to develop an Action Plan and its performance indicators and develop joint activities between countries on innovation for sustainable development.

Output 1.4: Produce a final draft of the SPECA Innovation Strategy Action Plan and secure its formal approval by the SPECA Governing Council, with attendance of the national focal points and experts (a session within the framework of the SPECA Economic Forum and the SPECA Governing Council).

Output 1.5: Organize a subregional workshop to measure the implementation progress of the Action Plan based on its key performance indicators.

WS 2

Enhanced capacity to design and carry out effective innovation policy and institutional reform.

Output 2.1: Develop training materials on good practices on innovation policies and institutions for sustainable development, including technology transfer and science and technology parks (based on activities with ESCAP and IATT on science, technology, and innovation under A2.1).

IA 2.1: At least 3 of the 7 SPECA countries designed at least one initiative to improve support for innovation for sustainable development.

Output 2.1: Within the framework of a national capacity building agreement with 3 SPECA countries, organize 6 national workshops (2 for each country) on a specific topic to address key issues at the national level identified by the gap analysis as part of national consultations carried out under OP1.1, as reflected in the SPECA Innovation Strategy Action Plan. In parallel with these events, UNECE and ESCAP will provide targeted support to three countries on three specific issues.

Source: The Project Design Document, UNECE (2020).

11. In total, in accordance with the Project framework, the United Nations Economic Commission

for Europe (UNECE) team carried out 23 capacity building activities (including face-to-face and online

workshops, study tours and side events, SPECA Working Group meetings, and Expert and Technical

Group meetings) (Table 2).

Page 13 of 84

Table 2: The Project Activities8

# Activity Date and venue

1 1st Session of the SPECA Working Group on Innovation and

Technology for Sustainable Development.

30 July 2020, virtual modality

(Country chair - Kazakhstan).

2 Subregional Workshop: Science, Technology and Innovation

(STI) gap assessment of SPECA countries.

26 November 2020, virtual

modality.

3 Capacity building: 1st meeting of the Task Force on the

Roadmap for the Development of Innovation Support

Infrastructure in Kyrgyz Republic until 2025.

9 June 2021, virtual modality.

4 Capacity building: 2nd meeting of the Task Force on the

Roadmap for the Development of the Innovation Ecosystem of

Kyrgyz Republic until 2025.

8 September 2021, virtual

modality.

5 2nd Session of the SPECA Working Group on Innovation and

Technology for Sustainable Development and Subregional

workshop: Action plan of the SPECA Innovation Strategy for

Sustainable Development.

20-21 October 2021, Hybrid

mode: online platform,

(Bishkek, Kyrgyz Republic).

6 16th session of the SPECA Governing Council: Approval of the

Action Plan of the SPECA Innovation Strategy for Sustainable

Development.

19 November 2021, (Hybrid

session) Tashkent,

Uzbekistan.

7 Capacity building: Challenges and opportunities for supporting

innovative, high-growth enterprises in the SPECA countries.

23 December 2021, Virtual

mode: online

platform (Geneva,

Switzerland).

8 Capacity building for SPECA policymakers on supporting

innovative high-growth enterprises in the SPECA subregion.

15 February and 17 February

2022, Virtual mode: online

platform (Geneva,

Switzerland).

9 Capacity building: “New approaches to innovation policy in the

transition economies of the SPECA subregion.”

19 May 2022, Virtual mode:

online platform (Geneva,

Switzerland).

10 Expert Group Meeting on Innovation and Technology for

Sustainable Development: Proposal for a SPECA Network of

Business Incubators and Accelerators for Sustainable

Development (NBIASD).

19-20 July 2022, Hybrid

mode (Almaty, Kazakhstan)

11 3rd Session of the SPECA Working Group on Innovation and

Technology for Sustainable Development: Approval of

the creation of a SPECA NBIASD with national focal points.

20 July 2022, Hybrid

mode (Almaty, Kazakhstan)

12 Capacity building: Fostering innovative entrepreneurship

through educational curricula in the SPECA subregion.

19 September 2022, Virtual

mode: online platform,

(Geneva, Switzerland)

13 Capacity building for SPECA countries "Effective management

and development of business incubators and accelerators."

27-29 September 2022,

Virtual mode: online platform,

(Geneva, Switzerland).

8 Innovation and Technology for Sustainable Development | UNECE.

Page 14 of 84

14 Capacity building for SPECA countries “Development of business

incubators and start-up accelerators: training for university

management.”

21 February 2023, Virtual

mode: online platform,

(Geneva, Switzerland).

15 Working with venture funds and business angels: UNECE

capacity building for staff of business incubators and

accelerators in the SPECA subregion.

16 March 2023, Virtual mode:

online platform, (Geneva,

Switzerland).

16 Side event to the 70th session of the Economic Commission for

Europe: Fostering circular solutions through innovation.

3 April 2023 (online, Geneva).

17 UNECE capacity building workshop for Chief Innovation Officers

on innovation for the circular economy.

2 May 2023 (Tashkent,

Uzbekistan).

18 Side event to the 16th session of the UNECE Committee on

Innovation, Competitiveness and Public-Private Partnerships:

Promoting women’s entrepreneurship in transition economies

for sustainable development.

1 June 2023 (Geneva,

Switzerland).

19 Study tour to Georgia for innovation policymakers from the

SPECA subregion.

10-11 July 2023 (Tbilisi,

Georgia).

20 Capacity building: Enhancing venture capital in Azerbaijan.

July-August 2023, Virtual

mode: online platform.

21 UNECE B2B capacity building “Innovation for the Circular

Economy: Bridging Start-Ups and Corporations”).

18 October 2023 (Tashkent,

Uzbekistan

22 4th Session of the SPECA Working Group on Innovation and

Technology for Sustainable Development and Subregional

workshop: Progress Update on the Action Plan of the SPECA

Strategy on Innovation for Sustainable Development.

18-19 October 2023

(Tashkent, Uzbekistan)

23 Innovation for the Circular Economy: Empowering Start-ups to

Achieve More with Less: UNECE Study Tour for SPECA

participating States.

20-21 December 2023 (Tbilisi,

Georgia)

Source: Innovation and Technology for Sustainable Development | UNECE.

2.4 Target countries and beneficiaries

12. At its inception, the Project was intended to target six member states of the UNECE and one

member state of the United Nations Economic and Social Commission for Asia and the Pacific (ESCAP):

Afghanistan, the Republic of Azerbaijan, the Republic of Kazakhstan, the Kyrgyz Republic, the Republic

of Tajikistan, the Republic of Turkmenistan, and the Republic of Uzbekistan. During the

implementation, Afghanistan (a member of SPECA and UN ESCAP) participated in subregional

activities supported by UN ESCAP.

2.5 Key partners and other key stakeholders

13. UNECE implemented the Project in close cooperation with the UN ESCAP. Both regional

commissions also provided analytical support to the intergovernmental discussions among SPECA

participating States. In addition, under the framework of the SPECA Working Group meetings, UNECE

cooperated with the United Nations Development Programme (UNDP), Asian Infrastructure

Investment Bank (AIIB), International Telecommunications Union (ITU), United Nations Institute for

Training and Research (UNITAR), and with various ministries and governmental agencies in the SPECA

participating States.

Page 15 of 84

2.6 Resources

14. The approved budget for the Project was $ 483,316. About 45 % of the budget was allocated

for consultancy and expert fees and 24 % for workshop and study tours (Table 3).

Table 3: Approved Budget of the Project

Budget Item Amount ($) % of the Total Budget

Other staff cost - general temporary assistance $ 25,000 5.2 %

Consultants and experts $ 221,316 45.8 %

Travel of staff $ 81,000 16.8 %

Contractual services $ 23,500 4.9 %

General operating expenses $ 16,500 3.4 %

Supply and materials - 0

Furniture and equipment - 0

Workshops/Study tours (grants and contributions) $ 116,000 24.0 %

Total $ 483,316 100 %

Source: The Project Design Document, UNECE (2020).

2.7 Link to the Sustainable Development Goals (SDGs)

15. The Project was fully aligned with four SDGs such as:

- SDG 8 (“Promote sustained, inclusive, and sustainable economic growth, full and productive

employment and decent work for all”),

- SDG 9 (“Build resilient infrastructure, promote inclusive and sustainable industrialization,

and foster innovation”),

- SDG 12 (“Ensure sustainable consumption and production patterns”), and

- SDG 17 (“Strengthen the means of implementation and revitalize the Global Partnership for

Sustainable Development”).

2.8.1 Innovative elements

16. The Project was built on the strong interest of the SPECA participating States in innovative

entrepreneurship and designed to support the implementation of the SPECA Innovation Strategy for

Sustainable Development and its Action Plan. In addition, the Project activities were expected to

strengthen the beneficiary countries' capacity to integrate circular economy principles and innovative

practices into their agenda.

3. Evaluation objectives, scope and questions

3.1 Purpose and objectives

17. The evaluation was commissioned by the UNECE and was implemented in February–June

2024. Its aim was to assess the extent to which the objectives of the Project had been achieved. In

addition, the evaluation outlined any impacts the Project may have had in relation to progressing

human rights, gender equality, disability inclusion, climate change, and disaster risk reduction. The

Page 16 of 84

evaluation also addressed the impact of the COVID-19 crisis and, where relevant, the UNECE’s COVID-

19 early response in relation to the Project.

3.2.1 Evaluation scope, criteria and questions

18. The evaluation applied the criteria of relevance, efficiency, effectiveness, and sustainability

introduced by the Organisation for Economic Co-operation and Development (OECD)9. In addition, the

evaluation addressed gender mainstreaming and assessed any potential or actual impact of the

Project on progressing human rights, gender equality, disability inclusion, climate change, and disaster

risk reduction. The evaluation provided answers to key questions listed below to assess whether the

Project had delivered the optimal outcomes in the most efficient way and to identify key lessons

learned (Table 4).

Table 4: Key Evaluation Questions

CRITERIA KEY EVALUATION QUESTIONS

Relevance 1. To what extent did the Project respond to the priorities and needs of participating

countries?

2. To what extent were the Project activities consistent with global and regional priorities and

aligned with the SDGs?

3. How relevant were the Project activities vis-à-vis the programme of work of the UNECE?

What value have UNECE’s efforts added in this area?

4. To what extent were gender, human rights, disability perspectives and climate change

considerations integrated into the design and implementation of the Project? How can

these perspectives be better included in future projects’ design and implementation?

Effectiveness 5. To what extent were the Project design and set-up effective for meeting the needs of the

beneficiary countries?

6. To what extent were the Project objectives and expected results achieved?

7. To what extent were the Project activities coherent and harmonized with those of other

partners operating within the same context, particularly those of other UN system entities?

8. What were the challenges/obstacles (including COVID-19) to achieving the expected

results? How successfully did the Project overcome these?

Efficiency 9. Were the resources adequate for achieving the results?

10. Were the results achieved on time and were all activities organized efficiently?

11. To what extent were the resources used economically and how could the use of resources

be improved?

Sustainability 12. What measures were adopted to ensure that project outcomes would continue after the

Project ended and to what extent have these measures addressed the existing risks for

sustainability?

13. To what extent do the partners and beneficiaries own the outcomes of the Project? How

is the stakeholders’ engagement likely to continue, be scaled up, replicated, or

institutionalized?

Source: The TOR of the Evaluation, UNECE (2023).

9 Organization for Economic Co-operation and Development – Development Assistance Committee (OECD –

DAC)

Page 17 of 84

4. Methodology

19. The evaluation applied a participatory approach by including all relevant stakeholders in the

process to achieve a high level of ownership with respect to its output. Furthermore, the evaluation

methodology incorporated the analysis of quantitative and qualitative data, with particular attention

given to the cross-validation of information. Data were collected through different methods, including

document review, online mini-surveys among participants of the capacity building activities, as well

as online meetings and key informant interviews (KIIs) with representatives of the UNECE and the

United Nations Economic and Social Commission for Asia and the Pacific (UN ESCAP), international

and national experts engaged in the Project’s implementation, national delegates who delivered

presentations during the Project’s implementation, and the key experts from the SPECA Network of

Business Incubators and Accelerators for Sustainable Development. In addition, the evaluator

attended the workshops and discussions that took place in the course of the following selected

activities of the Project as a direct observer:

➢ “The 4th Session of the SPECA Working Group on Innovation and Technology for Sustainable

Development and Subregional Workshop: Progress Update on the Action Plan of the SPECA

Strategy on Innovation for Sustainable Development” (8-19 October 2023; Tashkent,

Uzbekistan); and

➢ “Innovation for the Circular Economy: Empowering Start-ups to Achieve More with Less:

UNECE Study Tour for SPECA participating States” (20-21 December 2023; Tbilisi, Georgia).

20. Overall, the evaluation consisted of the following main phases: the inception phase; the

fieldwork phase; the reporting phase; and the closure phase. During the inception phase, the evaluator:

➢ Reviewed the relevant documentation (primary and secondary data and Project papers).

➢ Carried out an inventory of the Project’s activities.

➢ Finalized the list of internal and external stakeholders to be reached out to during the

evaluation.

➢ Developed the evaluation framework and data collection tools such as the mini-survey and

interview protocols.

➢ Developed the sampling, data-gathering strategies, and evaluation timeframe; and

➢ Prepared the inception report for the evaluation.

21. During the fieldwork phase, the evaluator gathered data through different methods/sources

such as document analysis, KIIs (online and in-person), direct observations of the workshops, and

online mini-survey results. Meanwhile, the evaluator ensured adequate interaction and consultation

with different internal and external stakeholders in a participatory manner. Overall, the table below

(Table 5) presents the clusters of key stakeholders consulted during the course of this evaluation and

the relevant data-gathering methods (e.g., interviews and online mini-surveys).

Page 18 of 84

Table 5: Clusters of Stakeholders Contacted

Clusters of Stakeholders Data Gathering Means

The Project Team In-depth interviews (online)

Participants of the capacity building workshops Online mini- survey

Invited speakers (workshop presenters) In-depth interviews (face-to-face and online)

Members/participants of the Working and

Technical Groups

Online mini-surveys and in-depth interviews

(face-to-face and online)

Source: The Evaluation Dataset (2024).

22. The evaluator used the most reliable and appropriate sources of information and triangulated

(i.e. cross-validated) primary data (e.g., technical and financial reports of the Project, planning and

monitoring documents, reviews, and studies) and secondary data (e.g., relevant studies/notes

produced by the beneficiary countries, and the UNECE documents and standards).

23. Furthermore, the evaluation entailed the implementation of online in-depth interviews and

the distribution of online mini-surveys to the selected beneficiaries of the Project as well as key

stakeholders to gather relevant qualitative and qualitative data. In total, the evaluation conducted in-

depth interviews with key stakeholders such as the UNECE staff, independent experts, and the direct

beneficiaries/participants of the Project. ANNEX 4 presents the list of stakeholders interviewed over

the course of this evaluation.

24. Furthermore, the evaluation gathered online feedback from 25 respondents, of whom 28

percent (seven out of 25) represented state agencies of UNECE member States, 24 percent (six out of

25) came from the non-governmental sector, 24 percent (six out of 25) were independent experts,

and 20 percent (five out of 25) represented academia (Figure 1). Notably, some respondents reported

several functionalities (e.g., representing both the state sector and academia, or representing a non-

governmental organization and academia at the same time). In addition, 42 percent (10 out of 24)

respondents were female, and 54 percent (13 out of 24) of them were male10.

Figure 1: Profile of the Online Mini-survey Respondents

Source: Evaluation Dataset (2024).

10 One respondent did not answer a gender-related question.

7 (28%)

0 (0%)

3 (12%)

6 (24%)

6 (24%)

5 (20%)

3 (12%)

0 1 2 3 4 5 6 7 8 9

Government - UNECE Member State

Government - Non UNECE Member State

UN Agency

Non-Governmental Organization

Independent Expert

Academia

Other

Page 19 of 84

25. The reporting phase was mainly devoted to the preparation of the evaluation report. Here,

the evaluator ensured that the report’s assessments were objective and balanced, that its findings were

accurate and verifiable, and that its recommendations were realistic. The evaluator prepared the

evaluation report in accordance with the norms and standards of the United Nations Evaluation Group

(UNEG). More specifically, the evaluation process was aligned with the principles of integrity,

accountability, respect, and beneficence:

- Integrity: The evaluator adhered to high professional standards and acted with honesty,

transparency, and objectivity.

- Accountability: The evaluator was accountable for the quality and usefulness of the

evaluation work and ensured that the findings were communicated clearly and accurately.

- Respect: The evaluator respected the rights and dignity of all stakeholders, including the

Project staff, and national decision-makers.

- Beneficence: The evaluator strove to conduct evaluations that was beneficial to the targeted

stakeholders to promote positive social change.

26. Eventually, a draft report was sent to the UNECE to obtain final feedback.

27. During the closure phase, the evaluator gathered feedback from the UNECE and incorporated

it into the final evaluation report.

4.1 Evaluation Limitations 28. The several inherent limitations in the design of this evaluation include the following:

a. Key informants were interviewed on the basis of their availability, which did not allow for proper randomization and left open the possibility of selection bias. In addition, some key informants declined an interview. Accordingly, the sample obtained is not fully representative of the population that is supposed to be analyzed. b. The response rate for the online mini-survey distributed among the Project’s key stakeholders was modest. c. Due to the seizure of power by the Taliban in August 2021, gathering data in Afghanistan was less feasible than initially planned, and Afghanistan was thus removed from the scope of this evaluation.

5. Findings

5.1 Relevance

Finding 1: In-person interviews, online survey responses and desk research confirmed that the Project

to a great extent accommodated the needs and priorities of six out of the seven targeted countries.

However, due to the challenging political situation in Afghanistan since 2021, the country has not

participated in the Project’s activities, although it was one of the seven beneficiary countries of the

Project.

Finding 2: The evaluation validated that the Project was fully aligned with global and regional priorities

and four SDGs, namely SDG 8 (“Promote sustained, inclusive, and sustainable economic growth, full

and productive employment and decent work for all”), SDG 9 (“Build resilient infrastructure, promote

inclusive and sustainable industrialization, and foster innovation”), SDG 12 (“Ensure sustainable

Page 20 of 84

consumption and production patterns”), and SDG 17 (“Strengthen the means of implementation and

revitalize the Global Partnership for Sustainable Development”).

Finding 3: In-depth interviews confirmed that all six targeted countries prioritized innovation and

digitalization. However, the circular economy concept was not fully reflected in national or regional

priorities, largely because this was a new notion for Central Asian countries. Meanwhile, the

evaluation found that the Project’s scope was entirely relevant to UNECE sub-programme 4

(“Economic Cooperation and Integration”), which contributed to the substantive work being

completed under the SPECA framework and was partially aligned with sub-programme 1

(“Environment”).

Finding 4: The scope of the Project was aligned with the strategic reviews conducted by the UNECE

for the member States - namely the Innovation for Sustainable Development Reviews (I4SDRs) and

the Environmental Performance Reviews (EPRs).

Finding 5: Key stakeholders highlighted the exceptional position held by the UNECE in securing high-

level political support, cross-regional engagement and collaboration, and the provision of multisector

technical and analytical expertise.

Finding 6: The evaluation confirmed that the Project’s results framework did not apply gender, human

rights, disability, and climate-change-sensitive indicators by design. Moreover, the Project documents

demonstrated uneven reference to gender, human rights, disability, and climate change dimensions;

some referred to the climate change agenda, linking it to the Project’s thematic areas but neglected

to refer to the gender, human rights, and disability dimensions.

Finding 7: Despite lacking gender equality, human rights, disability, and climate change-sensitive

indicators, the UNECE incorporated climate change and the gender mainstreaming agenda in the

Project’s activities. The side event to the 16th session of the UNECE Committee on Innovation,

Competitiveness and Public-Private Partnerships promoted women’s entrepreneurship in transition

economies for sustainable development and the Work Plan for the SPECA WG on ITSD approved at

the 4th Session of the SPECA WG on ITSD referred to the climate change dimension.

Key Evaluation Question 1: To what extent did the Project respond to the priorities and needs of

the participating countries?

29. The Project assisted in putting into practice the SPECA Innovation Strategy for Sustainable Development11. It is important to note that the preparation of that strategy was originally proposed at the 12th session of the SPECA Governing Council in December 2017 in Dushanbe (Tajikistan) and was prepared by the UNECE Secretariat in cooperation with the SPECA participating States. It was then approved at the 14th Session of the SPECA Governing Council in November 2019. The above- mentioned strategy aimed to strengthen the capacity of the SPECA participating States in formulating and implementing innovation policies for sustainable development and advancing regional cooperation in this regard.

30. The strategy also emphasized that the SPECA participating States should develop and agree on an Action Plan to implement the SPECA Innovation Strategy for Sustainable Development.

31. The desk research validated the fact that the Project supported the development of a subregional gap analysis covering the targeted SPECA participating States to inform the first Action

11 SPECA_Innovation_Strategy_English.pdf (unece.org).

Page 21 of 84

Plan for the SPECA Innovation Strategy for Sustainable Development. The Project also helped to develop the Action Plan for the strategy with relevant performance indicators.

32. All key informants interviewed over the course of this evaluation confirmed that the Project activities were fully relevant for their countries’ needs. The evaluation also noted that, while Afghanistan was included in the list of targeted countries during the design phase of the Project, the country’s engagement was no longer feasible after the Taliban came to power in 2021.

33. The Project introduced diverse activities to fulfill its objectives such as the sessions of the SPECA Working Group (WG) on Innovation and Technology for Sustainable Development (ITSD), capacity building meetings and regional workshops, side events, study tours, Expert Group meetings, and the session of the SPECA Governing Council (Table 6).

Table 6: A Snapshot of the Project Activities

Type of the Project Activity

Activity Title

Sessions of the SPECA WG on ITSD

- 1st session of the SPECA WG on ITSD (30 July 2020). - 2nd session of the SPECA WG on ITSD (20-21 October 2021). - 3rd session of the SPECA WG on ITSD (20 July 2022). - 4th session of the SPECA WG on ITSD (18-19 October 2023).

Session of the SPECA Governing Council

16th session of the SPECA Governing Council: Approval of the Action Plan of the SPECA Innovation Strategy for Sustainable Development (19 November 2021).

Expert Group Meeting

Expert Group Meeting on Innovation and Technology for Sustainable Development: Proposal for a SPECA Network of Business Incubators and Accelerators for Sustainable Development (NBIASD) (19-20 July 2022).

Capacity building meetings and workshops

- Science, Technology and Innovation (STI) gap assessment of SPECA countries (26 November 2020)

- 1st meeting of the Task Force on the Roadmap for the Development of Innovation Support Infrastructure in Kyrgyz Republic until 2025 (9 June 2021).

- 2nd meeting of the Task Force on the Roadmap for the Development of the Innovation Ecosystem of Kyrgyz Republic until 2025 (8 September 2021).

- Challenges and opportunities for supporting innovative, high-growth enterprises in the SPECA countries (23 December 2021).

- Capacity building for SPECA policymakers on supporting innovative high- growth enterprises in the SPECA subregion (15 February and 17 February 2022).

- New approaches to innovation policy in the transition economies of the SPECA subregion (19 May 2022).

- Fostering innovative entrepreneurship through educational curricula in the SPECA subregion (19 September 2022).

- Effective management and development of business incubators and accelerators (27-29 September 2022).

- Development of business incubators and start-up accelerators: training for university management (21 February 2023).

- Working with venture funds and business angels: UNECE capacity building for staff of business incubators and accelerators in the SPECA subregion (16 March 2023).

- Workshop for Chief Innovation Officers on innovation for the circular economy (2 May 2023).

- Enhancing venture capital in Azerbaijan (July-August 2023).

Page 22 of 84

- Innovation for the Circular Economy: Bridging Start-Ups and Corporations (18 October 2023).

Side events ➢ Side event to the 70th session of the Economic Commission for Europe: Fostering circular solutions through innovation (3 April 2023).

➢ Side event to the 16th session of the UNECE Committee on Innovation, Competitiveness and Public-Private Partnerships: Promoting women’s entrepreneurship in transition economies for sustainable development (1 June 2023).

Study tours ➢ Study tour to Georgia for innovation policymakers from the SPECA subregion (10-11 July 2023).

➢ Innovation for the Circular Economy: Empowering Start-ups to Achieve More with Less: UNECE Study Tour for SPECA-participating States (20-21 December 2023).

Source: Innovation and Technology for Sustainable Development | UNECE.

34. All interviewed stakeholders and most of the mini-survey respondents confirmed that the Project was highly relevant to the needs of their countries. Some interviewees also referred to specific legal acts and/or national policy documents that explicitly prioritized innovation in the country's development agenda. The table below (Table 7) presents the relevant documents highlighting the importance of innovation within national frameworks. Table 7: National Policy Documents and Legal Acts Related to Promoting Innovation

Country Relevant Legal Act/Policy Document

Reference to the Innovations and Circular Economy

The Republic of Azerbaijan

Azerbaijan 2030: National Priorities for Socio-economic Development (2021).

Refers to five national socio-economic priorities for 2022-2030: (1) a sustainably growing competitive economy; (2) a dynamic, inclusive, and socially just society; (3) competitive human capital and space for modern innovations; (4) a large-scale return to the “territories liberated from occupation”; and (5) clean environment and a "green growth" country.

Decree of the President of the Republic of Azerbaijan as of 10 January 2019.

Mandated the Presidential Administration to develop a national innovation strategy and related action plan.

"National Strategy for the Development of Information Society in the Republic of Azerbaijan for 2014-2020".

Focused on the establishment of an information society, through the use of ICT by citizens, community, the private sector, and government agencies.

The Law on Science (adopted on 14 June 2016).

Encourages, inter alia, entrepreneurship and innovation among scientists to contribute towards competitive scientific-technological development in the country.

The Republic of Kazakhstan

Entrepreneur Code of the Republic of Kazakhstan (dated 29 October 2015).

Increasing the overall innovation activities in the Republic of Kazakhstan, including the promotion of the development of high-tech and knowledge-intensive industries.

National Development Plan of the Republic of Kazakhstan through 2025.

Emphasizes building a diversified and innovative economy, critical for growth and competitiveness in the global digital economy.

Kyrgyz Republic

National Strategy of Development of the Kyrgyz Republic for 2018-2040.

Sets out that Kyrgyz Republic will have a competitive economy that is focused on the application of innovative and environmentally friendly nature-saving technologies, as well as an economy that is diversified,

Page 23 of 84

balanced, and inclusive, with a favorable investment environment. States that Kyrgyz Republic will accelerate innovative socio-economic development and will strive to advance where the country has the greatest competitive advantages in a globally competitive and open economy. The Kyrgyz government's efforts are aimed at significantly improving the investment climate throughout the country in order to turn the country into an "investment oasis" that will be an attractive place for investment, which has a competitive advantage compared to other countries in the region and the Eurasian Economic Union.

Strategy for Sustainable Industrial Development of the Kyrgyz Republic 2019- 2023.

Focused on innovation, and outlined that the transition to an innovative path of development and the formation of progressive structural changes would require the adoption of urgent measures.

Decree on the approval of the strategy of sustainable development of industry of the Kyrgyz Republic for 2019-2023.

Refers to the development of information and telecommunication employment necessary for industrial development, as well as the introduction of new technologies into the industry.

Concept for the Scientific and Innovative Development of the Kyrgyz Republic until 2022.

Stipulates that the state’s policy for the creation of innovation systems should be implemented in a manner which results in a favorable economic and legal framework that promotes innovative activity, and the building of infrastructure for innovative systems and creating a state-backed system for the commercialization of new intellectual property.

The Republic of Tajikistan

National Development Strategy of the Republic of Tajikistan for the Period up to 2030.

The following three basic principles of future development form the basis for the country’s development strategy until 2030: (1) prevention or prediction (reduction) of vulnerability in future development; (2) industrialization or more efficient use of national resources; and (3) innovation or development through the integration of new developments into the country's social and economic mainstreams.

The Innovation Development Programme of the Republic of Tajikistan for 2011-2020.

The programme sets out to stimulate scientific, technical and innovative activity, form a regulatory legal framework for innovation, exploit the country's scientific potential in innovative processes, effectively use scientific and technological developments and inventions, and create and develop innovative infrastructures.

The Law on Innovation Activity (dated 16 April 2012).

The law outlined innovation activities and established the broad parameters of the state’s innovation policy.

The Republic of Turkmenistan

Concept for the Development of the Digital Economy in Turkmenistan for 2019-2025 (dated 2018).

Aimed to increase the efficiency of the functioning of all sectors of the economy and the public sector through the use of information technology.

Page 24 of 84

Concept for the Development of the Digital Education System for 2019- 2025 (dated September 2017).

Sets out to create an information-based educational environment and the effective provision of sufficient technical equipment.

The Law of Turkmenistan “On innovation activity” (dated August 2014).

Covered the legal, economic, and organizational relationships between/among the subjects of innovation and the factors arising from its implementation.

The Republic of Uzbekistan

Action Strategy on Five Priority Areas of Development of the Republic of Uzbekistan in 2017–2021

Pledged to stimulate research and innovation activities, create effective implementation mechanisms to apply scientific and innovative achievements, and establish scientific and research laboratories and high-level technology centers, as well as technology parks at universities.

Development Strategy 2022- 2026

In 2019, the Agency for Educational Institutions of the Republic of Uzbekistan, established by presidential decree under the Cabinet of Ministers, broke ground on the first four of 14 purpose-built Presidential Schools12 with the mission of delivering world-class education to the most gifted and talented students in the country. This represented the first step in the country’s aspirational vision of developing future leaders in sciences, engineering, social spheres and politics, who would contribute to the innovation economy.

Strategy of Innovative Development for 2019-2021.

Focused on the development of human capital as the main factor determining the country’s competitiveness in the world arena and its innovative progress.

Data: Evaluation Dataset (2024).

35. Over the course of the Project’s implementation, the targeted countries continued with the implementation of structural reforms to advance innovation-supportive ecosystems. For example, pursuant to the Decree of the President of the Republic of Azerbaijan, “On some measures to improve governance in the field of digitalization, innovation, high technologies and communications in the Republic of Azerbaijan” (dated 11 October 2021), the Government of Azerbaijan established the Innovation and Digital Development Agency.

36. Similarly, on 26 January 2023, the President of the Republic of Kazakhstan issued a decree to establish the National Council for Science and Technology. The Council was mandated to improve the management of the national scientific system and elaborate recommendations for setting priorities for the development of state policy in the field of science, and the scientific and technical activities of the country. Furthermore, in January 2024, the Government of Kazakhstan adopted the Law on Science and Technology Policy, which set the goal to double GDP by 2029 through, inter alia, the commercialization of scientific and technical activities. Key Evaluation Question 2: To what extent were the project activities consistent with global and

regional priorities and aligned with the SDGs?

12 The main purpose of the Presidential schools is to identify and educate gifted children through the use of

advanced technologies in the educational process, as well as to support and encourage gifted youth.

Page 25 of 84

37. Desk research validated that the Project activities were designed to contribute to the achievement of several SDGs, namely SDG 8, SDG 9, SDG 12, and SDG 17 (Table 8), as defined in the Project Design Document (hereafter “Prodoc”). Table 8: The Contribution of the Project Activities to the SDGs

SDG SDG Target

SDG 8: Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all.

Target 8.2: Achieve higher levels of economic productivity through diversification, technological upgrading and innovation, including through a focus on high-value added and labour-intensive sectors. Target 8.3: Promote development-oriented policies that support productive activities, decent job creation, entrepreneurship, creativity and innovation, and encourage the formalization and growth of micro-, small- and medium- sized enterprises, including through access to financial services.

SDG 9: Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation.

Target 9.4: By 2030, upgrade infrastructure and retrofit industries to make them sustainable, with increased resource-use efficiency and greater adoption of clean and environmentally sound technologies and industrial processes, with all countries taking action in accordance with their respective capabilities. Target 9.5: Enhance scientific research, upgrade the technological capabilities of industrial sectors in all countries, in particular developing countries, including, by 2030, encouraging innovation and substantially increasing the number of research and development workers per 1 million people and public and private research and development spending. Target 9.b: Support domestic technology development, research and innovation in developing countries, including by ensuring a conducive policy environment for, inter alia, industrial diversification and value addition to commodities.

SDG 12: Ensure sustainable consumption and production patterns.

Target 12.a: Support developing countries to strengthen their scientific and technological capacity to move towards more sustainable patterns of consumption and production.

SDG 17: Strengthen the means of implementation and revitalize the Global Partnership for Sustainable Development.

Target 17.6: Enhance North-South, South-South and triangular regional and international cooperation on and access to science, technology and innovation and enhance knowledge sharing on mutually agreed terms, including through improved coordination among existing mechanisms, in particular at the United Nations level, and through a global technology facilitation mechanism. Target 17.7: Promote the development, transfer, dissemination and diffusion of environmentally sound technologies to developing countries on favourable terms, including on concessional and preferential terms, as mutually agreed. Target 17.8: Fully operationalize the technology bank and science, technology and innovation capacity building mechanism for least developed countries by 2017 and enhance the use of enabling technology, in particular information and communications technology.

Source: The Project Design Document, UNECE (2020).

Page 26 of 84

38. At the Regional Workshop on Innovation and Technology Application for Sustainable

Development held in June 2019, the participating parties were informed about the rationale behind

developing the SPECA Innovation Strategy for Sustainable Development. One of the invited experts

highlighted that, by 2018, the SPECA participating States already faced significant challenges in

achieving the SDGs and lagged behind countries in high-income regions such as Europe (Table 9).

Overall, according to recommendations issued at the 2017 SPECA Economic Forum on “Innovation for

the SDGs in the SPECA Region,” the 12th Session of the SPECA Governing Council (organized in

December 2017) underscored the importance of elaborating an innovation strategy for the SPECA

region to support national governments in achieving the SDGs.

Table 9: Status of the SPECA participating States in Achieving the SDGs (2018 data)

Source: “SPECA Innovation Strategy for Sustainable Development,” Rumen Dobrinsky European Alliance for

Innovation (2019).13

39. Furthermore, about 71 percent (17 out of 24) of the mini-survey respondents confirmed that

the Project activities were consistent to a great extent with global and regional priorities and aligned

with the SDGs (Figure 2).

Figure 2: Feedback from Mini-survey Respondents on the Project Alignment with SDGs and Regional Priorities

Data: The Evaluation Dataset (2024).

13

https://www.unescap.org/sites/default/files/3.%20SPECA%20Innovation%20Strategy%20for%20Sustainable%

20Development.pdf.

17 (0%)

7 (0%)

(0%)

0

5

10

15

20

25

0

5

10

15

20

Were consistent to a great extent Partially consistent Not consistent at all

Page 27 of 84

40. In-person interviews validated that while the targeted countries prioritized innovations and a digitalization agenda, the concept of a circular economy (as distinct from the concept of the green economy) was still new for the Central Asian countries and was not necessarily aligned with national or regional priorities. This was reported in the policy document produced under the Project: “Implementing circular economy practices and policies in SPECA countries is still at an early stage, and many challenges need to be addressed, among which are lack of awareness and knowledge, limited access to funding and resources, infrastructure and logistics challenges, as well as cultural and regulatory barriers.”14

41. In-depth interviews and desk research revealed that the beneficiary countries15 confirmed making efforts to achieve SDG targets, including SDG 9, which sets out to build resilient infrastructure, promote sustainable industrialization, and foster innovation. Moreover, all beneficiary countries except Afghanistan completed their Voluntary National Review (VNR), 16 thereby prioritizing the achievement of the SDGs (Table 10). Table 10: Status of the VNRs in the Targeted Countries

No Beneficiary Country VNR Completion Date

1 Republic of Azerbaijan 201717, 201918 and 202119

2 Republic of Kazakhstan 201920 and 202221

3 Kyrgyz Republic 202022

4 Republic of Tajikistan 201723 and 202324

5 Republic of Turkmenistan 201925 and 202326

6 Republic of Uzbekistan 202027 and 202328 Data: The Evaluation Dataset (2024).

42. Azerbaijan was the first among the post-Soviet beneficiary countries to submit two VNR reports by 2019. In addition, the Government of Azerbaijan launched the National Information Portal on SDGs,29 serving as an interactive dashboard that collects consolidated data on the SDGs from selected government institutions and monitors progress made toward the SDGs in real-time. 43. The President of the Republic of Kazakhstan prioritized the SDGs at the Voice of Global South Summit and called on all countries to focus their efforts on achieving the SDGs30. He also pledged to

14 https://unece.org/sites/default/files/2023-10/6.1.%20%20Road%20Map%20NBIASD-

Circular%20STEP_0.pdf. 15 Noteworthy, the evaluation limitations were affected by the political changes in Afghanistan. 16 https://www.canada.ca/en/employment-social-development/programs/agenda-2030/voluntary-national-review.html; VNRs

are reports that countries submit to the United Nations to report on the actions taken and progress made to advance the

implementation of the United Nations 2030 Agenda for Sustainable Development and the 17 SDGSs.

17 Voluntary National Review 2017, Azerbaijan | High-Level Political Forum. 18 Voluntary National Review 2019, Azerbaijan | High-Level Political Forum. 19 Voluntary National Review 2021, Azerbaijan | High-Level Political Forum. 20 Voluntary National Review 2019, Kazakhstan | High-Level Political Forum. 21 Voluntary National Review 2022, Kazakhstan | High-Level Political Forum. 22 Voluntary National Review 2020, Kyrgyz Republic | High-Level Political Forum. 23 Voluntary National Review 2017, Tajikistan | High-Level Political Forum. 24 Voluntary National Reviews 2023, Tajikistan | High-Level Political Forum. 25 Voluntary National Review 2019, Turkmenistan | High-Level Political Forum. 26 Voluntary National Reviews 2023, Turkmenistan | High-Level Political Forum. 27 Voluntary National Review 2020, Uzbekistan | High-Level Political Forum. 28 Voluntary National Reviews 2023, Uzbekistan | High-Level Political Forum. 29 Home - DIM - National Information Portal for Sustainable Development (azstat.gov.az). 30 Kazakh President Tokayev Addresses SDGs, Food and Water Security at Voice of Global South Summit - The

Astana Times.

Page 28 of 84

prioritize the development of innovation and achievement of the SDGs. In addition, in 2019, the country presented for the first time its VNR31 at the UN High-Level Political Forum in New York, USA. In addition, the second VNR for Kazakhstan was conducted in 202232. 44. In 2020, Kyrgyz Republic conducted a VNR on the implementation of the SDGs in the country. Meanwhile, the SDGs were included in several national policies and reflected in the National Development Strategy (2018–2040) as well as in the Government of Kyrgyzstan’s “Unity, Trust, Creation” (2018–2022) programme.

45. In 2016, Tajikistan developed and adopted the National Development Strategy to 2030 (NDS- 2030), defining the main directions regarding achievement of the SDGs in Tajikistan. Moreover, Tajikistan also conducted VNRs in 201733 and 202334. Elsewhere, in 2023, Turkmenistan was part of a VNR on the progress of the implementation of the Global Agenda for Sustainable Development 202335. 46. Uzbekistan has confirmed its commitment to the achievement of the 2030 Agenda, and its 2017-2021 National Action Strategy served as a pathway towards the SDG’s implementation. According to the secondary data “To fully implement the SDGs, Uzbekistan is actively seeking investment and technology, at the same time strengthening closer ties with Central Asian countries.”36 Uzbekistan also carried out two VNRs, the first in 202037 and the second in 202338.

47. Furthermore, regarding global and regional priorities, several interviewees referred to the Global Innovation Index (GII)39, an annual ranking of countries' capacity for and success in innovation. It is published by the World Intellectual Property Organization (WIPO). For ease of reference, the GII scores for some of the Project’s beneficiary countries are presented below (Figure 3).

Figure 3: Global Innovation Index (GII) for the Project’s Beneficiary Countries (2019-2023)40

Source: GII Reports for 2019-2023, WIPO.

31 Voluntary National Review 2019, Kazakhstan | High-Level Political Forum. 32 Voluntary National Review 2022, Kazakhstan | High-Level Political Forum. 33 Voluntary National Review 2017, Tajikistan | High-Level Political Forum. 34 Voluntary National Reviews 2023, Tajikistan | High-Level Political Forum. 35 VNR-2023 Turkmenistan Report EN.pdf (un.org) 36 Uzbekistan's National Sustainable Development Goals: Progress and Challenges in Achieving the SDGs |

United Nations Development Programme (undp.org). 37 Voluntary National Review 2020, Uzbekistan | High-Level Political Forum. 38 Voluntary National Reviews 2023, Uzbekistan | High-Level Political Forum. 39 GII is useful for policymakers, businesses, and others to assess innovation progress based on innovation metrics

across 132 economies. It includes themes beyond rankings, i.e., funding for innovation.

40 In the context of GII, the lowest rate (i.e., “1”) signifies a stronger innovation environment.

84 82 80 93 89 79 77 79 83 81

90 94 98 94 106 100 109 103 104 111

0

93 86 82 82

2019 2020 2021 2022 2023

Azerbaijan Kazakhstan Kyrgyzstan Tajikistan Uzbekistan

Page 29 of 84

48. The GII for 2019 emphasized that “the subregion of Central Asia is noteworthy for starting to prioritize innovation activities and related policies in a sustained manner”41 and that while no GII data were available for Uzbekistan in 2019, the country was at that time making continuous progress in data collection to be included later in the GII rankings. Moreover, Uzbekistan has committed to improving its GII position and strives to enter the world’s top 50 by 2030. Meanwhile, no GII data were available for Turkmenistan and Afghanistan42 for the selected period (2019-2023).

49. The evaluation also analyzed the Terms of Reference (ToR) for the sessions of the SPECA WG on ITSD (a total of four sessions were held)43 conducted under the Project’s framework. An in-depth analysis of the ToR proved that the SPECA WG on ITSD was mandated, among other things, to:

➢ […Provide a platform for supporting progress towards achieving the Sustainable Development Goals with a focus on the SDGs directly related to innovation and technology development (including, but not limited to SDG 9 on industry, innovation and infrastructure; SDG 8 on decent work and economic growth; SDG 13 on combatting climate change and its impacts; and SDG 17 on partnerships).

➢ Improve awareness of SPECA countries on the innovation and technology development related SDGs, provide information and share best practice experiences with regard to the implementation of these SDGs, taking into account interlinkages between the SDGs…]44

50. The ToR of the SPECA WG on ITSD also outlined the WG’s composition. Each SPECA country45 had to nominate national representatives meeting the following criteria: “[…senior officials involved in the formulation and implementation or be aware of national policies and programmes, which affect national policies related to innovation and technology development in their respective countries and able to effectively follow up the recommendations and decisions of the WG on ITSD…]. Further desk research and in-depth interviews validated that the sessions of the SPECA WG on ITSD were attended by senior-level state sector officials (e.g. directors, deputy directors, board members, senior advisors, senior managers, etc.) from the SPECA participating States. Key Evaluation Question 3: How relevant were the project activities vis-à-vis the programme of

work of the UNECE? What value has UNECE’s efforts added in this area?

51. Desk research and in-person interviews confirmed that the scope of the Project was determined by the mandate of the SPECA. The evaluation also acknowledged that the UNECE subprogramme 4, “Economic Cooperation and Integration,” implemented by the Economic Cooperation and Trade Division, provided a considerable contribution to the substantive work completed under the SPECA framework. That subprogramme aims to support UNECE member States in designing and implementing policies, processes, and initiatives to strengthen good governance practices and build innovative, competitive, and inclusive societies progressing towards the achievement of the SDGs in the areas of innovation policy and public-private partnership (PPP). The figure below (Figure 4) delineates the relevance of the Project’s activities with regard to the overall scope and structure of the UNECE.

41 Global Innovation Index 2019, WIPO, 2019.

42 Officially, the Islamic Emirate of Afghanistan. 43 1st session of the SPECA WG on ITSD (30 July 2020); 2nd session of the SPECA WG on ITSD (20-21

October 2021); 3rd Session of the SPECA WG on ITSD (20 July 2022); 4th session of the SPECA WG on ITSD

(18-19 October 2023). 44 UNECE (unescap.org). 45 SPECA participating States: Afghanistan, Azerbaijan, Kazakhstan, Kyrgyz Republic, Tajikistan, Turkmenistan, and

Uzbekistan.

Page 30 of 84

Figure 4: Conceptual Roadmap of the Project and its Link to the UNECE’s Mandate

A multilateral platform that facilitates

greater economic integration &

cooperation among its member States and

promotes sustainable development and

economic prosperity.

Economic Cooperation

and

Trade Division

Contribution

Sessions of the SPECA WG on ITSD

16th Session of the SPECA Governing Council

Expert Group Meeting

Capacity building meetings & workshops

Side events & Study tours

Source: The Evaluation Dataset, 2024.

52. Four sessions of the SPECA WG on ITSD were conducted during the Project. Notably, the SPECA WG on ITSD is a subsidiary body within the governing structure of the SPECA, created on the basis of the decision of the SPECA Governing Council in November 2019. The sessions of the SPECA WG on ITSD served different objectives including:

➢ Delivering presentations on the developments in the area of innovation and technology for sustainable development in the SPECA participating States.

➢ Giving presentations on the status of digital connectivity and e-resilience as a foundation of infrastructure during the COVID-19 pandemic and related Project implementation progress.

➢ Discussing the SPECA participating States' efforts to harness innovation and enhance regional cooperation in the area of innovation and supporting implementation of the SPECA Innovation Strategy for Sustainable Development, approved by the SPECA Governing Council in November 2019.

➢ Endorsing the implementation of the Action Plan of the SPECA Innovation Strategy for Sustainable Development and getting the SPECA participating States to comment on and contribute to the activities of the UNECE and the UN ESCAP in support of the Innovation Strategy and the Action Plan.

➢ Presenting the main findings and recommendations of the following UNECE handbooks: “Business Incubators for Sustainable Development in the SPECA Subregion” (2021) 46 ;

46 Business Incubators for Sustainable Development in the SPECA Sub-region | UNECE.

UNECE sub-programme 4:

“Economic Cooperation and

Integration”

The United Nations Special

Programme for the Economies of

Central Asia (SPECA)

UNDA Project 2023N:

“Strengthening Innovation Policies for

SPECA Countries in Support of the 2030

Agenda for Sustainable Development”

Page 31 of 84

“Supporting Innovative High-Growth Enterprises in the SPECA Subregion” (2022)47; and “New Innovation Policy in the SPECA Subregion” (2023)48.

➢ Issuing presentations on the upcoming activities of the SPECA Network of Business Incubators and Accelerators for Sustainable Development targeted at promoting a circular economy.

53. Under the Project’s framework, in November 2019, the UNECE organized the 16th Session of the SPECA Governing Council that adopted 13 decisions, including the decision on the “Tashkent Statement.” The latter was introduced to promote sustainable transport and trade, using the UN’s legal norms, standards, best practice recommendations, and other tools to support sustainable development and a circular economy in the region. 49 In addition, the SPECA Governing Council approved the Action Plan of the SPECA Innovation Strategy for Sustainable Development.

54. An Expert Group Meeting on Innovation and Technology for Sustainable Development was organized immediately after the 3rd session of the SPECA WG on ITSD (in 2022) to discuss the implementation of the SPECA Work Plan on ITSD 2022-2023 in coherence with the Action Plan 2022- 2026 of the Asia-Pacific Information Superhighway. The Expert Group Meeting also served the purpose of reviewing the UNECE tools supporting the implementation of the SPECA Innovation Strategy for Sustainable Development.

55. Furthermore, according to key stakeholders, the scope of the Project was aligned with other strategic activities of the UNECE such as the Innovation for Sustainable Development Reviews (I4SDRs)50 and Environmental Performance Reviews (EPRs)51 conducted by the UNECE for its SPECA participating States as per their requests. The table below (Table 11) presents a snapshot of the latest I4SDRs and EPRs conducted for SPECA participating States.

Table 11: I4SDRs and EPRs requested by SPECA participating States

SPECA Country I4SDRs EPR

Status Year Status Year

The Republic of Azerbaijan Not conducted N/A Conducted 2022

The Republic of Kazakhstan Conducted 2012 Conducted 2019

Kyrgyz Republic Conducted 2019 Conducted 2022

The Republic of Tajikistan Conducted 2015 Conducted 2017

The Republic of Turkmenistan Not conducted N/A Conducted 2012

The Republic of Uzbekistan Conducted 2022 Conducted 2020 Source: The Evaluation Dataset, 2024.

56. Moreover, according to the key stakeholders, the EPR team from the UNECE Environmental Policy Division (subprogramme 1: “Environment”) cooperated with the Economic Cooperation and Trade Division (subprogramme 4: “Economic Cooperation and Integration”) to integrate EPR

47 Supporting Innovative High-Growth Enterprises in the SPECA sub-region, UNECE Policy Handbook |

UNECE. 48 New Innovation Policy for transition economies in the SPECA subregion | UNECE. 49 https://unece.org/sites/default/files/2022-

01/Decisions%20of%20the%2016th%20session%20of%20the%20SPECA%20Governing%20Council_ENG.pd

f. 50 The I4SDRs focus on evaluating a country's national and regional innovation system in the context of

sustainable development. 51 The UNECE conducts EPRs to assess how well member countries are doing in terms of environmental

protection.

Page 32 of 84

recommendations into the I4SDR recommendations52 as both serve the purpose of supporting the member States in achieving and monitoring review-relevant SDGs.

57. In addition, with regard to the UNECE’s efforts to add value to the Project’s implementation, key stakeholders highlighted the following main strategic contributions:

➢ Participation of key UNECE personnel and experts in, and their contextual contribution to, the Project’s activities.

➢ The UNECE assumed a specific niche in the SPECA participating States by supporting policy and analytical work, and facilitating information sharing and capacity building in particular thematic areas (e.g., the circular economy) not addressed by other international organizations in the targeted countries.

➢ The UNECE provided a collaborative platform engaging national and international stakeholders and experts, enabling them to address national and regional agendas related to the development of innovation ecosystems.

58. The evaluation also validated that the UNECE, in addition to providing thematic expertise, secured high-level political support for the Project. For example, UNECE and UN ESCAP senior-level staff (namely, the Deputy Executive Secretary, SPECA Coordinator at the UNECE and the Deputy Executive Secretary of the UN ESCAP and SPECA Coordinator at the UN ESCAP) took part in the 12th session of the SPECA Governing Council along with high-level representatives of Afghanistan, Azerbaijan, Kazakhstan, Kyrgyz Republic, Tajikistan, Turkmenistan, and Uzbekistan (the Project’s beneficiary countries). The 12th session of the SPECA Governing Council was followed by the 2017 SPECA Economic Forum on “Innovation for the SDGs in the SPECA Region” (held in Dushanbe on 5-6 December 2017) during which the Governing Council highlighted the importance of developing an innovation strategy for the SPECA region to achieve the SDGs.

Key Evaluation Question 4: To what extent were gender, human rights, disability perspectives and

climate change considerations integrated into the design and implementation of the Project? How

can these perspectives be better included in future projects design and implementation?

59. In-person interviews and online mini-survey results revealed varying opinions about the extent to which gender equality and mainstreaming, human rights, disability inclusion, and climate change dimensions were integrated into the Project. Many respondents of the online mini-survey expressed a belief that all of the above-mentioned dimensions were integrated to a great extent. For example, 54.5 percent (12 out of 22) were certain that gender equality and mainstreaming aspects were presented and integrated to a great extent, and 45.5 percent (10 out of 22) claimed they were integrated to a moderate extent. Very few respondents believed that all of the dimensions cited in the question were not reflected in the Project’s design and implementation (Figure 5).

52I.e., the recommendations related to innovative policies in environmental management and environmental

policies.

Page 33 of 84

Figure 5: Opinions of Mini-survey Respondents on Gender Equality and Mainstreaming, Human

Rights, Disability Perspectives, and Climate Change Considerations

Source: The Evaluation Dataset (2024).

60. The majority of the key stakeholders confirmed that in the future there would be a need to strengthen gender equality and mainstreaming, human rights, disability perspectives, and climate change dimensions in the Project’s scope. In addition, a few key stakeholders stated that while the UNECE had advocated a balanced approach to selecting speakers and participants, there was still a discrepancy between the gender-inclusive intentions and the actual distribution of responsibilities in the partnering national institutions in Central Asian countries, which resulted in an uneven gender representation at certain events. With regard to this, some key stakeholders claimed that engaging in dialogue with national partners was crucial to encourage more active participation of female leaders. Some also asserted that additional support, resources, and training would be required to empower women and build a culture of gender equality at decision-making levels. Key stakeholders also mentioned that the beneficiary countries were encouraged to nominate female participants to attend the Project’s events. 61. The evaluation acknowledged the receipt of very diverse feedback from the in-depth interviews regarding the extent to which the Project had integrated gender equality and mainstreaming, human rights, disability, and climate change measurements. Many interviewees were certain that the Project integrated gender equality and mainstreaming and climate change dimensions but was lacking with regard to human rights and disability.

62. The evaluation noted that the project results framework did not apply gender, human rights, disability, and climate change-sensitive indicators. It did not allow the collection of comprehensive data to monitor and track progress in the implementation of the above-mentioned dimensions. Nevertheless, the Project team organized a side event with a specific focus on the gender mainstreaming agenda, and a side event53 to the 16th session of the UNECE Committee on Innovation,

53 Flyer – Side event “Promoting women’s entrepreneurship in transition economies for sustainable

development” | UNECE.

Page 34 of 84

Competitiveness and Public-Private Partnerships promoted women’s entrepreneurship in transition economies for sustainable development.

63. Furthermore, desk research validated that the documents/reports produced during the Project referred to some of these dimensions but lacked references and/or recommendations related to others. For example, the Work Plan for the SPECA WG on ITSD approved at the 4th Session of the SPECA WG on ITSD incorporated several activities to be conducted between October 2023 and December 2025. One of these activities was focused on the climate change agenda, with the following title: “Capacity development to reduce and mitigate disaster risks in endorheic (inland) water basins in Central Asia.” Meanwhile, the policy document produced under the “Development of Synergies Between the NBIASD and Circular STEP, UNECE’s Stakeholder Engagement Network for Circular Economy Road Map” project outlined the link between the circular economy and climate change agenda, as well as highlighting the priorities of the targeted countries in addressing climate change.

5.2 Effectiveness

Finding 8: The evaluation validated that the Project’s design and implementation, including the

sessions and the composition of the SPECA WG on the ITSD and expert groups, effectively addressed

the needs of the beneficiary countries.

Finding 9: The evaluation found that the Project’s objectives and results were achieved at the outcome and output levels. At the same time, the evaluation observed overlaps between the outcome level indicator IA1.1 (“An action plan for the SPECA innovation strategy with performance indicators aligned with the SDGs is developed and agreed with the SPECA participating States and adopted by the SPECA Governing Council”) with the outcome level indicator OP1.4 (“Produce a final draft of the SPECA Innovation Strategy Action Plan and secure its formal approval by the SPECA Governing Council, with the attendance of the national focal points and experts”). Finding 10: The evaluation validated the thematic coherence of the Project activities with the interventions (which vary per country) funded by other international donors (including UN agencies), addressing the development needs of the targeted countries. Finding 11: The Project experienced some external challenges (i.e., the COVID-19 pandemic and political changes in the host and beneficiary countries). The Project team applied an agile management approach to overcome such obstacles and to deliver the planned results.

Key Evaluation Question 5: To what extent were the project design and set-up effective for

meeting the needs of the beneficiary countries?

64. To address this evaluation question, the evaluation reviewed the organizational and decision- making practices applied under the Project. Pursuant to the Prodoc, both were clearly defined among the partnering institutions and partners. Thus, the UNECE Secretariat was fully responsible for the Project’s administration and implementation. In the meantime, UN-ESCAP played the role of a committed equal partner leading or co-leading several activities and/or providing a venue for selected activities. The evaluation confirmed the above-mentioned arrangements over the course of the Project’s implementation. 65. Furthermore, according to the Prodoc, the UNECE engaged with other UN organizations through the Inter-Agency Task Team on Science, Technology, and Innovation (TTSTI). The evaluation

Page 35 of 84

found no evidence of the meetings of the Inter-Agency TTSTI being conducted under the Project framework. It did, however, confirm that meetings had taken place between the SPECA WG on ITSD and the Inter-Agency TTSTI. Elsewhere, the evaluation noted other strategic meetings having been conducted under the Project such as those of the SPECA WG on ITSD and another of the SPECA Governing Council.

66. According to key stakeholders, the existing intergovernmental structures such as the SPECA Governing Council and the SPECA WG on ITSD oversaw the progress made in implementing the SPECA Innovation Strategy for Sustainable Development. In addition, desk research validated their crucial role in addressing the needs of the beneficiary countries: “The SPECA Governing Council is the supreme management body which provides overall policy guidance to the work carried out within the SPECA framework, oversees the work of the SPECA Project Working Groups, takes stock of progress achieved, identifies priorities for the future, and approves the SPECA’s work plans. It is constituted of senior policymakers at the level of vice prime minister or minister from the SPECA countries, the UNECE, and the UN ESCAP executive secretaries.”54

67. Key stakeholders also reported that the SPECA WG on ITSD served as the main mechanism to ensure that the Project activities were aligned with the needs of the Project's targeted countries. For ease of reference, the SPECA WG on ITSD is a subsidiary body within the governing structure of the UN (SPECA) created by the decision of the SPECA Governing Council in November 201955. As reported, the SPECA WB on ITSD consists of “the senior policymakers, researchers, decision-makers, and implementers from the SPECA countries responsible for developing guidance and finding practical solutions to problems identified by the countries.”56

68. The evaluation also noted the holding of other strategic meetings such as: ➢ Task Force Meeting on the Roadmap for the Development of the Innovation Ecosystem of

Kyrgyz Republic until 2025. ➢ Expert Group Meeting on Innovation and Technology for Sustainable

Development: Proposal for a SPECA Network of Business Incubators and Accelerators for Sustainable Development (NBIASD).

➢ A side event around the 70th session of the UNECE. 69. The evaluation validated that the events listed above were attended by representatives of key state agencies from the targeted countries to ensure that the consolidated decisions aligned with the national priorities, and to endorse follow-up activities. 70. Finally, all key stakeholders, that were consulted over the course of this evaluation, confirmed that the Project design and setup were efficient when it came to addressing the needs of the targeted countries.

Key Evaluation Question 6: To what extent were the project objectives and expected results

achieved?

71. To address this key question, the evaluation reviewed the Project’s results framework to assess the progress made against output and outcome level indicators.

54 Governing Council | UNECE. 55 Innovation and Technology for Sustainable Development | UNECE. 56 SPECA Working Group on Innovation and Technology for Sustainable Development | ESCAP (unescap.org).

Page 36 of 84

Outcome-level Achievements

72. The desk research validated the full achievement of all three outcome-level indicators by the time of the Project’s completion (Table 12). The baseline for the indicators IA1.1 and IA1.2 was reported to be zero as there was neither an action plan in place for the SPECA Innovation Strategy for Sustainable Development nor any joint initiatives conducted on innovation for sustainable development involving two or more SPECA participating States by the time of the Project’s commencement. Table 12: Progress Made Against Outcome-level Indicators

Intended Outcomes (OCs) Indicator of achievement (IA) at the start of the project

Indicator of achievement (IA) at the end of the project

OC1: Strengthened co- operation on innovation to promote sustainable development and deliver agenda 2030 in the SPECA subregion.

IA1.1: An Action Plan for the SPECA innovation strategy with performance indicators aligned with the SDGs is developed and agreed with SPECA countries and adopted by the SPECA Governing Council.

Fully Achieved

IA1.2: At least two joint initiatives on innovation for sustainable development involving two or more SPECA countries developed and endorsed by the beneficiary.

Fully Achieved

OC2: Enhanced capacity of national policymakers and stakeholders to design and carry out effective innovation policy and institutional reform.

IA2.1: At least 3 of the 7 SPECA countries designed at least one initiative to improve support for innovation for sustainable development.

Fully Achieved

Source: Evaluation Dataset and Final Report for the 12th Tranche of the Development Account, UNECE (May

2024).

73. By the time of the Project’s completion, the UNECE reported the finalization of the Action Plan for the SPECA Innovation Strategy for Sustainable Development. Initially, the draft action plan for implementing the SPECA Innovation Strategy for Sustainable Development 57 was prepared for consideration at the 2nd Session of the SPECA WG on ITSD that took place on 20-21 October 2021. Later, the final version of the action plan was approved at the 16th session of the SPECA Governing Council conducted in Tashkent (Uzbekistan) on 19 November 2021. 74. With regard to IA1.2, the UNECE reported the existence of two joint initiatives:

i. The SPECA Network of Business Incubators and Accelerators for Sustainable Development, endorsed by the beneficiary countries.

ii. A proposal to establish the Digital Solutions Centre for Sustainable Development (DSCSD) for the SPECA participating States and Mongolia58.

75. Moreover, according to the key stakeholder, “The project’s activities contributed to the

development of an initiative group that launched a series of events to improve the innovation

57 2 Action Plan_SPECA Innovation Strategy for Sustainable Development_Draft_ENG.pdf (unece.org). 58 https://www.unescap.org/sites/default/d8files/event-documents/D5a-RoadmapDSCSD-E.pdf,

Page 37 of 84

ecosystem and sustainable development goals.” The stakeholder also reported on the creation of the

Kazakstan Association of the Universities Business Incubators and Accelerators (KAUBIA,

www.kaubia.kz). Overall, 22 organizations are members of the KAUBIA: 19 universities, two venture

capital funds, and one information technology hub. According to the stakeholder, “the Association

disseminates the knowledge acquired during the UNECE training to middle managers and interacts

with the government to create initiatives for the development of innovation.”

The SPECA Network of Business Incubators and Accelerators for Sustainable Development endorsed by

the beneficiary countries.

76. At the 3rd session (on 20 July 2022) of the SPECA WG on ITSD a proposal was presented to set up the SPECA Network of Business Incubators and Accelerators for Sustainable Development.59 This network was designed to provide a platform for the beneficiary countries to exchange solutions and best practices supporting innovative entrepreneurship through the development of business incubators (BIs) and business accelerators (BAs) for sustainable development in the SPECA countries. In this regard, the anticipated activities of the network included workshops, roundtables, trainings, study tours, and other information exchange initiatives covering broad areas (i.e., the transition to a circular economy, green entrepreneurship, and the commercialization of innovation and technology). In addition, the UNECE produced several policy papers and conducted trainings on the policy topics covered; all were instrumental for the network with respect to building the capacity of BIs and BAs. Specific materials included: ➢ A policy handbook - “Business Incubators for Sustainable Development in the SPECA Subregion”

(2021) (English60 and Russian61). ➢ A policy handbook - “Supporting Innovative High-growth Enterprises in the SPECA Subregion”

(2023) (English62). ➢ A policy paper - “New Innovation Policy in the Transition Economies of the SPECA Subregion”

(2024) (English63).

77. The network reported delivering online training sessions on the above-mentioned policy agendas: ➢ A webinar - "Promoting Innovative Entrepreneurship through Educational Programs in the

SPECA Subregion" (19 September 2022)64. ➢ Online training - "Effective Operation and Development of a Business Incubator and Start-up

Accelerator" (27-29 September 2022)65.

Establishment of the Digital Solutions Centre for Sustainable Development (DSCSD) for SPECA

participating States and Mongolia.

78. In October 2023, at the 4th session of the SPECA WG on ITSD, the participating parties were informed about the initiative of the Government of Kazakhstan, through the UN ESCAP resolution

59 https://www.unescap.org/sites/default/d8files/event-documents/D8a-ConceptNote-SPECA-

NetworkBusinessIncubatorsAccelerators-E.pdf. 60 https://unece.org/sites/default/files/2023-

10/3.%20Business%20incubators%20for%20sustainable%20development%20in%20SPECA-2021-ENG_0.pdf. 61 https://unece.org/sites/default/files/2023-10/3R%20SPECA-2021_RU%20Corr_0.pdf. 62https://unece.org/sites/default/files/2023-03/2227804_E_ECE_CECI_33_WEB_144dpi.pdf. 63https://drupal-main-staging.unece.org/sites/default/files/2024-01/NIPTE-SPECA-2023-EN-

WEB%20SIGNED.pdf. 64 https://unece.org/sites/default/files/2024-

02/UNECE_report_webinar_SPECA_Network__19092022_ENG_0.pdf. 65 https://unece.org/info/SPECA/events/371190.

Page 38 of 84

79/1066 and resolution 80/167 to establish the DSCSD. In May 2024, the Government of Kazakhstan issued a concept note for the development of the DSCSD entitled “UN ESCAP Digital Solutions Centre for Sustainable Development.”68 The DSCSD was expected to meet the following two main purposes: ➢ Accelerating digital transformation towards a sustainable, inclusive, and resilient digital

economy and society; and ➢ Strengthening coordination and cooperation among multiple stakeholders in the digital

transformation process.

79. The Government of Kazakhstan committed to providing premises for the DSCSD, and in the DSCSD concept note69 it presented a collaboration model between the DSCSD and other relevant institutions as follows:

➢ The UN ESCAP Asian and Pacific Centre for Transfer of Technology (APCTT) in the area of digital technology transfer.

➢ The UN ESCAP Asian and Pacific Training Centre for Information and Communication Technology for Development (APCICT) to deliver training sessions for the representatives of beneficiary countries.

➢ The UN ESCAP Asian and Pacific Centre for Development of Disaster Information Management (APDIM) in the field of digital solutions for disaster management.

80. With regard to the outcome-level indicator IA2.1, the Project reported that three SPECA participating States (Azerbaijan, Kyrgyz Republic, and Uzbekistan) out of seven had designed national initiatives to support innovation for sustainable development.

➢ Azerbaijan: The UNECE supported the development of the innovation ecosystem in Azerbaijan by conducting three online consultations 70 (July-August 2023) with national policymakers on enhancing venture capital in Azerbaijan.

➢ Kyrgyz Republic: Under the Project framework, the UNECE supported the State Agency for Intellectual Property and Innovation of Kyrgyz Republic (Kyrgyzpatent) in creating the Roadmap for the Development of the Innovation Ecosystem of Kyrgyz Republic 71 and establishing an innovation center in Bishkek as well as BIs at 11 universities across the country. In this regard, the UNECE hosted the following two virtual capacity building activities:

o Capacity building: 1st Meeting of the Task Force on the Roadmap for the Development of Innovation Support Infrastructure in Kyrgyz Republic until 2025 (9 June 2021); and

o Capacity building: 2nd Meeting of the Task Force on the Roadmap for the Development of the Innovation Ecosystem of Kyrgyz Republic until 2025 (8 September 2021).

66 https://www.unescap.org/sites/default/d8files/event-documents/ESCAP_RES_79_10_E_0.pdf. 67

https://documents.un.org/doc/undoc/gen/240/021/6e/pdf/2400216e.pdf?token=q0LJDmiM00bme5lp1x&fe=true. 68

https://acpr.unescap.org/specialsessions/documents/Explanatory_Note_DSC_for_SD_Kazakhstan_Ver_7May20

24.pdf. 69

https://acpr.unescap.org/specialsessions/documents/Explanatory_Note_DSC_for_SD_Kazakhstan_Ver_7May20

24.pdf. 70 https://unece.org/sites/default/files/2024-

02/Questions%20and%20Answers%20-%20Enhancing%20Venture%20Capital%20in%20Azerbaijan.pdf. 71 Roadmap_ Development of the Innovation Ecosystem in Kyrgyzstan_RUS.pdf (unece.org).

Page 39 of 84

➢ Uzbekistan: The UNECE supported the preparation of the Innovation for Sustainable Development Review of Uzbekistan.72 Furthermore, the country established the Agency for Innovative Development, under the newly restructured Ministry of Higher Education, Technology and Innovation, and adopted the new National Science, Technology and Innovation Policy Strategy for 2022-2030.

Output-level Achievements

81. The evaluation acknowledged that while the results framework did not refer to output-level indicators, it did allow for the measurement of outputs in the “output description” section. The evaluation confirmed the achievement of all eight outputs reported in the results framework (Table 13).

Table 13: Progress Made Against Outputs

Output number

Output description Output Status

OP1.1 Develop a subregional gap analysis covering the 7 SPECA countries to inform the first Action Plan for the SPECA Innovation Strategy.

Achieved

OP1.2 Organize a subregional workshop to present and validate the gap analysis.

Achieved

OP1.3 Organize a subregional workshop to develop an Action Plan and its performance indicators and develop joint activities between countries on innovation for sustainable development.

Achieved

OP1.4 Produce a final draft of the SPECA Innovation Strategy Action Plan and secure its formal approval by the SPECA Governing Council, with attendance of the national focal points and experts (a session within the framework of the SPECA Economic Forum and the SPECA Governing Council).

Achieved

OP1.5 Organize a subregional workshop to measure the implementation progress of the Action Plan based on its key performance indicators.

Achieved

OP2.1. Develop training materials on good practices on innovation policies and institutions for sustainable development, including technology transfer and science and technology parks (based on activities with ESCAP and IATT on science, technology, and innovation under A2.1).

Achieved

OP2.2 Within the framework of a national capacity building agreement with 3 SPECA countries, organize 6 national workshops (2 for each country) on a specific topic to address key issues at the national level identified by the gap analysis as part of national consultations carried out under OP1.1, as reflected in the SPECA Innovation Strategy Action Plan. In parallel with these events, UNECE and ESCAP will provide targeted support to three countries on three specific issues.

Achieved

OP 2.3 Development and launch of the SPECA Network of Business Incubators and Accelerators for Sustainable Development, which was established to specifically assist innovative entrepreneurship support institutions in the SPECA subregion.

Achieved

Source: The Evaluation Dataset (2024) and Final Report for the 12th Tranche of the Development Account,

UNECE (2024).

72 unece.org/sites/default/files/2022-06/9789211172966_I4SDR_UZBEKISTAN_2022_web_full%2Bcover.pdf.

Page 40 of 84

82. More specifically, under OP1.1, the Project conducted gap analyses/assessments for the targeted countries as follows:

➢ Science, Technology, and Innovations (STI) Gap Analysis of Afghanistan.73 ➢ Science, Technology, and Innovation (STI) Gap Assessment of Azerbaijan.74 ➢ Science, Technology, and Innovation (STI) Gap Assessment of Kazakhstan.75 ➢ Science, Technology, and Innovation (STI) Gap Analysis of Kyrgyz Republic.76 ➢ Science, Technology, and Innovation (STI) Gap Analysis of Uzbekistan.77 ➢ Science, Technology, and Innovation (STI) Gap Analysis of Tajikistan (English 78 and Russian79

versions). ➢ Science, Technology, and Innovation (STI) Gap Assessment of Turkmenistan.80

83. In addition, under OP1.1, the UNECE reported producing a background paper entitled “Prospects for SPECA Regional Cooperation on Innovation for Sustainable Development.”81

84. Under OP1.2, in November 2020, the UNECE delivered an online workshop entitled “Science, Technology and Innovation (STI) Gap Assessment of the SPECA Countries: Paving the Way for Action Under the SPECA Innovation Strategy for Sustainable Development”82 to present and validate the findings of the STI Gap Assessment papers produced for the SPECA participating States.

85. Under OP1.3, the UNECE organized a subregional workshop on innovation for sustainable development (2nd Session of the SPECA WG on ITSD).83 The UNECE presented a draft action plan for implementing the SPECA Innovation Strategy for Sustainable Development, and the WG on ITSD approved the submission of the draft action plan for consideration at the next session of the SPECA Governing Council.

86. Under OP1.4, the UNECE was expected to produce a final draft Action Plan for the SPECA Innovation Strategy for Sustainable Development and secure its formal approval from the SPECA Governing Council. The evaluation noticed a certain overlap between this output and outcome-level indicator IA1.1 (“An action plan for the SPECA Innovation Strategy for Sustainable Development with performance indicators aligning with the SDGs is developed and agreed with the SPECA participating States and adopted by the SPECA Governing Council”). At the same time, the evaluation validated the

73 https://unece.org/eci/documents/2021/03/reports/science-technology-and-innovation-sti-gap-analysis-

afghanistan. 74 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Azerbaijan_Report_Yulia%20Alieva.pdf. 75 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Kazakhstan_Report_Elena%20Shevchenko.pdf. 76 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Kyrgyzstan_Report_%20Aziz%20Soltobaev.pdf . 77 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Uzbekistan_Report_Nodira%20Kurbanbaeva.pdf . 78 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Tajikistan_Report_%20Bahodur%20Mengliev_0.pdf . 79 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Tajikistan_Report_%20Bahodur%20Mengliev_RUS.pdf. 80 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Turkmenistan%20Report_Yuriy%20Aronskyi.pdf . 81 https://unece.org/sites/default/files/2021-

10/UNECE_%20Background%20paper_SPECA_%20potential%20areas%20for%20cooperation%20on%20ISD

_2020_ENG.pdf . 82 SPECA_Webinar_ Report_ 26 Nov.pdf (unece.org) 83 https://unece.org/sites/default/files/2020-12/SPECA_Webinar_%20Report_%2026%20Nov.pdf.

Page 41 of 84

production of the Action Plan in two languages English84 and Russian.85 The Action Plan was approved at the 16th Session of SPECA Governing Council: “The Council approves the Action Plan for the SPECA Innovation Strategy for Sustainable Development. It expresses its gratitude to the UNECE and UN ESCAP secretariats and the SPECA participating countries for the development of this document.”86

87. Under OP1.5, the UNECE organized a subregional workshop (4th Session of the SPECA WG on ITSD) to measure the implementation progress of the Action Plan developed for the SPECA participating States. In this regard, the UNECE also produced an assessment report entitled “Assessment Report on the Implementation of the SPECA Innovation Strategy for Sustainable Development and its Action Plan.”87 At the 4th Session of the SPECA WG on ITSD, the WG also expressed its appreciation regarding the nomination of national focal points (by the SPECA participating States) to support the implementation of the Action Plan for the SPECA Innovation Strategy for Sustainable Development.

88. Under OP2.1, the UNECE produced/organized several handbooks (serving as a basis for follow- up trainings) and training materials/events, as follows:

➢ Online workshop - “Fostering Innovative Entrepreneurship through Educational Curricula in the SPECA Subregion”88 (19 September 2022).

➢ Online training for SPECA countries - “Effective Management and Development of Business Incubators and Accelerators in the SPECA Subregion”89 (27-29 September 2022).

➢ Online training for SPECA countries - “Development of Business Incubators and Start-up Accelerators: Training for University Management”90 (21 February 2023).

➢ Online training for the staff of BIs and BAs in the SPECA subregion - “Working with Venture Funds and Business Angels”91 (16 March 2023).

➢ A policy handbook - “Handbook on Business Incubators for Sustainable Development in the SPECA Subregion.”92

➢ A policy handbook - “Supporting Innovative High-growth Enterprises in the SPECA Subregion.”93

➢ A policy paper - “New Innovation Policy for Transition Economies in the SPECA Subregion.”94

84 https://unece.org/sites/default/files/2021-

11/SPECA%20Innovation%20Strategy%20for%20Sustainable%20Development_Action%20Plan_Draft%20for

%20GC_ENG_FINAL_0.pdf. 85 https://unece.org/sites/default/files/2021-

11/SPECA%20Innovation%20Strategy%20for%20Sustainable%20Development_Action%20Plan_Draft%20for

%20GC_RUS_FINAL.pdf. 86 https://unece.org/sites/default/files/2022-

01/Decisions%20of%20the%2016th%20session%20of%20the%20SPECA%20Governing%20Council_ENG.pd

f. 87 https://unece.org/sites/default/files/2023-11/Assessment%20Report_full%20first%20draft.pdf. 88 https://unece.org/speca/events/webinar-fostering-innovative-entrepreneurship-through-educational-curricula-

speca-sub. 89 Online training for SPECA countries " Effective management and development of business incubators and

accelerators" | UNECE. 90 Development of business incubators and start-up accelerators: training for university management (Развитие

бизнес-инкубаторов и стартап акселераторов – тренинг для менеджмента ВУЗов) | UNECE. 91 Working with venture funds and business angels: UNECE practical online training for staff of business

incubators and accelerators in the SPECA sub-region | UNECE. 92 https://unece.org/sites/default/files/2021-

10/Business%20incubators%20for%20sustainable%20development%20in%20SPECA-2021-ENG.pdf. 93 Supporting Innovative High-Growth Enterprises in the SPECA sub-region, UNECE Policy Handbook |

UNECE. 94 https://unece.org/sites/default/files/2024-01/NIPTE-SPECA-2023-EN-WEB%20SIGNED.pdf.

Page 42 of 84

89. Under OP2.2, the UNECE committed to “…organize 6 national workshops (2 for each country) on a specific topic to address key issues at the national level...” In the final report for the Project, the UNECE reported delivering six capacity building workshops and/or consultations: two in the Kyrgyz Republic, one in Uzbekistan, and three in Azerbaijan (Table 14). The UNECE arranged a workshop in Uzbekistan jointly with the National Office for Innovation Implementation and Technology Transfer for chief innovation officers from over 100 state-owned enterprises. The workshop aimed to enhance the efforts of the Uzbek government to foster a culture of innovation in the beneficiary state-owned enterprises. Meanwhile, the workshop in the Kyrgyz Republic was conducted as part of the 2nd Meeting of the Task Force on the Roadmap for the Development of the Innovation Ecosystem of the Kyrgyz Republic until 2025. Table 14: Project Activities Under OP 2.2

Country Event Title Implementation Date

Azerbaijan Three online consultations with local policymakers on enhancing venture capital in Azerbaijan95

July-August 2023

Kyrgyz Republic

Two capacity building activities to support development of a Roadmap for the Development of Innovation Support Infrastructure in the Kyrgyz Republic until 2025.96,97

September 2021

Uzbekistan National capacity building activities to help implement the recommendations of the UNECE Innovation for Sustainable Review of Uzbekistan98. These included UNECE capacity building workshop for Chief Innovation Officers on innovation for the circular economy99 and UNECE B2B capacity building “Innovation for the Circular Economy: Bridging Start-Ups and Corporations”.100

May 2023

Source: The Evaluation Dataset (2024).

90. Under OP2.3, the Project was expected to develop and launch the SPECA Network of Business Incubators and Accelerators for Sustainable Development (NBIASD). The evaluation confirmed the achievement of this output. First and foremost, the proposal to develop the SPECA NBIASD was presented at the Expert Group Meeting on Innovation and Technology for Sustainable Development in July 2022. At the 4th session of the SPECA WG on ITSD, the parties were updated on the progress of the creation of the SPECA NBIASD. In 2022-2023, the NBIASD organized several training sessions for BIs (Table 15). Table 15: Project Activities Under OP 2.3

Event Title Implementation Modality

Implementation Date

“Fostering Innovative Entrepreneurship through Educational Curricula in the SPECA Subregion”101

Online 19 September 2022

95 Capacity building: Enhancing venture capital in Azerbaijan, July-August 2023 | UNECE. 96 https://unece.org/info/SPECA/events/357216. 97 https://unece.org/info/SPECA/events/359822. 98 UNECE Innovation for Sustainable Development Review of Uzbekistan | UNECE. 99 https://unece.org/info/events/event/378783. 100 https://unece.org/speca/events/unece-b2b-conference-innovation-circular-economy-bridging-start-ups-and-

corporations. 101 Webinar : Fostering innovative entrepreneurship through educational curricula in the SPECA sub-region |

UNECE.

Page 43 of 84

“Effective Management and Development of Business Incubators and Accelerators"102

Online 27 - 29 September 2022

“Development of Business Incubators and Start-up Accelerators: Training for University Management”103

Online 21 February 2023

“Working with Venture Funds and Business Angels: UNECE Practical Online Training for Staff of Business Incubators and Accelerators in the SPECA Subregion”104

Online 16 March 2023

“Fostering Circular Solutions through Innovation”105 Online 3 April 2023 Source: The Evaluation Dataset (2024) and Final Report for the 12th Tranche of the Development Account,

UNECE (2024).

Key Evaluation Question 7: To what extent were the project activities coherent and harmonized

with those of other partners operating within the same context, particularly those of other UN

system entities?

91. Desk research identified other international donors engaged with the targeted countries to support innovation and the circular economy agenda (Table 16). The evaluation observed that the United Nations Development Programme (UNDP) supported the Accelerator Labs initiative, consisting of 91 labs in 115 countries, including Azerbaijan, Kyrgyz Republic, and Uzbekistan, to accelerate learning to achieve the SDGs by 2030. The Accelerator Lab network, a joint venture of the Qatar Fund for Development and the Federal Ministry for Economic Cooperation and Development of Germany, serves as an incubator tapping into local innovations to accelerate progress towards realizing the 2030 Agenda. The evaluation also identified targeted country-focused interventions supporting innovative approaches. However, the desk research confirmed that the strategic focus of international donor- funded programs and projects varied per country, as they were adjusted to fit the national needs and realities.

92. The evaluation also noted the UN Global Compact business incubator and accelerators initiative aimed to increase the capacity and awareness of business in support of achieving the SDGs.

The Republic of Azerbaijan

93. The United Nations Industrial Development Organization (UNIDO), with a financial contribution from the Government of Slovenia, has been working with the Government of Azerbaijan to strengthen the country’s innovation ecosystem and unlock its potential for digital transformation. In 2019, the Ministry of Transport, Communications and High Technologies of the Republic of Azerbaijan requested the UNIDO’s support in acquiring knowledge and experience and sharing the best practices in promoting the country’s innovation ecosystem and facilitating the competitiveness of innovative enterprises.

94. The European Commission (EC), through UNDP, supported the development and growth of civil society in the country. It did this by promoting a social entrepreneurship culture and building an enabling ecosystem and capacities for social entrepreneurship and social innovation through the “Developing Innovation-driven and Sustainable Civil Society in Azerbaijan” project.

The Republic of Kazakhstan

102 Online training for SPECA countries " Effective management and development of business incubators and

accelerators" | UNECE. 103 Development of business incubators and start-up accelerators: training for university management (Развитие

бизнес-инкубаторов и стартап акселераторов – тренинг для менеджмента ВУЗов) | UNECE. 104 Working with venture funds and business angels: UNECE practical online training for staff of business

incubators and accelerators in the SPECA sub-region | UNECE. 105 Fostering Circular Solutions through Innovation | UNECE.

Page 44 of 84

95. The Global Environment Facility (GEF) provided funding for the UNIDO to implement the “Global Cleantech Innovation Programme in Kazakhstan - Promoting cleantech innovation and entrepreneurship in small- and medium-sized enterprises for green jobs in Kazakhstan” project. It was aimed at supporting clean energy technology innovations and entrepreneurship in the country through the development of a cleantech innovation platform and accelerator programme.

96. Another notable intervention was the “Fostering Productive Innovation Project for Kazakhstan” supported by the World Bank. It incorporated the following four main thematic components:

➢ Development of a knowledge base for innovation aimed at promoting high-quality, nationally relevant research and development as well as advanced human capital activities through the provision of junior researcher group grants, senior scientist group grants, and PhD research and training grants.

➢ Innovation consortia through promoting collaboration among existing scientific research institutes and design bureaus, as well as scientific and engineering laboratories in Kazakhstan.

➢ Consolidation of the technology commercialization cycle through complementing the existing financial instruments and solutions suitable for different stages of start-up company development.

➢ Strengthening coordination of the national innovation system and enhancing the capacity of the existing institutional structures through complementing the existing financial instruments and solutions that fit the various stages of start-up company development.

Kyrgyz Republic and The Republic of Tajikistan

97. The Foreign, Commonwealth and Development Office (FCDO) allocated GBP 20,000,000 to implement the “Enterprise and Innovation Programme (EIP)” in Tajikistan and Kyrgyz Republic. The EIP has been implemented by a consortium led by DAI Global Ltd which included the University of Central Asia, VentureHive, IMKON, BizExpert, and International Alert. Under the EIP framework, DAI Global has reported establishing four hybrid “business innovation centres” (BICs) (three in Kyrgyz Republic, and one in Tajikistan), through which 40 business support programmes have been designed and delivered (26 in Kyrgyz Republic, and 14 in Tajikistan).

98. In January 2024, the EU transferred around EUR 3 million to the Cabinet of Ministers of the Kyrgyz Republic to support digital transformation. Specifically, this assistance is expected to promote the work of the Innovation Center of Digital Competencies under the High Technology Park, and promote the “Sanarip Aimak”/“Digital Aimak” system in all regions of the country. It should ensure the provision of electronic services to the population and enhance the country’s cybersecurity.

The Republic of Turkmenistan

99. UNDP is actively engaged with the Government of Turkmenistan in supporting the implementation of national priorities in the area of digital development outlined in the “Concept for the Development of the Digital Economy in Turkmenistan for 2019-2025” and the “State Programme for the Development of the Digital Economy in Turkmenistan for 2021-2025.” UNDP has also provided support through the “Assistance in the implementation of a pilot electronic data interchange system in Turkmenistan” project. It mainly addresses the issue of developing and deploying a pilot electronic data interchange system in Turkmenistan. The Republic of Uzbekistan

100. Uzbekistan benefits from various initiatives funded by donors including the United Nations Educational, Scientific and Cultural Organization (UNESCO), the Islamic Development Bank (IsDB), the

Page 45 of 84

Republic of Korea, and the UNIDO. In 2022, the Government of Uzbekistan, UNESCO, and the IsDB presented the new National Science, Technology and Innovation Policy of Uzbekistan (2022-2030), which was developed within the framework of a joint IsDB-UNESCO initiative entitled “Strengthening the inclusive Science, Technology and Innovation (STI) system in Uzbekistan.” For ease of reference, UNESCO systematically provides technical assistance and advisory, and capacity building activities to foster the development and governance of STI, and strengthen national and regional STI systems. 101. Furthermore, the Government of Korea sponsored the “UNIDO ITPO Korea Advisory Programme.” It aimed to address the issue of attracting foreign investment partners and accessing innovative technologies to become competitive in the global market. The programme contributed to the achievement of SDG 8 (“Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all”), SDG 9 (“Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation”), and SDG 17 (“Strengthen the means of implementation and revitalize the global partnership for sustainable development”).

Page 46 of 84

Table 16: International Donor-funded Initiatives to Develop the Innovation Ecosystem in the Targeted Countries

Beneficiary Country

International Donor

Intervention Title Implementatio n dates

Budget Intervention Goal/Objectives

The Republic of Azerbaijan

UNIDO and the Government of Slovenia

Development of an innovation ecosystem and support infrastructure, including a Digital Education and Innovation Centre in Azerbaijan.106

January 2020 - June 2023

EUR 199,725.50

The project objective is innovation ecosystem building, including fostering an environment for Start-Ups and Scale-Ups and skill upgrading, as well as raising awareness of the opportunities and challenges of the fourth industrial revolution (4IR) for pursuing Inclusive and Sustainable Industrial Development (ISID) in Azerbaijan.107

EU/UNDP “Developing Innovation- driven and Sustainable Civil Society in Azerbaijan”

November 2020 - November 2023

US$ 4,129,704

The project objective is to contribute to building a conducive environment for a vibrant, sustainable and innovation-driven civil society in Azerbaijan.

The Republic of Kazakhstan

GEF/UNIDO

“Global Cleantech Innovation Programme in Kazakhstan - Promoting cleantech innovation and entrepreneurship in small and medium-sized enterprises for green jobs in Kazakhstan”

January 2020 - December 2023

US$ 1,817,862

The project aimed to accelerate cleantech innovation and entrepreneurship by SMEs and start-ups, and to strengthen the cleantech innovation and entrepreneurship ecosystem of Kazakhstan.

The World Bank

“Fostering Productive Innovation Project for Kazakhstan”

December 2014 - October 2024

US$ 81.20 million

The project objective is to promote high-quality, nationally relevant research and commercialization of technologies.

Kyrgyz Republic

FCDO/DAI Central Asia Enterprise and Innovation Programme (EIP)

January 2018 - December 2024

£20,000,00 0

The programme aims at providing technical assistance to promote a stronger, diversified and more inclusive private sector in Kyrgyz Republic and Tajikistan. The Republic

of Tajikistan

106 https://open.unido.org/projects/AZ/projects/?_ga=2.93484062.229922146.1716975936-1759072182.1712852663. 107 Microsoft Word - UNIDO - ProDoc Azerbaijan Final_101219.docx.

Page 47 of 84

The Republic of Turkmenista n

UNDP

“Assistance in the implementation of a pilot electronic data interchange system in Turkmenistan”

March 2023 - December 2025

US$ 2,488,861

The project goal is to strengthen the institutional capacity of the Agency through the introduction of digital solutions in the work of the Agency in the provision of public services to citizens and assistance in the implementation of a pilot electronic data interchange system108.

The Republic of Uzbekistan

IsDB-UNESCO

Strengthening the inclusive Science, Technology and Innovation (STI) system in Uzbekistan.

Data not available online

Data not available online

Project goal is to enhance Uzbekistan's capacity to leverage science, technology, and innovation as drivers of sustainable development and inclusive growth.

Republic of Korea/UNIDO

UNIDO ITPO Korea advisory programme

January 2023 - December 2025

$613,335 This project aimed to address the issue of attracting foreign investment partners and in accessing innovative technologies to be competitive in the global market.

Source: The Evaluation Dataset (2024).

108 Assistance in the implementation of a pilot electronic data interchange system in Turkmenistan | UNDP Transparency Portal.

Key Evaluation Question 8: What were the challenges/obstacles (including COVID-19) to achieving the

expected results? How successfully did the Project overcome these?

102. Based on the in-person interviews and desk research, the evaluation identified several challenges that affected the implementation of the Project activities. It is important to note that all such challenges were external, and thus beyond the control of the Project team. First and foremost, due to travel-related restrictions associated with the COVID-19 pandemic, the Project team had to cancel some in-person activities in 2020-2022, and shift its capacity building activities to an online mode in 2022-2023.

103. Key stakeholders also mentioned staff turnover at the national institutions from beneficiary SPECA participating States (main counterparts) as a hindrance. Changes in this regard delayed the nomination of focal points, which was thus beyond the Project team’s control. Some key stakeholders also mentioned the structural changes in Azerbaijan (namely, the establishment of the Innovation and Digital Development Agency) and Uzbekistan (namely, the establishment of the new Ministry of Innovative Development). Others also cited regular changes to the delegates attending the sessions of the SPECA WG on ITSD.

104. Furthermore, due to the fluctuating political situation in Afghanistan, the country’s engagement in the Project’s activities became unfeasible as of 2021. Further political challenges arose because of the worsening security situation in Israel by the end of 2023 (Q4 2023). According to the Work Plan, the Golda Meir Mashav-Carmel International Training Center in Haifa (Israel) was expected to host a training session for beneficiaries of the Project in November 2023. However, with that being unfeasible, the UNECE replaced this event with a study tour in Georgia that was conducted in December 2023.

5.3 Efficiency

Finding 12: The quantitative and qualitative analysis of the technical and financial aspects of the Project

confirmed that the modified budget allowed for all the results to be achieved as planned at the outcome

and output levels.

Finding 13: The Project was completed with a six-month delay mainly caused by external factors. The Project team changed the delivery mode when appropriate to secure a more effective implementation of activities. Finding 14: The evaluation validated the occurrence of a shift in resource allocation after the COVID-19 pandemic, which resulted in additional operational opportunities under the Project framework (i.e., organizing regional webinars and producing extra policy handbooks). Furthermore, the budget analysis showed a significant increase in spending on budget items related to external contractual services and consultant and expert fees. Finding 15: The project staffing was limited to part-time engagement of the UNECE P-Level and G-level staff.

Page 49 of 84

Key Evaluation Question 9: Were the resources adequate for achieving the results?

105. Analysis of the actual expenditures of the Project against the budget allocated thereto at the design stage revealed that the budget utilization rate at the Project’s completion was 86.2 percent of the revised budget. More detailed analysis of the actual expenditure per budget item revealed significant changes in actual vs. planned expenditures for some budget items such as “Consultants and experts,” “Travel of staff,” “Contractual services,” and “Workshops/study tours (grants and contributions).” Indeed, the actual expenditure for the budget item “Workshops/study tours” was 29.8 percent of the planned amount, and there was a significant increase in actual expenditures compared to what was budgeted for in “Consultants and experts” (124.7 percent) and “Contractual services” (278.5 percent) (Table 17). Table 17: The Project Budget in USD (Planned vs. Actual)

Budget Item The budget

allocated at

the design

The revised

budget

Actual

expenditures

Funds

consumed (%

of revised

budget)

Other staff cost - General

Temporary Assistance (GTA)

25,000 24,000 23,959.00 99.83%

Consultants and experts 221,316 276,478 275,928.00 99.80%

Travel of staff 81,000 12,283 11,145.00 90.74%

Contractual services 23,500 65,598 65,446.00 99.77%

General operating expenses 16,500 6,475 5,545.00 85.64%

Supplies and materials 0 0 0 0.00%

Furniture and equipment 0 147 147.00 100.00%

Workshops/study tours (grants

and contributions)

116,000 50,003 34,623.00 69.24%

Total (total) 483,316 434,984109 416,793.00 86.24%

Source: The Project Financial Report (2024).

106. The above-mentioned budgetary changes were caused by the modalities of the Project having to

be altered due to COVID-19-related restrictions. In particular, travel, face-to-face workshops, and study

tours were replaced with virtual capacity building activities. This had affected the hosting fees, including

the remuneration for contracted consultants.

Key Evaluation Question 10: Were the results achieved on time and were all activities organized

efficiently?

107. The initial completion date for the Project was June 2023. However, it was extended (through a

no-cost extension) until December 2023. The desk research and in-person interviews confirmed that the

delay in the Project’s completion was caused by external factors (e.g., the COVID-19 pandemic and the

109 The UN Department of Economic and Social Affairs (UN DESA) reduced the project budget to $434,984 during the

course of the project (in 2022).

Page 50 of 84

challenging political environment in Israel) that affected the actual delivery and/or mode of the capacity

building activities. Yet, the Project team applied an agile management approach to adjust the Work Plan

to the changing circumstances.

108. All stakeholders that were consulted over the course of this evaluation confirmed their

satisfaction with the efficiency of the Project activities (Figure 6). Several key stakeholders mentioned

that these were well planned as they included an initial analysis of the existing situation/processes to

identify strategic gaps/solutions which served as a foundation for follow-up activities. At the same time,

some key stakeholders referred to the preliminary work conducted before the Project’s commencement

that was logically integrated into the Project’s design and implementation.

109. Overall, about 57 percent of the respondents (13 out of 23) confirmed that the Project activities

met their expectations to a great extent, and 43 percent (10 out of 23) claimed that the activities met

their expectations.

Figure 6: Feedback from Mini-survey Respondents on their Satisfaction with Project Events

Source: Evaluation Dataset (2024).

110. Many key stakeholders also mentioned that while online capacity building events were useful in

terms of providing information on principles and practices, face-to-face meetings and events allowed for

continuous information and knowledge sharing in an informal environment (after the completion of the

event). In this regard, many beneficiaries referred to continuing open dialogue and brainstorming with

the host country representatives and the invited experts and guests, leading to more effective idea

generation through the sharing of diverse perspectives.

Key Evaluation Question 11: To what extent were the resources used economically and how could the

use of resources be improved?

111. Analysis of the Project spending per budget item proved that the share for some budget items significantly increased after the COVID-19 pandemic broke out. In particular, the budget allocated to “Contractual services” and “Consultants and Experts” came to constitute 15.7 percent and 66.2 percent of the total budget, respectively (Figure 7).

0 (0%)

0 (0%)

13 (57%)

10 (43%)

0 2 4 6 8 10 12 14 16

Was worse than expected

Difficult to answer

Met my expectations to a great extent

Was about what I expected

Page 51 of 84

Figure 7: Budget Item share to the Total Budget (Planned vs. Actual)

Source: Evaluation Dataset (2024).

112. While the COVID-19 pandemic caused some delays and drove the need for some adjustments, the Project team efficiently reallocated resources (e.g., by reducing travel costs) to provide an alternative to the approved work plan, and carried out additional online subregional events that were not included in the original Work Plan. In addition, the UNECE published three new policy handbooks, which were not part of the approved Action Plan for the SPECA Innovation Strategy for Sustainable Development, as follows:

➢ UNECE Policy Handbook: Business Incubators for Sustainable Development in the SPECA Subregion.110

➢ UNECE Policy Handbook: Supporting Innovative High-growth Enterprises in the SPECA Subregion.111

➢ UNECE Policy Handbook: New Innovation Policy for Transition Economies in the SPECA Subregion.112

113. The evaluation acknowledged a high level of satisfaction among the beneficiaries of the Project

with regard to the quality, importance, and relevance of the Project activities, which were mainly carried

out after the COVID-19 pandemic. At the same time, key stakeholders highlighted the importance of

putting more effort, and allocating more of the budget, toward organizing face-to-face meetings and

events focusing on lessons learned, such as by introducing case studies of specific reforms in partnering

countries and sharing experiences of specific challenges faced and practical solutions applied.

114. The evaluation also noted the Project staffing cost share in relation to the total budget of the

Project constituted 5.2 percent at the design stage and 5.7 percent by completion. The UNECE part-time

assigned one P-level staff to administer the Project.113

110 unece.org/sites/default/files/2021-10/Business incubators for sustainable development in SPECA-2021-ENG.pdf. 111 https://unece.org/sites/default/files/2023-03/2227804_E_ECE_CECI_33_WEB_144dpi.pdf. 112 https://unece.org/sites/default/files/2024-01/NIPTE-SPECA-2023-EN-WEB%20SIGNED.pdf. 113 There was a part-time engagement of G-level staff as well.

5,2%

45,8%

16,8%

4,9%

3,4%

0,0%

0,0%

24,0%

5,7%

66,2%

2,7%

15,7%

1,3%

0,0%

0,0%

8,3%

0,0% 20,0% 40,0% 60,0% 80,0%

Other staff cost - General Temporary Assistance (GTA)

Consultants and experts

Travel of staff

Contractual services

General operating expenses

Supplies and materials

Furniture and equipment

Workshops/study tours (grants and contributions)

Percent of the total expenditures Percent of the total budget allocated

Page 52 of 84

5.4 Sustainability

Finding 15: The Project document incorporated a strategy to sustain the results and applied those

directly related to the Project’s implementation. Yet, the Project reports and in-person interviews

demonstrated a lack of formal commitment on behalf of the donor community to build on the Project

results.

Finding 16: The evaluation confirmed the interest of the beneficiary countries and a sense of ownership with regard to the Project results (i.e., the Action Plan for the SPECA Innovation Strategy and SPECA Network of Business Incubators and Accelerators for Sustainable Development). At the same time, the evaluation acknowledged that several strategic challenges directly affected the sustainability, scale-up, and replication of the Project results, including the lack of financial resources, political influences from neighboring countries, changes in political agenda, staff turnover, and the shortage of a structured knowledge transfer system in beneficiary countries. Finding 17: The UNECE and SPECA participating States accepted there is a need to sustain economic trends

and investment in the SPECA region and agreed to set up the SPECA Multi-Partner Trust Fund (MPTF)

under the UN’s management. The evaluation also acknowledged the proactive actions of the Secretariat

to raise funds and continue supporting SPECA participating States in strengthening their innovation

policies and putting more focus on gender equality and the empowerment of women and girls.

Key Evaluation Question 12: What measures were adopted to ensure that project outcomes would

continue after the Project ended and to what extent have these measures addressed the existing risks

for sustainability?

115. The evaluation acknowledged that the Prodoc incorporated an exit strategy to sustain the achieved results after the Project’s completion. Its exit strategy included but was not limited to:

➢ Making an essential contribution to secure the agreement on an action plan. ➢ Piloting selected activities in line with the action plan and national priorities. ➢ Starting to engage with all countries to explore areas of existing or potential interest. ➢ Engaging systematically with potential donors and lenders to help countries to formulate new

project ideas that would likely attract funding. ➢ Strengthening long-standing regional coordination mechanisms set up under the SPECA WG on

ITSD that served as both a steering entity and a network of policymakers from the subregion.

116. The evaluation verified that measurements were taken for all of the above categories except for systematic engagement with potential donors and lenders. Meanwhile, no evidence was provided to demonstrate that the UNECE had helped the targeted countries to formulate new project ideas to attract donors, with the exception of the DSCSD and KAUBIA114 concepts developed as a result of the UNECE115 and UN ESCAP support.

114 Created by the initiative group that was formed under the Project. 115 This was an initiative of the Government of Kazakhstan, through the UN ESCAP resolution 79/10 and resolution 80/1 to

establish the DSCSD.

Page 53 of 84

117. The evaluation also acknowledged the participatory nature of the Project’s design and implementation. First and foremost, the Project’s initiation and design were fully aligned with the priorities of the targeted countries, as expressed and approved at the 12th session of the SPECA Governing Council in December 2017 in Dushanbe (Tajikistan) and at the 14th Session of the SPECA Governing Council in November 2019.

118. Moreover, the evaluation confirmed that all documents produced over the course of the Project were publicly available in English and Russian. All stakeholders reached out to in the evaluation confirmed that the thematic and policy documents produced under the Project were highly appreciated. A few of them highlighted a need at the local level to transfer knowledge of the conceptual aspects of the produced document further by using plainer language.

Key Evaluation Question 13: To what extent do the partners and beneficiaries ‘own’ the outcomes of

the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or

institutionalized?

119. The evaluation also took note of the implementation modalities of the Project (i.e., the sessions of the WG on ITSD) that secured the approval and interest of the participating parties, thereby securing their ownership. At the same time, in-person interviews also revealed that on some occasions the changes in focal points affected the agendas of the beneficiary countries and the consistency of the discussions and consultations to a certain extent.

120. All national stakeholders consulted during this evaluation expressed high appreciation for the achieved results. Many also highlighted that while there is a political will to replicate and scale-up the achieved results, that might not be feasible due to financial constraints.

121. Some stakeholders also mentioned that more engagement from the third sector (i.e., representatives of private companies, associations, and non-government institutions) in the activities of similar interventions would strengthen the “buy-in” at national level and increase the chance of a follow- up improvement at country level as capacity building investment in the third sector would spur proactive cooperation and initiatives to improve the innovative ecosystem.

122. Overall, the key stakeholders reported the following challenges/risks to sustaining, replicating, and scaling-up the Project results:

➢ Availability of financial resources as some countries struggle to acquire the necessary funds to preserve the results achieved.

➢ The continuous necessity to build the capacity of human resources. ➢ Maintaining the level of involvement and participation of stakeholders and partners without a

coordination body and funds allocated for these purposes. ➢ Lack of a structured knowledge transfer mechanism to ensure that the knowledge and

methodologies are transferred locally. ➢ Lack of resources (financial and human) to maintain systematic monitoring and evaluation of the

Project results. ➢ Changes in the political or economic climate of a country or region that may affect the

sustainability of Project results.

Page 54 of 84

➢ Sociocultural factors possibly causing certain resistance or inconsistency of social and cultural norms and values that may in turn influence the implementation and sustainability of Project initiatives.

➢ Political influence from neighboring countries that are introducing their own initiatives and structures to replace those of the UNECE.

➢ The need for systematic technical support, advisory services, and updates to maintain the functionality of the systems and technologies created within the Project.

123. At the same time, it is important to highlight that at the SPECA Week (which was beyond the scope of the Project) that took place in Baku (Azerbaijan) in November 2023, the participating countries underlined the importance of collaboration in the SPECA and supported the establishment of the SPECA Multi-Partner Trust Fund (SPECA MPTF) under the UN’s management. Around the same time, the UNECE issued the ToR regulating the functional modalities of the SPECA MPTF, stating: “The SPECA MPTF is a UN- managed pooled fund mechanism established to operationalize stronger cooperation and integration in the SPECA region through programmatic interventions. The scope, theory of change, and governance mechanisms of the SPECA MPTF are specified in its Terms of Reference. The Fund will be governed by the SPECA UN Trust Fund Steering Committee (Azerbaijan). The co-chair(s) of the Steering Committee of the SPECA Trust Fund will present an update on the SPECA Trust Fund’s work to the decision-making bodies of the SPECA upon request, including annual reports, to the SPECA Governing Council.” 116

124. In addition, the evaluation verified the proactive fundraising of the UNECE team to address the requests and needs of the SPECA participating States. After the completion of the Project, UNECE drafted a new project proposal to be submitted to UN DESA117 to continue supporting SPECA participating States in strengthening their innovation policies through analytical work and the production of policy publications, capacity building activities, and facilitation of the digital transformation of innovative enterprises. This new project also considers addressing the challenges related to the gender divide and the exclusion of vulnerable groups from digital transformation.

116 SPECA ToR_EN.pdf (unece.org). 117 Under the UNDA 18th Tranche (2026-2029).

Page 55 of 84

6. Conclusions and Recommendations

125. The evaluation reached the following conclusions:

➢ The Project's participatory design and implementation modalities allowed for its complete alignment with the national and global development agendas. At the same time, the Project modalities were not heavily focused on the inclusion of the third sector (i.e., private sector, associations, and relevant non-state actors). Moreover, the operational, structural, and conceptual arrangement of the Project safeguarded its direct contribution to the UNECE’s overall mandate, and its work programme related to economic cooperation and integration and the environment. It also enabled the UNECE to leverage the best cross-sector expertise and resources to deliver the results in the most efficient manner possible.

➢ The absence of indicators to measure progress across gender, human rights, disability, and climate change dimensions resulted in an information gap and low awareness among beneficiary countries regarding the correlation between the innovation agenda and the above- mentioned dimensions. This might have lessened the impact of the Project deliverables on the capacity of the participating countries to prioritize advances in the relevant cross-sectoral processes and structures that would eventually contribute to the achievement of the relevant SDGs (i.e., SDG 5118, SDG 10119, and SDG 13120).

➢ While the Project fully achieved the planned results at the outcome and output levels121. It provided a platform for policy-level discussions within the framework of the SPECA Working Group on Innovation and Technology for Sustainable Development and the 16th session of the SPECA Governing Council and served as an information hub for SPECA participating States. AT eth same time, the overlap between the outcome level and output level indicators renders a distinctive qualitative assessment of the reported results impossible.

➢ The scope and objective of the Project exclusively focused on providing harmonized and strategic cross-country and cross-regional assistance across policy and individual capacity building levels.

➢ The Project team successfully applied agile management practices to secure an iterative and incremental implementation of the Project, focusing on flexibility, collaboration, and the satisfaction of the participating countries and institutions.

➢ ➢ The Project's human resource’s structure needed readjustment to secure advanced

backstopping (at the administration level) and continuous engagement of external thematic experts and service providers. Likewise, the implementation modalities of the Project highlighted the need for better-balanced face-to-face and online capacity building activities.

118 SDG 5: “Gender Equality.” 119 SDG 10: “Reduced Inequality.” 120 SDG 13: “Climate Action.” 121 Outcome level indicator IA1.1 (“An action plan for the SPECA innovation strategy with performance indicators

aligned with the SDGs is developed and agreed with SPECA countries and adopted by the SPECA Governing

Council”); Output level indicator OP1.4 (“Produce a final draft of the SPECA Innovation Strategy Action Plan and

secure its formal approval by the SPECA Governing Council, with the attendance of the national focal points and

experts (a session within the framework of the SPECA Economic Forum and the SPECA Governing Council)”).

Page 56 of 84

➢ The sustainability, replication, and scale-up of the Project’s results cannot be guaranteed unless the UNECE and SPECA participating States proactively address the external risks and challenges associated with financial limitations through the sustainability strategy to support the SPECA participating States across all relevant projects.

126. Based on its findings and conclusions, the evaluation issues the following four recommendations for supporting the SPECA participating States:

a. The evaluation recommends continuing to apply participatory and agile management modalities in future project design and implementation. It will allow the project teams to get real-time insights into project progress and potential issues, identify risks, and mitigate them early on, resulting in increased efficiency and effectiveness of the projects

b. The evaluation strongly advises engaging the private sector, associations, and non-state actors in project activities to strengthen in-country and regional collaboration across sectors. This would also secure a better balance of political support and a sense of ownership with industry- specific insights, data-driven evidence (providing the data crucial for policy decision-making), and in-country lobbying for policy changes.

c. The evaluation also heavily recommends revising the formulation of outcome and output level indicators to avoid overlap and ensure that complex concepts are broken down into measurable terms at the outcome level. This could be done in consultation with the relevant stakeholders to evaluate the validity and reliability of the indicators selected. Furthermore, the evaluation recommends incorporating gender, human rights, disability, and climate change indicators into the results framework to enable continuous reporting on progress made. Notably, some indicators could be developed at the output level (i.e. disability or gender indicators), and others can be elaborated based on outcome-level measurements (i.e. climate change indicators). However, it will be up to the project team to decide on the measurement layer (outcome or output) of the indicators selected, depending on the context of the upcoming projects and the potential availability and reliability of the data pertinent to the given indicator.

d. With regard to the implementation modalities of the capacity building activities, the evaluation recommends the development of online webinars with face-to-face meetings. In this regard, the UNECE is advised to arrange online national capacity building and awareness-raising events for a broader audience. However, face-to-face regional meetings and site visits would be more relevant for the sort of capacity building activities that would potentially lead to cross-sector strategic partnerships and provide first-hand experience of the practical implications and challenges of the approaches applied in the host countries.

e. The evaluation recommends reconsidering project staffing patterns to ensure smooth implementation and efficient continuity of the projects’ activities in case of emergencies. While the UNECE staff will remain engaged on a part-time basis, their level of engagement can be increased depending on the lifetime of the project. In this regard, the evaluation recommends including the budget lines associated with the salaries of the support and professional staff in the new project proposal. The availability of the qualified workforce assigned to the specific project is particularly vital for the efficient and smooth implementation of the project activities. For ease of reference, the budget share of both analytical and capacity building activities of the Project under evaluation constituted 86.6 % of the total budget).

Page 57 of 84

7. Lessons learned and good practices

127. In terms of the lesson learned, it is crucial to continue using participatory and agile management

approaches to foster stakeholder buy-in, increase adaptability, and improve decision-making. By involving

stakeholders at every step and adapting to changing circumstances, UNECE was able to navigate

challenges, such as the unexpected impact of the COVID-19 pandemic, and achieve the Project goals more

effectively. For example, by quickly transitioning to virtual meetings and online collaboration tools, the

Project team was able to maintain momentum and keep stakeholders engaged. It’s clear that participatory

and agile approaches are essential for future projects, ensuring that UNECE stays responsive, innovative,

and aligned with the needs of the member States. Overall, UNECE can maximize the benefits of

participatory and agile methodologies by incorporating techniques like design thinking and co-creation

workshops. These approaches can help us generate innovative ideas, solve complex problems, and build

strong relationships with key stakeholders.

128. Another important lesson learned is that in order to design effective indicators, it is crucial to

focus on clarity, relevance, feasibility, timeliness, flexibility, balance, and stakeholder involvement. By

ensuring that indicators are specific, measurable, achievable, relevant, and time-bound, and by aligning

them with project goals, UNECE can track progress, measure outcomes, and make informed decisions.

Additionally, considering the practicality of data collection, the balance of quantitative and qualitative

data, and stakeholder input is essential for designing a robust indicator framework.

129. Furthermore, it is important to avoid overlap between outcome and output indicators. This can

lead to confusion and hinder effective monitoring and evaluation. By carefully defining the hierarchy of

indicators and ensuring that they measure distinct levels of achievement, we can improve the quality and

usefulness of our data.

Page 58 of 84

Annexes

Annex 1 Evaluation ToR

Annex 2 Evaluation Matrix

Annex 3 Data Collection Tools

Annex 4 List of Stakeholders Interviewed

Annex 5 List of Documents Reviewed

Annex 6 Management Response and Recommendation Action Plan

Page 59 of 84

Annex 1: Evaluation ToR

TERMS OF REFERENCE

UNDA2023N: Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for

Sustainable Development.

I. Purpose

The purpose of this evaluation is to assess the extent to which the objectives of the UNECE project 2023N

“Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable

Development” were achieved.

The evaluation will assess the relevance, effectiveness, efficiency and sustainability of the project in

supporting the seven SPECA participating States in their efforts to spur innovation as a central driver of

the increasingly urgent transition from a low productivity and resource-intensive model of economic

development to knowledge-based and more sustainable economic growth, in line with the SDGs.

The evaluation will also assess any impacts the project may have had on progressing human rights, gender

equality, disability inclusion, climate change and disaster risk reduction in the context of this engagement.

The evaluation will finally look at the activities repurposed to address the impact of the COVID-19 crisis,

and assess, where relevant, UNECE’s COVID-19 early response through this project.

II. Background

The project aimed to support the seven SPECA participating States in their efforts to spur innovation as a

central driver of the increasingly urgent transition from a low productivity and resource-intensive model

of economic development to knowledge-based and more sustainable economic growth, in line with the

SDGs. It builds on a clear mandate: the SPECA participating States, recognizing this challenge, formally

decided at the 12th session of the SPECA Governing Council in 2017 to “Develop an innovation strategy

to promote sustainable development in the SPECA region” – a process that, with UNECE and ESCAP

support, has led to a final draft that will be submitted for approval at the 2019 SPECA governing council.

In this context, the project aimed to support and create sustainable further momentum around the first

steps towards putting this strategy into practice. Specifically, the objective was to strengthen institutional

capacities to harness innovation as a driver of sustainable development and regional integration.

The first target outcome was strengthened cooperation on innovation among the SPECA participating

States. Central to this outcome was to work with the SPECA participating States to put together a concrete

first action plan under the strategy, including several joint initiatives. This involved a detailed gap analysis

covering all seven countries to inform the development of the draft action plan, which was discussed in

detail at a subregional workshop before finalization and submission for approval. Towards the end of the

implementation period, a subregional workshop served to measure implementation progress based on

the indicators in the action plan. In addition to the national initiatives, two initiatives involving cooperation

among the SPECA participating States were established: (i) SPECA Network of Business Incubators and

Accelerators for Sustainable Development (NBIASD); (ii) Progress towards creation of a subregional Digital

Solutions Centre for Central Asia in Kazakhstan.

Page 60 of 84

The second outcome on enhanced capacity to design and carry out effective innovation policy and

institutional reform, aimed to address some of the leading constraints in putting the strategy into

practice among the SPECA participating States. This involved both a series of subregional seminars on

selected issues of common concern, as well as in-depth trainings for at least three countries.

Importantly, this project foresaw substantial flexibility to respond to country requests and priorities and

to build on existing or impending momentum.

ESCAP, as the co-lead in providing Secretariat support to the SPECA Working Group on Innovation and

Technology for Sustainable Development, played a role in most activities. There was an aim to engage the

donor community throughout, with the clear purpose of finding projects that contribute to the strategy

and the action plan that qualify for donor funding – sustaining the momentum beyond the closure of this

project.

III. Evaluation objectives, scope and questions

The evaluation will be guided by the objectives and indicators of achievement established in the results

framework of the project document. The evaluation will be conducted in Q4 of 2023. It will cover the full

implementation of the project, from January 2020 to December 2023 in the SPECA participating States of

the UNECE region, Azerbaijan, Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan and Uzbekistan.

The evaluation criteria are relevance, effectiveness, efficiency and sustainability.

Relevance 1. To what extent did the project respond to the priorities and needs of participating countries? 2. To what extent were the project activities consistent with global and regional priorities and

aligned with the SDGs? 3. How relevant were the project activities vis-à-vis the programme of work of the UNECE? What

value has UNECE’s efforts added in this area? 4. To what extent were gender, human rights, disability perspectives and climate change

considerations integrated into the design and implementation of the project? How can these perspectives be better included in future projects design and implementation?

Effectiveness 5. To what extent were the project design and set-up effective for meeting the needs of the

beneficiary countries? 6. To what extent were the project objectives and expected results achieved? 7. To what extent were the project activities coherent and harmonized with those of other partners

operating within the same context, particularly those of other UN system entities? 8. What were the challenges/obstacles (including COVID-19) to achieving the expected results? How

successfully did the project overcome these?

Efficiency 9. Were the resources adequate for achieving the results? 10. Were the results achieved on time and were all activities organized efficiently? 11. To what extent were the resources used economically and how could the use of resources be

improved?

Page 61 of 84

12. Sustainability 13. What measures were adopted to ensure that project outcomes would continue after the project

ended and to what extent have these measures addressed the existing risks for sustainability? 14. To what extent do the partners and beneficiaries ‘own’ the outcomes of the project? How is the

stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

IV. Evaluation approach and methodology

The evaluation will be conducted in accordance with: the ECE Evaluation Policy122; the Administrative

instruction guiding Evaluation in the UN Secretariat123; and the United Nations Evaluation Group (UNEG)

Norms and Standards for Evaluation 124 . Human rights and gender equality considerations will be

integrated at all stages of the evaluation125: (i) in the evaluation scope and questions; (ii) in the methods,

tools and data analysis techniques; (iii) in the findings, conclusions and recommendations of the final

report. The evaluator will explicitly explain how human rights, gender, disability, SDGs, and climate change

considerations will be taken into account during the evaluation.

The evaluator is required to use a mixed-method approach, including qualitative as well as quantitative

data gathering and analysis as the basis for a triangulation exercise of all available data to draw

conclusions and findings. The evaluator shall conduct online surveys and interview a wide range of diverse

stakeholders.

The evaluation should be conducted based on the following mixed methods to triangulate information:

1. A desk review of all relevant documents, including the project document and information on project

activities (monitoring data); materials developed in support of the activities (agendas, plans,

participant lists, background documents, donor reports and publications); proposed programme

budgets covering the evaluation period; project reports to the donor.

2. Online survey of key stakeholders and beneficiaries: the survey will be developed by the consultant

on her/his preferred platform.

3. Interviews (in-person and/or by telephone/video): the evaluator shall interview a wide range of

diverse stakeholders and beneficiaries including policymakers, representatives of the government,

international organisations, academia, civil society as applicable.

4. Observation of workshops and meetings, including the SPECA Working Group on Innovation and

Technology for Sustainable Development in Tashkent on 18-19 October 2023.

The evaluator will further elaborate on the evaluation methodology in the Inception Report that will

among others include the electronic survey questions, interview guide, and whether any of the six

countries will be selected for an in-depth assessment. The evaluation report will be written in English, will

consist of approximately 30 pages and will include an executive summary (max. 2 pages) describing the

evaluation methodology, key findings, conclusions and recommendations. The evaluator will also

produce an Evaluation Brief summarizing key evaluation findings, lessons learned and recommendations,

including through images and infographics.

122 UNECE Evaluation policy 123 ST/AI/2021/3 124 UNEG 2016 Norms and Standards for Evaluation 125 In line with UNEG Guidance contained in Integrating Human Rights and Gender Equality in Evaluations

Page 62 of 84

V. Evaluation schedule126

August 2023 ToR finalized

August 2023 Evaluator selected

September 2023 Contract signed. Evaluator starts the desk review

October 2023 Evaluator submits inception report including survey design

October 2023 Launch of data gathering, including survey and interviews

November 2023 Evaluator submits draft evaluation report and evaluation brief

November 2023 Evaluator submits final evaluation report and evaluation brief

VI. Resources and Management of the evaluation

An independent consultant will be engaged to conduct the evaluation under the management of the

Programme Management Unit (PMU). Payment will be made upon satisfactory delivery of work.

The PMU will manage the evaluation and will be involved in the following steps: Selection of the evaluator;

Preparation and clearance of the Terms of Reference; Provision of guidance to the Project Manager and

evaluator as needed on the evaluation design and methodology; Clearance of the final report after quality

assurance of the draft report.

The Project Manager, in consultation with the Division Director, will be involved in the following steps:

Provide all documentation needed for desk review, contact details, support and guidance to the

evaluation consultant as needed throughout the timeline of the evaluation; Advise the evaluator on the

recipients for the electronic survey and for follow-up interviews; Process and manage the consultancy

contract of the evaluator, along the key milestones agreed with PMU.

VII. Intended use / Next steps

The results of the evaluation will be used in the planning and implementation of future activities of the

UNECE Economic Cooperation and Integration Subprogramme. Findings of this evaluation will be used

when possible to:

• Improve direct project’s follow up actions, implementation of products by project beneficiaries

and dissemination of the knowledge created through the project.

• Assess the gaps and further needs of countries in the area of this project.

• Formulate tailored capacity building projects to strengthen the national capacity in enhancing

innovation.

The results of the evaluation will be reported to the inter-governmental Team of Specialists on Innovation

and Competitiveness Policies and the Committee on Innovation, Competitiveness and Public-Private

Partnerships, as well as with the Executive Committee if required.

Following the issuance of the final report, the Project Manager will develop a Management Response for

addressing the recommendations made by the evaluator. The final evaluation report, the management

response and the progress on implementation of recommendations will be publicly available on the

UNECE website.

126 Final timetable to be agreed following engagement of the evaluator

Page 63 of 84

VIII. Criteria for evaluators

The evaluator should have:

1. An advanced university degree or equivalent background in relevant disciplines.

2. Knowledge of and experience in sustainable economic development and/or innovation policy.

3. Relevant professional experience in design and management of evaluation processes with multiple

stakeholders, survey design and implementation, project planning, monitoring and management,

gender mainstreaming and human-rights due diligence.

4. Demonstrated methodological knowledge of evaluations, including quantitative and qualitative data

collection and analysis for end-of-cycle project evaluations, including demonstrated experience in

conducting questionnaires and interviews.

5. Fluency in written and spoken English and Russian.

Evaluators should declare any conflict of interest to UNECE before embarking on an evaluation project,

and at any point where such conflict occurs.

Annex 2: Evaluation Matrix

Evaluation Question Data collection methods

Data source

Relevance

To what extent did the Project respond to the priorities and needs of participating countries?

Desk research, online survey and KIIs

UNECE & UNESCAP reports (secondary data) and Project reports primary data). KIIs with the UNECE & UNESCAP staff: the project team and UNECE regional advisors. Survey among the participants (from the targeted countries) of the capacity-building activities. KIIs with the invited experts (key participants of SPECA network, international and national experts and speakers, and WG delegates.).

To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

Desk research and KIIs UNECE & UNESCAP reports (secondary data) and Project reports primary data). KIIs with the UNECE & UNESCAP staff. KIIs with the invited experts (key participants of SPECA network, international and national experts and speakers, and WG delegates.)..

How relevant were the project activities vis-à-vis the programme of work of the UNECE? What value has UNECE’s efforts added in this area?

Desk research and KIIs UNECE reports (secondary data) and Project reports primary data). KIIs with the UNECE project staff and regional advisors.

To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project? How can these perspectives be better included in future projects design and implementation?

Desk research, online survey and KIIs

UNECE & UNESCAP reports (secondary data) and Project reports (primary data). KIIs with the UNECE & UNESCAP staff. KIIs with the invited experts (key participants of SPECA network, international and national experts and speakers, and WG delegates.). Survey among the participants (from the targeted countries) of the capacity-building activities.

Effectiveness:

To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries?

Desk research, online survey and KIIs

The Project reports (primary data). Survey among the participants (from the targeted countries) of the capacity-building activities. KIIs with the UNECE & UNESCAP project staff.

To what extent were the project objectives and expected results achieved?

Desk research and KIIs The Project reports (primary data). KIIs with the UNECE & UNESCAP project staff.

To what extent were the project activities coherent and harmonized with those of other partners operating within the same context, particularly those of other UN system entities?

Desk research and KIIs The Project reports (primary data). KIIs with the UNECE & UNESCAP staff (including the project team).

Page 65 of 84

What were the challenges/obstacles (including COVID-19) to achieving the expected results? How successfully did the Project overcome these?

Desk research, online survey and KIIs

The Project reports (primary data). KIIs with the UNECE & UNESCAP staff (including the project team). KIIs with the invited experts (key participants of SPECA network, international and national experts and speakers, and WG delegates.). Survey among the participants (from the targeted countries) of the capacity-building activities.

Efficiency

Were the resources adequate for achieving the results? Desk research and KIIs The Project reports (primary data). KIIs with the UNECE & UNESCAP project team.

Were the results achieved on time and were all activities organized efficiently?

Desk research, online survey and KIIs

The Project reports (primary data). KIIs with the UNECE & UNESCAP project team

To what extent were the resources used economically and how could the use of resources be improved?

Desk research and KIIs The Project reports (primary data). KIIs with the UNECE & UNESCAP project team.

Sustainability

What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

Desk research, online survey and KIIs

The Project’s reports. KIIs with the UNECE & UNESCAP staff, and WG delegates. Survey among the participants (from the targeted countries) of the capacity-building activities.

To what extent do the partners and beneficiaries ‘own’ the outcomes of the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

Desk research, online survey and KIIs

The Project’s reports. KIIs with the UNECE & UNESCAP staff, and WG delegates. Survey among the participants (from the targeted countries) of the capacity-building activities.

Annex 3: Data Gathering Tools

Interview Protocol UNECE Project team

1. Name of Interviewee(s)

2. Organization

3. Position

4. Location

5. Date of Interview

Relevance

Q1 To what extent did the Project respond to the priorities and needs of participating countries at the design stage and/or implementation stage?

A1

Q2 Have you ever engaged UNECE Regional Advisor or United Nations Country Teams at the project design and implementation phases?

A2

Q3 To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

A3

Q4 Could you please the approaches taken to ensure that the Project responded to the priorities and needs of participating countries, regional and global priorities and SDGs?

A4

Q5 How relevant were the project activities vis-à-vis the programme of work of the UNECE?

A5

Q6 What value has UNECE’s efforts added in this area?

A6

Q7 To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project?

A7

Page 67 of 84

Q8 How can these perspectives be better included in future projects design and implementation?

A8

Effectiveness

Q9 To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries?

A9

Q10 To what extent were the project objectives and expected results achieved?

A10

Q11 What were the challenges/obstacles, internal or external, (including COVID-19) to achieving the expected results? How successfully did the Project overcome these?

A11

Q12 To what extent were the project activities coherent and harmonized with those of other partners operating within the same context, particularly those of other UN system entities? What kind of strategies/ approaches had been applied in this regard?

A12

Efficiency

Q13 Were the resources (HR, financial, etc.) adequate for achieving the results?

A13

Q14 Were the results achieved on time and were all activities organized efficiently?

A14

Q15 To what extent were the resources used economically and how could the use of resources be improved?

A15

Sustainability

Q16 What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

A16

Q17 To your opinion, to what extent do the partners and beneficiaries ‘own’ the outcomes of the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

Page 68 of 84

A17

Q18 What would you recommend to sustain, replicate and scale up the results of the project?

A18

Interview Protocol UNECE Regional Advisors

1. Name of Interviewee(s)

2. Organization

3. Position

4. Location

5. Date of Interview

Relevance

Q1 To what extent are you familiar with the goals and objectives of the project “Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable Development”?

A1

Q2 To what extent were you engaged in the project design and/or implementation?

A2

Q3 To what extent did the Project respond to the priorities and needs of participating countries127 at the design stage and/or implementation stage?

A3

Q4 To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

A4

Q5 Have you ever discussed the project objectives with the United Nations Country Teams at the project design phase?

A5

127 The Regional Advisor will be interviewed about the countries of their respective geographic coverage.

Page 69 of 84

Q6 How relevant were the project activities vis-à-vis the programme of work of the UNECE? What value has UNECE’s efforts added in this area?

A6

Q7 To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project? How can these perspectives be better included in future projects design and implementation?

A7

Effectiveness

Q8 To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries128?

A8

Effectiveness

Q9 What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

A9

Sustainability

Q10 What would you recommend to sustain, replicate and scale up the results of the project?

A10

Interview Protocol UNESCAP staff

1. Name of Interviewee(s)

2. Organization

3. Position

4. Location

5. Date of Interview

Relevance

Q1 To what extent are you familiar with the goals and objectives of the project “Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable Development”

A1

128 The Regional Advisor will be interviewed about the countries of their respective geographic coverage.

Page 70 of 84

Q2 To what extent were you engaged in the project design and/or implementation?

A2

Q2 To what extent did the Project respond to the priorities and needs of participating countries129 at the design stage and/or implementation stage?

A2

Q3 To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

A3

Q4 How relevant were the project activities vis-à-vis the programme of work of the UNECE? What value has UNECE’s efforts added in this area?130

A4

Q5 To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project? How can these perspectives be better included in future projects design and implementation?

A5

Effectiveness

Q6 To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries (Afghanistan in particular)?

A6

Q7 To what extent were the project objectives and expected results achieved?

A7

Q8 To what extent were the project activities coherent and harmonized with those of other partners operating within the same context, particularly those of other UN system entities?

A8

Q9 What were the challenges/obstacles (including COVID-19) to achieving the expected results? How successfully did the Project overcome these?

129 The Regional Advisor will be interviewed about the countries of their respective geographic coverage. 130 The questions will be addressed to the UNECE regional Advisors only.

Page 71 of 84

A9

Efficiency

Q10 Were the resources adequate for achieving the results?

A10

Q11 Were the results achieved on time and were all activities organized efficiently?

A11

Q12 To what extent were the resources used economically and how could the use of resources be improved?

A12

Sustainability

Q13 What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

A13

Q14 To what extent do the partners and beneficiaries ‘own’ the outcomes of the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

A14

Q15 What would you recommend to sustain, replicate and scale up the results of the project?

A15

A

No Interview Protocol Key participants of SPECA network, international and

national experts and speakers, Invited experts, WG

delegates.

1. Name of Interviewee(s)

2. Organization

3. Position

4. Location

5. Date of Interview

Relevance

Page 72 of 84

Q1 To what extent are you familiar with the goals and objectives of the project “Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable Development”

A1

Q2 To what extent did the Project respond to the priorities and needs of participating countries131 at the design stage and/or implementation stage?

A2

Q3 To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

A3

Q4 To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project? How can these perspectives be better included in future projects design and implementation?

A4

Effectiveness

Q5 To what extent were the project design and set-up effective for meeting the needs of the beneficiary countries (Afghanistan in particular)?

A5

Q6 What were the challenges/obstacles (including COVID-19) to achieving the expected results? How successfully did the Project overcome these?

A6

Efficiency

Q7 Were the results achieved on time and were all activities organized efficiently?

A7

Sustainability

Q8 What measures were adopted to ensure that project outcomes would continue after the Project ended and to what extent have these measures addressed the existing risks for sustainability?

A8

Q9 To what extent do the partners and beneficiaries ‘own’ the outcomes of the Project? How is the stakeholders’ engagement likely to continue, be scaled up, replicated, or institutionalized?

131 Afghanistan, Azerbaijan, Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan, and Uzbekistan.

Page 73 of 84

A9

Q10 What would you recommend to sustain, replicate and scale up the results of the project?

A10

Online Survey

Consent and Confidentiality Statement The UNECE cordially invites you to participate in the independent evaluation of the project “Strengthening innovation policies for SPECA countries in support of the 2030 Agenda for Sustainable Development” (the UNECE Project) conducted in 2020-2023. With this regard, we are sending you the online questionnaire to fill in. The information received through this questionnaire will be treated confidentially with no reference to the names of the respondents. Completing the survey will only take 15 minutes of your time. This survey will be available from April 15, 2024, through April 29, 2024. The UNECE would like to thank you in advance for your support and input.

1. Where do you work (please check the one that applies)

☐ Government - UNECE Member State

☐ Government - Non UNECE Member State

☐ UN Agency

☐ Non-Governmental Organization

☐ Independent Expert

☐ Academia

☐ Other (please specify): …….

2. Country (Please specify):

3. Gender:

Page 74 of 84

4. Please specify your participation in the Project.

☐ National consultant

☐ International consultant

☐ Participant of the event

☐ Working Group delegate

☐ Other

If other, please specify.

5. In case you took part in the capacity-building activities under this Project, please select all that apply:

Option Activity Date and venue

☐ 1st Session of the SPECA Working Group on Innovation and Technology for Sustainable Development.

30 July 2020, virtual modality (Country chair - Kazakhstan).

Subregional Workshop: Science, Technology and Innovation (STI) gap assessment of SPECA countries.

26 November 2020, virtual modality.

☐ Capacity building: 1st meeting of the Task Force on the Roadmap for the Development of Innovation Support Infrastructure in Kyrgyzstan until 2025.

9 June 2021, virtual modality.

☐ Capacity building: 2nd meeting of the Task Force on the Roadmap for the Development of the Innovation Ecosystem of Kyrgyzstan until 2025.

8 September 2021, virtual modality.

☐ 2nd Session of the SPECA Working Group on Innovation and Technology for Sustainable Development and Subregional workshop: Action plan of the SPECA Innovation Strategy for Sustainable Development.

20-21 October 2021, Hybrid mode: online platform, (Bishkek, Kyrgyzstan).

☐ 16th session of the SPECA Governing Council: Approval of the Action Plan of the SPECA Innovation Strategy for Sustainable Development.

19 November 2021, (Hybrid session) Tashkent, Uzbekistan.

☐ Capacity building: Challenges and opportunities for supporting innovative, high-growth enterprises in the SPECA countries.

23 December 2021, Virtual mode: online platform (Geneva, Switzerland).

☐ Capacity building for SPECA policymakers on supporting innovative high-growth enterprises in the SPECA sub-region.

15 February and 17 February 2022, Virtual mode: online platform (Geneva, Switzerland).

☐ Capacity building: “New approaches to innovation policy in the transition economies of the SPECA sub-region.”

19 May 2022, Virtual mode: online platform (Geneva, Switzerland).

Page 75 of 84

☐ Expert Group Meeting on Innovation and Technology for Sustainable Development: Proposal for a SPECA Network of Business Incubators and Accelerators for Sustainable Development (NBIASD).

19-20 July 2022, Hybrid mode (Almaty, Kazakhstan)

☐ 3rd Session of the SPECA Working Group on Innovation and Technology for Sustainable Development: Approval of creation of a SPECA NBIASD with national focal points.

20 July 2022, Hybrid mode (Almaty, Kazakhstan)

☐ Capacity building: Fostering innovative entrepreneurship through educational curricula in the SPECA sub-region.

19 September 2022, Virtual mode: online platform, (Geneva, Switzerland)

☐ Capacity building for SPECA countries "Effective management and development of business incubators and accelerators."

27-29 September 2022, Virtual mode: online platform, (Geneva, Switzerland).

☐ Capacity building for SPECA countries “Development of business incubators and start-up accelerators: training for university management.”

21 February 2023, Virtual mode: online platform, (Geneva, Switzerland).

☐ Working with venture funds and business angels: UNECE capacity building for staff of business incubators and accelerators in the SPECA sub-region.

16 March 2023, Virtual mode: online platform, (Geneva, Switzerland).

☐ Side event to the 70th session of the Economic Commission for Europe: Fostering circular solutions through innovation.

3 April 2023 (online, Geneva).

☐ UNECE capacity building workshop for Chief Innovation Officers on innovation for the circular economy.

2 May 2023 (Tashkent, Uzbekistan).

☐ Side event to the 16th session of the UNECE Committee on Innovation, Competitiveness and Public-Private Partnerships: Promoting women’s entrepreneurship in transition economies for sustainable development.

1 June 2023 (Geneva, Switzerland).

☐ Study tour to Georgia for innovation policymakers from the SPECA sub-region.

10-11 July 2023 (Tbilisi Georgia).

☐ Capacity building: Enhancing venture capital in Azerbaijan.

July-August 2023, Virtual mode: online platform.

☐ UNECE B2B capacity building “Innovation for the Circular Economy: Bridging Start-Ups and Corporations”).

18 October 2023 (Tashkent Uzbekistan

☐ 4th Session of the SPECA Working Group on Innovation and Technology for Sustainable Development and Subregional workshop: Progress Update on the Action Plan of the SPECA Strategy on Innovation for Sustainable Development.

18-19 October 2023 (Tashkent Uzbekistan)

☐ Innovation for the Circular Economy: Empowering Start-ups to Achieve More with Less: UNECE Study Tour for SPECA Participating States.

20-21 December 2023 (Tbilisi, Georgia)

6. To what extent did the project activities respond to the priorities and needs of your and/or targeted country (ies)?

☐ Responded to a great extent

☐ Partially responded

☐ Did not respond at all

☐ I do not know/Cannot answer

Page 76 of 84

Please specify your answer.

7. To what extent were the project activities consistent with global and regional priorities and aligned with the SDGs?

☐ Were consistent to a great extent

☐ Partially consistent

☐ Not consistent at all

☐ I do not know/have no answer Please specify your answer…

8. To what extent were the project activities you attended relevant to the needs of your and/or targeted country

(ies)?

☐ Was relevant very much

☐ The activity was more or less relevant to my country needs

☐ Was not relevant at all to my country's needs

☐ Difficult to answer Please specify your answer…

9. To what extent were the project activities you attended met your expectations?

☐ It met my expectations to a great extent

☐ It was about what I expected

☐ It was worse than expected

☐ Difficult to answer

Please specify your answer…

10. To what extent were gender, human rights, disability perspectives and climate change considerations integrated into the design and implementation of the Project?

Page 77 of 84

Measurement Gender equality & mainstreaming

Human rights Disability perspectives

Climate change issues

To a great extent

To a moderate extent

Not integrated at all

Can’t answer

11. What needs to be done to better integrate gender, human rights, disability and climate change perspectives in future projects design and implementation?

12. Please specify any challenges to sustaining the project outcomes after completion of the Project.

13. Please specify the likelihood of sustaining, scaling up, replicating, or institutionalizing the stakeholders’ engagement after the project completion. 14. What would you recommend to sustain, replicate and scale up the results of the project? 15. What would you recommend as a potential follow-up activity/project?

Thank you for your input!

Page 78 of 84

Annex 4: List of Stakeholders Interviewed

No Name Position Organisation

1 Ms. Aisuluu

Mustapakulova Head of Innovations Division Kyrgyzpatent

2 Ms. Anastasia Pankova Consultant UNECE

3 Ms. Basak Demir Co-Founder DCube Circular Economy and

Sustainability

4 Mr. Christopher Athey Economic Affairs Officer UNECE

5 Ms. Ekaterina Guznova International Expert UNECE

6 Ms. Elif Kizildeli Associate Economic Affairs

Officer UNECE

7 Ms. Elizabeth Tuerk Director of Economic

Cooperation and Trade Division UNECE

8 Mr. Farid Huseynov

Project Manager of Startups

Support and Acceleration

Department

Innovation and Digital

Development Agency of

Azerbaijan

9 Mr. Oleg Dzioubinski Regional Advisor on Sustainable

Energy UNECE

10 Ms. Sarangoo

Radnaaragchaa

Regional Advisor on

Environment UNECE

10 Mr. Serdar Ishangulyyev Head of Youth Start-ups Union of Economists of

Turkmenistan

11 Ms. Yelena Shevchenko Head of Projects

Innovation Cluster at Nazarbaev

University of the Republic of

Kazakhstan

12 Mr. Yerbolat Orazbekuly First vice-president

Association of Business Incubators

and Accelerators based at

universities (Kazakhstan)

Page 79 of 84

Annex 5: List of Documents Reviewed

1. The UNDA 2023N Project Annual Progress Report for 2020, UNECE;

2. The UNDA 2023N Project Annual Progress Report for 2021, UNECE;

3. The UNDA 2023N Project Annual Progress Report for 2022, UNECE;

4. The 2023N Project Document “Strengthening innovation policies for SPECA countries in support of the

2030 Agenda for Sustainable Development,” UNECE, 2020;

5. Draft decisions of the 4th session of the SPECA WG on ITSD, UNECE, 2023;

6. Final Report for the 12th Tranche Of The Development Account “Strengthening innovation policies for

SPECA countries in support of the 2030 Agenda for Sustainable Development,” UNECE, March 2024 ;

7. “Draft Action Plan for implementing the SPECA Innovation Strategy for Sustainable Development

Prepared for consideration by the 16th Session of the SPECA Governing Council,” UNECE, 19 November 2021,

Tashkent, Uzbekistan;

8. “SPECA Innovation Strategy for Sustainable Development,” 14th Session of the Speca Governing

Council (Ashgabat, Turkmenistan) UNECE, 21 November 2019,

unece.org/fileadmin/DAM/SPECA/documents/gc/session14/SPECA_Innovation_Strategy_English.pdf ;

9. Draft discussion paper on establishing a “Digital Solutions Centre for Sustainable Development”:

options for operational modalities, UN ESCAP, 14 July 2022,

https://www.unescap.org/sites/default/d8files/event-documents/D5a-RoadmapDSCSD-E.pdf;

10. Assessment Report on the Implementation of the SPECA Innovation Strategy for Sustainable

Development and its Action Plan, Mr. Rumen Dobrinsky and Ms. Lyudmyla Tautiyeva, Assessment Report_full

first draft.pdf (unece.org);

11. “SPECA Innovation Strategy for Sustainable Development,” Rumen Dobrinsky European Alliance for

Innovation (2019),

https://www.unescap.org/sites/default/files/3.%20SPECA%20Innovation%20Strategy%20for%20Sustainable%

20Development.pdf;

12. “Azerbaijan 2030: National Priorities for Socio-economic Development,” The Government of the

Republic of Azerbaijan, 2021, Azerbaijan 2030: National Priorities for Socio-economic Development – Policies -

IEA;

13. Development of Synergies Between the NBIASD and Circular STEP, UNECE’s Stakeholder Engagement

Network for Circular Economy Road Map, UNECE, 2023, https://unece.org/sites/default/files/2023-

10/6.1.%20%20Road%20Map%20NBIASD-Circular%20STEP_0.pdf;

14. Global Innovation Index Reports for 2019-2023, WIPO;

15. Terms Of Reference of the SPECA Working Group on Innovation and Technology for Sustainable

Development, UNECE, 2019,

https://www.unescap.org/sites/default/files/ToR_SPECA_WG_on_ITSD_English.pdf;

16. “Business Incubators for Sustainable Development in the SPECA Subregion,” UNECE, 2021, Business

Incubators for Sustainable Development in the SPECA Sub-region | UNECE.

17. “Supporting Innovative High-Growth Enterprises in the SPECA Subregion,” UNECE, 2022, Supporting

Innovative High-Growth Enterprises in the SPECA sub-region, UNECE Policy Handbook | UNECE;

18. “New Innovation Policy in the SPECA Subregion,” UNECE, 2023, New Innovation Policy for transition

economies in the SPECA subregion | UNECE;

Page 80 of 84

19. DECISIONS of the 16th Session of the SPECA Governing Council, UNECE and UNESCAP, 19 November

2021, https://unece.org/sites/default/files/2022-

01/Decisions%20of%20the%2016th%20session%20of%20the%20SPECA%20Governing%20Council_ENG.pdf;

20. Draft discussion paper on establishing a “Digital Solutions Centre for Sustainable Development”:

options for operational modalities, UNESCAP, 2023, https://www.unescap.org/sites/default/d8files/event-

documents/D5a-RoadmapDSCSD-E.pdf;

21. Concept note SPECA Network of Business Incubators and Accelerators for Sustainable Development,

D8a-ConceptNote-SPECA-NetworkBusinessIncubatorsAccelerators-E.pdf (unescap.org);

22. “Business incubators for sustainable development in the SPECA subregion UNECE Policy Handbook,”

UNECE, 2021, 3. Business incubators for sustainable development in SPECA-2021-ENG_0.pdf (unece.org);

23. Бизнес-инкубаторы для устойчивого развития в субрегионе СПЕКА, UNECE, 2021, 3R SPECA-

2021_RU Corr_0.pdf (unece.org);

24. “Supporting Innovative High-Growth Enterprises in the SPECA Sub-Region: UNECE Policy Handbook,”

UNECE, 2023, 2227804_E_ECE_CECI_33_WEB_144dpi.pdf (unece.org);

25. « New Innovation Policy for transition economies in the SPECA subregion,” UNECE, 2024, NIPTE-SPECA-

2023-EN-WEB SIGNED.pdf (unece.org);

26. Recommendations for Enhancing Venture Capital in Azerbaijan, UNECE, 2024, Questions and Answers -

Enhancing Venture Capital in Azerbaijan.pdf (unece.org);

27. “Concept, UN ESCAP Digital Solutions Centre for Sustainable Development,” Explanatory Note to the

United Nations Economic and Social Commission for Asia and the Pacific (ESCAP) Sponsored by the

Government of Kazakhstan, 2024, Explanatory_Note_DSC_for_SD_Kazakhstan_Ver_7May2024.pdf

(unescap.org);

28. “Innovation for Sustainable Development: Review of Uzbekistan,” UNECE, 2022,

unece.org/sites/default/files/2022-06/9789211172966_I4SDR_UZBEKISTAN_2022_web_full%2Bcover.pdf;

29. Science, Technology, and Innovations (STI) Gap Analysis of Afghanistan., UNECE, 2021,

https://unece.org/eci/documents/2021/03/reports/science-technology-and-innovation-sti-gap-analysis-

afghanistan;

30. Science, Technology, and Innovation (STI) Gap Assessment of Azerbaijan, UNECE, 2021,

https://unece.org/sites/default/files/2021-03/STI%20gap%20analysis_Azerbaijan_Report_Yulia%20Alieva.pdf;

31. Science, Technology, and Innovation (STI) Gap Assessment of Kazakhstan, UNECE, 2021,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Kazakhstan_Report_Elena%20Shevchenko.pdf;

32. Science, Technology, and Innovation (STI) Gap Analysis of Kyrgyzstan, UNECE, 2021,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Kyrgyzstan_Report_%20Aziz%20Soltobaev.pdf;

33. Science, Technology, and Innovation (STI) Gap Analysis of Uzbekistan, UNECE, 2021,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Uzbekistan_Report_Nodira%20Kurbanbaeva.pdf;

34. Science, Technology And Innovation Gap Analysis Of Tajikistan, UNECE, 2020,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Tajikistan_Report_%20Bahodur%20Mengliev_0.pdf;

35. Анализ пробелов в сфере науки, технологий и инноваций (НТИ) в ТАДЖИКИСТАНЕ , UNECE, 2020, 1 https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Tajikistan_Report_%20Bahodur%20Mengliev_RUS.pdf;

Page 81 of 84

36. Science, Technology and Innovation (STI) Gap Assessment of Turkmenistan, UNECE, 2020,

https://unece.org/sites/default/files/2021-

03/STI%20gap%20analysis_Turkmenistan%20Report_Yuriy%20Aronskyi.pdf;

37. Development Strategy for New Uzbekistan for 2022-2026, Appendix #1 to the Decree of the President of the Republic of Uzbekistan #PF-60 of January “28”, 2022, development_strategy_of_new_uzbekistan_for_202.pdf (ecesbf.uz).

Annex 6: Management Response and Recommendation Action Plan

Management Response to the Terminal Evaluation of the Development Account 13th Tranche Project

Project title and ID: UNDA Project 2023N “Strengthening Innovation Policies for SPECA Countries in Support of the 2030 Agenda for

Sustainable Development.”

Entity name: UNECE

Overall Remarks by Management

Recommendation Action Plan

Recommendation Management response1 Planned action Responsible unit(s) Target

completion

date

1. The evaluation recommends continuing to apply

participatory and agile management modalities in future

project design and implementation. It will allow the

project teams to get real-time insights into project

progress and potential issues, identify risks, and mitigate

them early on, resulting in increased efficiency and

effectiveness of the projects.

2. The evaluation strongly advises engaging the

private sector, associations, and non-state actors in

project activities to strengthen in-country and regional

collaboration across sectors. This would also secure a

better balance of political support and a sense of

ownership with industry-specific insights, data-driven

1 Accepted, partially accepted, or rejected.

Page 83 of 84

evidence (providing the data crucial for policy decision-

making), and in-country lobbying for policy changes.

3. The evaluation also heavily recommends revising

the formulation of outcome and output level indicators to

avoid overlap and ensure that complex concepts are

broken down into measurable terms at the outcome

level. This could be done in consultation with the relevant

stakeholders to evaluate the validity and reliability of the

indicators selected. Furthermore, the evaluation

recommends incorporating gender, human rights,

disability, and climate change indicators into the results

framework to enable continuous reporting on progress

made. Notably, some indicators could be developed at the

output level (i.e. disability or gender indicators), and

others can be elaborated based on outcome-level

measurements (i.e. climate change indicators). However,

it will be up to the project team to decide on the

measurement layer (outcome or output) of the indicators

selected, depending on the context of the upcoming

projects and the potential availability and reliability of the

data pertinent to the given indicator.

4. With regard to the implementation modalities of

the capacity building activities, the evaluation

recommends the development of online webinars with

face-to-face meetings. In this regard, the UNECE is advised

to arrange online national capacity building and

awareness-raising events for a broader audience.

However, face-to-face regional meetings and site visits

would be more relevant for the sort of capacity building

activities that would potentially lead to cross-sector

strategic partnerships and provide first-hand experience

of the practical implications and challenges of the

approaches applied in the host countries.

5. The evaluation recommends reconsidering project staffing patterns to ensure smooth

Page 84 of 84

implementation and efficient continuity of the projects’ activities in case of emergencies. While the UNECE staff will remain engaged on a part-time basis, their level of engagement can be increased depending on the lifetime of the project. In this regard, the evaluation recommends including the budget lines associated with the salaries of the support and professional staff in the new project proposal. The availability of the qualified workforce assigned to the specific project is particularly vital for the efficient and smooth implementation of the project activities. For ease of reference, the budget share of both analytical and capacity building activities of the Project under evaluation constituted 86.6 % of the total budget).

Name Title Signature Date

1

AZERBAIJAN

Hydrocarbons have driven rapid economic expansion

The economy of Azerbaijan has expanded rapidly over the last two decades, as a result of the exploitation of its hydrocarbon resources. It multiplied by a factor of almost 4 between 2000 and 2010 but growth was more volatile and subdued in the following decade. This dramatic growth has been accompanied by some limited economic diversification, which has been linked to a large extent to the domestic demand create by energy-related rents. Regional disparities are significant and agriculture remains the main source of employment, accounting for around 36% of total employment.

Source: IMF, Central Bank of Azerbaijan

Further advances in trade facilitation would facilitate tapping into the country’s transit potential

In order to address existing structural weaknesses arising from the lack of economic diversification, the authorities undertook a simplification of the trade and custom regime. However, despite improvements, an untapped potential to leverage the geographic position of the country as a transit route remains. Trade facilitation indicators still point to a situation where there is room for further advances, including on logistics aspects. The quality of roads has improved significantly in recent years and is among the best in the region. However, a recent World Bank study estimates that the cost of road accidents was about 2.0% of GDP in 2019.

Source: OECD, World Bank Development Indicators

A stronger private sector would facilitate economic diversification

The necessary diversification of the economy depends on the development of the private sector. A high degree of informality constraints productivity increases. The strong state footprint restricts opportunities for private engagement. Regulatory quality has increased from rather low levels, but it has significant room for improvement. Competition is limited and leads to a high degree of concentration in some sectors. Innovation initiatives have focussed on high-tech sectors but the support to activities with lower technological intensity is more limited.

2

Source: SDG UNECE database, EBRD

Fossil fuels dominate the energy mix and contribute to air pollution

Energy efficiency compares well with other countries in the region but it has deteriorated. Fossil fuels subsidies, although they have been declining, are large, thus creating disincentives from saving and contributing to existing pollution problems. In line with what it is observed in other hydrocarbon producers in the region, the uptake of renewable energy is limited. However, recent initiatives have introduced favourable conditions to facilitate the development of renewable energy resources and tap into existing potential for development.

Source: OECD, SDG UNECE database

Public finances are healthy but the financial sector remains underdeveloped

General government debt was only about 17% of GDP by the end of 2022. While a public deficit emerged in 2020, under the impact of the COVID-19 pandemic, the budget has shown a surplus in recent years. The State Oil Fund has accumulated large savings from hydrocarbons revenues. By contrast, financial intermediation is limited with loans representing only 15% of GDP in 2022. Public- Private Partnerships are constrained by uncomplete legislative development. FDI outside the energy sector is limited.

Source: State Oil Fund of Azerbaijan, IMF

1

AZERBAIJAN

Hydrocarbons have driven rapid economic expansion

The economy of Azerbaijan has expanded rapidly over the last two decades, as a result of the exploitation of its hydrocarbon resources. It multiplied by a factor of almost 4 between 2000 and 2010 but growth was more volatile and subdued in the following decade. This dramatic growth has been accompanied by some limited economic diversification, which has been linked to a large extent to the domestic demand create by energy-related rents. Regional disparities are significant and agriculture remains the main source of employment, accounting for around 36% of total employment.

Source: IMF, Central Bank of Azerbaijan

Further advances in trade facilitation would facilitate tapping into the country’s transit potential

In order to address existing structural weaknesses arising from the lack of economic diversification, the authorities undertook a simplification of the trade and custom regime. However, despite improvements, an untapped potential to leverage the geographic position of the country as a transit route remains. Trade facilitation indicators still point to a situation where there is room for further advances, including on logistics aspects. The quality of roads has improved significantly in recent years and is among the best in the region. However, a recent World Bank study estimates that the cost of road accidents was about 2.0% of GDP in 2019.

Source: OECD, World Bank Development Indicators

A stronger private sector would facilitate economic diversification

The necessary diversification of the economy depends on the development of the private sector. A high degree of informality constraints productivity increases. The strong state footprint restricts opportunities for private engagement. Regulatory quality has increased from rather low levels, but it has significant room for improvement. Competition is limited and leads to a high degree of concentration in some sectors. Innovation initiatives have focussed on high-tech sectors but the support to activities with lower technological intensity is more limited.

2

Source: SDG UNECE database, EBRD

Fossil fuels dominate the energy mix and contribute to air pollution

Energy efficiency compares well with other countries in the region but it has deteriorated. Fossil fuels subsidies, although they have been declining, are large, thus creating disincentives from saving and contributing to existing pollution problems. In line with what it is observed in other hydrocarbon producers in the region, the uptake of renewable energy is limited. However, recent initiatives have introduced favourable conditions to facilitate the development of renewable energy resources and tap into existing potential for development.

Source: OECD, SDG UNECE database

Public finances are healthy but the financial sector remains underdeveloped

General government debt was only about 17% of GDP by the end of 2022. While a public deficit emerged in 2020, under the impact of the COVID-19 pandemic, the budget has shown a surplus in recent years. The State Oil Fund has accumulated large savings from hydrocarbons revenues. By contrast, financial intermediation is limited with loans representing only 15% of GDP in 2022. Public- Private Partnerships are constrained by uncomplete legislative development. FDI outside the energy sector is limited.

Source: State Oil Fund of Azerbaijan, IMF

MANAGEMENT RESPONSE

Evaluation Title: Global Initiative towards post-Covid-19 resurgence of the MSME sector (UNDA project 2023W)

Period of Review: 2020 - 2023

Date of Evaluation Report:

30/04/2024

Approved by:

Chief, PMU Nicolas Dath-Baron

Date Signature 05/06/2024

Cleared by:

Director of Divisions: Elisabeth Türk, Economic Cooperation and Trade Division Dario Liguti, Sustainable Energy Division

18/07/2024

Prepared by:

UNECE Project Managers: Hana Daoudi Oleg Dzioubinski

05/06/2024

1

2

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

Recommendation 1:

1. (i) UNCTAD should further leverage the experience gained through the project to map out how components of the Entrepreneurship Policy Framework relate to and can be best positioned to support MSMEs in their recovery from different types of crises and (ii) DESA and the Regional Commissions should add their analysis of how their work can contribute towards the objective.

2. UNCTAD, DESA and the Regional Commissions could build on the project to map their areas of intervention in support of the MSME sector, as well as capacities and knowledge on the implementation of the EPF components, including in a crisis context, and opportunities for broadening the uptake of EPF components at the regional level based on national needs. UNCTAD, DESA and the Regional

Accepted Through existing and future UNDA projects and subject to the availability of funds and the potential for synergies withs UNCTAD, DESA and the remaining regional commissions, UNECE will focus its actions on three key areas:

Integration of evidence-based sectoral insights, focused on the impact of trade disruptions and fluctuations in national, regional and global sustainable development for guiding the development of resilience strategies for MSMEs within the EPF.

Capacity-building: UNECE can help MSMEs implement the strategies outlined in the EPF through training on new technologies, sustainable trade practices to mitigate the effects of crises and capitalize on new market opportunities as they arise.

Strengthening broad-based networks and partnerships, which bring together national and local governments; UNECE sustainable trade expert communities, enterprise

UNECE Economic Cooperation and Trade Division and Sustainable Energy Division

2 years Yes

3

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

Commissions could further consider identifying areas of joint interventions that would trigger complementarities and synergies between the agencies. This could involve collaborating on the development of capacities of all national and sub-national actors, i.e. Governments, MSMEs, other partners (e.g. Chamber of Commerce, Business Incubators, etc.) in line with United Nations Sustainable Development Cooperation Frameworks (SDCF) in respective countries; strengthening or contributing to facilitate access to finance for the MSMEs (including seed money, grants, access to credit, etc.); improving the coordination of MSMEs related policies across ministries; increasing interventions at the local level, such as by supporting NGOs or MSMEs outside of the main cities. The application of a human rights-based approach, gender responsiveness and

support organizations working with UNECE and MSMEs to support the practical application of the EPF.

4

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

inclusion of other vulnerable groups (LNOB) should be ensured.

Recommendation 2:

UNCTAD should continue building on the momentum generated by the project to continue fostering knowledge exchanges and promoting the Entrepreneurship Policy Framework.

UNCTAD, DESA and the Regional Commissions should identify means to more meaningfully continue to share good practices and lessons learned on the demand-driven support they provide to the MSME sector, including on areas such as green/circular economy; innovation policies and ecosystems; MSME formalization and strengthening MSME capacities for the empowerment and leadership of women, youth and groups in vulnerable situations. UNCTAD and the Regional Commissions could also

Accepted Through existing and future UNDA projects and subject to the availability of funds and the potential for synergies with UNCTAD, DESA and the remaining regional commissions, UNECE’s actions can be framed around the following activities:

Facilitating knowledge exchange by participating in, and possibly co-hosting national and regional events for discussing emerging issues and sharing national and regional experiences.

Promoting best practices by leveraging its extensive network and expertise to compile and disseminate case studies and success stories from its member states.

Supporting national, regional and sector- specific initiatives that align with the EPF, including contributing to the development of policies and programs that foster innovation, the transition to a circular

UNECE Economic Cooperation and Trade Division and Sustainable Energy Division

2 years Yes

5

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

propose organizing regional events and/or a global conference to promote entrepreneurship policy and further advance the goals and impact of the Surge project. Furthermore, learnings from the experiences with the Surge about external partnerships should encourage UNCTAD to promote the EPF and entrepreneurship development to other UN agencies also engaged in this area (e.g. ILO, FAO, UN Women, etc.), to development banks, or to related initiatives such as the recent Global Accelerator on Jobs and Social Protection. UNCTAD could also consider joining and supporting events that promote the achievement of SDG 8.3 as an avenue to further promote the EPF.

economy, and enable MSMEs to integrate sustainability practices throughout their supply chains.

Recommendation 3:

The DA-PMT should develop a clear framework for assessing the costs and benefits of implementing a global or

6

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

inter-regional project versus regional projects.

As a global crisis can affect regions and countries differently, global or joint projects should be developed only when there are clear benefits of joint implementation. Thus, a global or inter-regional vs. regional response would not be a priori decision, but a result of a clear assessment. A specific guideline or framework should be developed for this. Some of the assessment criteria could include the need or demand from member States for an integrated response; the range of common versus entity-specific activities and added value of complementary interventions (i.e. in terms of geographical coverage, reach of the target population, capacity, and/or coverage of multiple reinforcing technical areas, or networks and partnerships); capability to execute joint interventions (e.g. in terms of time, resources, logistics, and flexibility to

7

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

pivot interventions without bureaucratic procedures); coordination costs (which increase with the number of participating UN entities); project inception modalities and governance and knowledge management requirements to facilitate collaboration and knowledge sharing; scaling and sustainability plans; expected results of spreading resources versus concentrating on fewer countries; and so forth. The time taken to conduct such assessments will increase project coherence and effectiveness and facilitate the identification of the financial and human resources needed for project implementation. The network of DA Focal Points is a key existing asset for this assessment.

Recommendation 4:

Implementing UN entities should ensure that they have a comprehensive Results Framework for the entire project as well as an

Accepted This guidance already exists for UNECE projects, whether funded from XB or UNDA. UNECE divisions will continue to apply and enrich in future projects.

UNECE Economic Cooperation and Trade Division and Sustainable Energy Division

8

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

adequate monitoring plan, with indicators that are designed to support the ongoing monitoring.

The results framework should have one objective and ideally have one outcome per cluster/workstream. The objective should state the intended goal of the project, describe the overall achievement targeted by the project, involving a process of change aimed at meeting the needs of identified beneficiaries, and reflect the overall funding available to the project. Each objective should include reference to the project’s beneficiaries and its substantive focus. The objective should not attempt to explain the ways in which the project intends to achieve the objective (i.e. it should not include the word ‘through’ or describe the internal work of the UN using verbs such as ‘support’, ‘facilitate’ or ‘contribute’). The outcomes (OCs) should describe the changes that are expected to occur as a result of the completion of outputs.

9

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

The OCs should be achievable within the project’s timeframe and budget, and should be specific enough to be measured by the associated indicators of achievement. The indicators of achievement (IAs) should provide measures for monitoring progress towards achieving the OCs and reporting on them after completion of the project. Every indicator needs to provide clearly defined baselines, units of measurement and targets, detailing the quantity, quality and timing of expected results. The monitoring, evaluation, accountability, and learning (MEAL) system should also be designed to capture HRBA, gender and LNOB aspects. In phased interventions or during project execution, any changes in the logframe if/when pivoting activities should be clearly explained to the wider team.

Recommendation 5: Accepted This guidance already exists for UNECE projects, whether funded from XB or

UNECE Economic Cooperation and

10

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

Implementing UN entities should ensure that sufficient resources are allocated to project coordination, technical collaboration, and partnership building.

The absorption capacity of implementing entities can be challenged by crisis response projects which add to the planned programme of work. This is further compounded by projects that come with an extensive UN partnership and a global scope. Sufficient resources should be dedicated to global coordination and to building global partnerships with strategic stakeholders (e.g., UN organisations engaged in supporting the MSME sector; development banks). Capacities should also be directed to supporting technical collaboration and the staff implementing interventions, including towards synergy or liaison with the UNCTs. When designing the project, implementing UN entities should consider featuring

UNDA. UNECE divisions will continue to apply and enrich in future projects.

Trade Division and Sustainable Energy Division

11

Evaluation Recommendation (a) Management

response (b) Accepted, partially

accepted or rejected

Management plan Actions to be taken, and/or

comments about partial acceptance or rejection (c)

Responsible unit (d)

Timeframe (e)

Resources required

(Y or N) (f)

coordination and partnership- building in the Theory of Change or logframe of the project. Tools to support continuous connections and knowledge exchange, and to ensure institutional memory should be part of the response package, such as a project website, SharePoint space for all team members, and a Yammer network or Teams channel. The integration of cross-cutting aspects (HRBA, gender responsiveness, LNOB) also requires expertise with sufficient and dedicated time and resources. Guidance could be development to project managers on how to do this.

Annex 7: Evaluation Terms of Reference Evaluation of the United Nations Development Account 12th tranche “Global Initiative towards post-Covid-19 resurgence of the MSME sector” (2023W)

TERMS OF REFERENCE

A. BACKGROUND A1. ABOUT THE DEVELOPMENT ACCOUNT

The Development Account (DA) is a mechanism to fund capacity development projects of the 10 economic and social entities of the United Nations Secretariat, namely: the Department of Economic and Social Affairs (DESA), the Economic Commission for Africa (ECA), the Economic Commission for Europe (ECE), the Economic and Social Commission for Asia and the Pacific (ESCAP), the Economic and Social Commission for Western Asia (ESCWA), the Economic Commission for Latin America and the Caribbean (ECLAC), the United Nations Conference on Trade and Development (UNCTAD), the United Nations Environment Project (UNEP), the United Nations Human Settlements Project (UN-Habitat) and the United Nations Office on Drugs and Crime (UNODC). The DA provides capacity development support to developing countries in their implementation of the 2030 Agenda for Sustainable Development, as well as recommendations and decisions made in intergovernmental processes and relevant governing bodies. The DA-funded projects build on the mandates, individual technical capacities and comparative advantages of the respective implementing entities, while providing those mostly non-resident entities with the ability to operationalize their knowledge and know-how to deliver capacity development support at regional, sub-regional and country levels. The Under-Secretary-General (USG) for Economic and Social Affairs is designated as the Project Manager of the Development Account with responsibility for overall coordination, programming, monitoring and evaluation, as well as for reporting to the intergovernmental bodies. The Project Manager is supported by the DA Steering Committee, who advises him/her on strategic policy and project-support matters.166 The Project Manager is also supported by the DA Project Management Team (DA-PMT) located within the Capacity Development Programme Management Office (CDPMO) of DESA, which assists with all aspects of the management of the DA, in particular with regard to programming, monitoring, evaluation and reporting. DA-PMT also liaises with the DA Focal Points in the implementing entities, who are most often the head of the entity’s unit responsible for project planning, project management, capacity development or technical cooperation, on all aspects of the management of DA-funded projects. In response to the COVID-19 crisis, the Development Account has funded five short-term joint projects to help developing countries alleviate the socio-economic impact of the pandemic, including the project on Global initiative towards post-COVID-19 resurgence of the MSME sector (2023W).

A2. ABOUT THE PROJECT The COVID-19 crisis is plunging the global economy into a deep recession and micro, small and medium enterprises – which play a major role in emerging economies – are amongst the hardest hit. Trapped in economic stagnation due to large-scale lockdowns, millions of MSMEs have become the most vulnerable to COVID-19 within the private sector. Compared with large firms, small businesses have fewer resources and lower capacities to cope with the abrupt economic shocks economies are currently facing. With more than two-thirds of the global population employed by MSMEs, the unprecedented outbreak of the pandemic has vividly shown how tightly their activities are woven into the economic and social fabric of the world, as well as their critical role in social and economic resurgence. The objective of the project is to develop and implement capacity-building tools for governments and micro, small and medium enterprises (MSMEs) to facilitate the resurgence and strengthen the resilience of MSMEs in developing countries and economies in transition. These capacity-building tools will seek to mitigate the

economic and social impact of the global COVID-19 crisis and to facilitate the contribution of MSMEs to the SDGs implementation. The project was designed based on the request for assistance for MSMEs from more than 50 Member States, including countries in Latin America and the Caribbean, Asia and the Pacific, Africa, Europe and the Arab regions, as well as intergovernmental demands and resolutions on COVID-19. The project is jointly implemented by UNCTAD, DESA, ECA, ECE, ECLAC, ESCAP and ESCWA, and builds upon the comparative advantages of the participating agencies to provide immediate advice, capacity-building and support to governments and MSMEs during the ongoing global pandemic. The project is structured in five clusters that address the most critical areas of the MSME recovery. Broadly, the roles and lead entities for each of the clusters and workstreams are as presented in the following table:

Table 1. Project clusters and leads Project cluster/workstream Lead agency

Overall coordination UNCTAD

Project cluster/workstream 1 Entrepreneurship and business skills promotion

UNCTAD

Project cluster/workstream 2 Business facilitation/formalization

UNCTAD/DESA

Project cluster/workstream 3 Access to finance/financial literacy

ESCAP

Project cluster/workstream 4 Access to technology and innovation

UNECA

Project cluster/workstream 5 Access to markets

UNECE

The beneficiary countries cover different geographical regions, as shown in Annex 2. The expected outcomes, indicators of achievement, and outputs are presented in the project results framework (Annex 3). The project was developed and implemented under three phases. A new set of outputs was designed or added at each of the three phases of the project. Under the three-phase approach, the project budget was approved by phase. In 2021, when the phase 3 budget was discussed, the Development Account faced a funding gap. To bridge the gap, in November of the same year, the five joint projects were requested to reduce their proposed phase 3 budget by 1 million USD, which led to the curtailment of certain planned activities. For this project, the budget was reduced by $310,000. Overall, a total of $4,490,500 was allocated under this project. Concretely, UNCTAD received $2,671,000, ECLAC received $134,000, ESCAP received $240,000, ECA received $467,000, ECE received $448,500, ESCWA received $370,000 and DESA received $160,000. The project started its implementation in May 2020 and was scheduled to conclude on 31 March 2022, but received approval in February 2022 for an extension until 30 June 2022.

B. EVALUATION OBJECTIVES AND SCOPE B1. EVALUATION PURPOSE, OBJECTIVES AND SCOPE

The present evaluation will constitute a terminal evaluation of the Project. Terminal evaluations are mandatory for all DA-funded projects with a value above $1 million. The evaluation will be largely guided by the UN Development Account Project Evaluation Guidelines, issued in October 2019 and the evaluation policies of the implementing entities, in particular, UNCTAD, which leads the evaluation. The main purpose of the evaluation will be to support accountability for results, and to enable learning. This terminal evaluation of the project has the following specific objectives:

• Assess the results and establish the link between achievements and activities of the intervention;

• Assess the response delivery and external coordination167, including the extent of gender, human rights and disability mainstreaming; and • Identify good practices and lessons learned from the project that could feed into and enhance the implementation of related interventions.

The primary intended users of the assessment are the management of the implementing entities. The evaluation will also provide accountability to project beneficiaries and member States. Furthermore, the evaluation will form a key input to the programme-level evaluation of the DA’s response to COVID-19 to be initiated by the CDPMO/DESA. The programme-level evaluation will entail: a synthesis of the terminal evaluations of five COVID-19 joint DA projects, including this project; a review of relevant 10th and 11th tranche DA projects; and a programme-level assessment. The primary audiences of the programme-level evaluation will include the DA Steering Committee, the DA-Programme Management Team (DA-PMT), and the management of the implementing entities. The results of the programme-level evaluation will also be presented to the General Assembly, through the biennial progress report on the implementation of the DA. The evaluation will cover the duration of the project from May 2020 to 30 June 2022, covering all phases, clusters and activities.

B2. EVALUATION CRITERIA AND QUESTIONS The evaluation will assess the Project’s performance against the main criteria of relevance, efficiency, effectiveness, coherence, sustainability, gender, human rights and disability. In particular, the evaluation is expected to address a number of questions under the following criteria168:

Table 2: Evaluation criteria and tentative questions Relevance 1. To what extent was the project designed to target the new

needs and priorities of participating countries as a result of COVID- 19?

Relevance 2. To what extent was the project aligned with the COVID-19 socio-economic responses of the participating countries (e.g. COVID-19 Socio-Economic Response Plan)?

Efficiency 3. How well coordinated was the response among the entities implementing the joint project?

Efficiency 4. How did the three-phase budgeting and programming approaches impact the efficient delivery of the project?

Effectiveness 5. To what extent did the programme (Development Account) and project governance and management structures and processes enabled, or hindered, the effective implementation of the joint project and the achievement of its results?

Effectiveness 6. To what extent has the project contributed to the expected outcomes as enunciated in the project document?

Effectiveness 7. How did the response contribute to the participating country Governments’ responses to COVID-19, especially in the area of MSME resurgence?

Effectiveness 8. What innovative approach or tool, if any, did the response use, and what were the outcomes and lessons learned from its application?

Sustainability 9. What measures were adopted to ensure that the outcomes of the response would continue after the project ended?

Coherence 10. To what extent was the project complementary to, and coordinated with, other work undertaken by the implementing entities?

Coherence 11. To what extent has the project been coordinated with, and complementary to, the response of other UN entities (Secretariat and non-Secretariat) to COVID-19 in delivering socio-economic support to Member States?

Gender, human rights and disability

12. To what extent were gender, human rights and disability perspectives integrated into the design and implementation of the project? What results can be identified from these actions?

C. EVALUATION APPROACH AND METHODOLOGY The evaluation will be a transparent and participatory process involving the Project’s implementing entities and key stakeholders. It will be conducted based on gender and human rights principles and adhere to the United Nations Evaluation Group (UNEG) Norms and Standards for Evaluation. The evaluation will apply a mixed-method design, including a combination of quantitative and qualitative analysis to inform findings.It is anticipated that travel of the evaluation team may take place in support of elaborating case study/ies, as well as to meet key project stakeholders in Geneva (UNCTAD and ECE). The selection of potential case study/ies and travel requirements will be developed as part of the inception report. Following a preliminary documentation review and a limited number of inception meetings with the core project team, the Evaluation Team will develop an inception report for the evaluation, which will include the finalized overall scope and focus of the evaluation, evaluation questions and methodology, including information on data sources and collection, sampling, key indicators, stakeholder mapping, selection of case study/ies, survey design, and the evaluation timeline. The tentative methodology for the evaluation is presented in Table 4.

Table 3: Tentative methodology for the assessment

a. A desk review of Project documents, including documents/data related to: o Project-level planning, implementation and results achievement, including but not limited to:

▪ Concept note, Phase 2 project proposal, and Phase 3 budget and outputs ▪ Progress report for Phases 1 and 2 (both financial and substantive/narrative report) ▪ Final report (both financial and substantive/narrative report) ▪ Meeting minutes, including the minutes of the bi-weekly/monthly DA network meetings ▪ Monitoring reports ▪ Information on non-DA resources, financial and in-kind, brought in by the participating entities ▪ Information on resources, financial and in-kind, contributed by partners/donors (including information requested under the “supplementary funding” section in the progress reports, which is often incomplete) ▪ Beneficiary/user feedback collected, including, but not limited to, workshop survey results, user feedback on publications, advisory services, guidelines, methodology documents, etc. ▪ Requests for assistance/services received ▪ List of activities completed and details about each activity, including but not limited to:

▪ Agenda, participant lists (name, title, division/unit, organization, country, gender, email address), report and any outcomes document, for each workshop/meeting ▪ Description of each advisory service, beneficiaries (including contact details of the contact persons) and any outputs/deliverables produced ▪ List and description of tool(s), research papers, policy briefs, studies published and information on how each product was disseminated and/or used, list of recipients/users of the product (e.g., dissemination lists)

▪ Documentation related to broader projects or sub-projects of the participating entities of which the Project or its component(s) has constituted an integral part or which are linked to and/or build upon/succeed the work undertaken as part of the Project ▪ Documents and literature related to the Project context ▪ Relevant web and social media metrics related to the outputs of the project;

o Project strategic documents, including but not limited to: ▪ General Assembly's Resolution on Global Solidarity to fight the coronavirus disease 2019 (COVID-19) (A/RES/74/270);

▪ Secretary General's report on "Shared responsibility, global solidarity: Responding to the socio-economic impacts of COVID-19"; ▪ UN Framework for the immediate socio-economic response to COVID-19; ▪ 2021 Programme budget and mandate of implementing entities; ▪ COVID-19 Socio-Economic Response Plan of participating countries.

b. Questionnaires/surveys (in appropriate languages in addition to English) to relevant stakeholders in countries participating in a sample of project activities; c. Telephone, online or in-person interviews with key stakeholders, including but not limited to:

o Project Coordination Team and project focal points of implementing entities o DA-PMT o DA focal points in participating entities o Sample of UN Resident Coordinators/Country Teams, as appropriate o Sample of key global partners o Sample of country-level stakeholders (mainly stakeholders from key beneficiary countries)

d. Case Study/ies, which may include for example a detailed examination of a particular intervention, or of project activities at a regional or national level.

In addition to assessing the mainstreaming of gender, human rights and disability perspectives in the design, implementation and monitoring of the Project (evaluation question 12), the evaluation will integrate these perspectives in the management of the evaluation, data collection and analysis, as well as the development of the evaluation report. Gender balance will be given full consideration in the composition of the Evaluation Reference Group, elaborated in Section D1 (Evaluation management), and the Evaluation Team. Data collected and analyzed in the course of the evaluation will be disaggregated by gender to the extent possible and whenever appropriate, and the evaluation findings, conclusions and recommendations to be presented in the evaluation report will reflect a gender analysis. The evaluation will be carried out according to the UNEG ethical principles and standards.169 The evaluators should demonstrate behavioural independence, impartiality, credibility, honesty, integrity and accountability in conducting the evaluation/assessment to avoid biasing the findings. The evaluators must also address in the design and conduct of the evaluation procedures to safeguard the rights and confidentiality of information providers. The Evaluation Manager will be responsible for ensuring that the Evaluation Team conducts the work assignments without any undue interference from those who were responsible for the implementation of the Project.

D. ORGANIZATION OF THE EVALUATION D1. EVALUATION MANAGEMENT

The independent final project evaluation will be managed/coordinated by UNCTAD’s Independent Evaluation Unit, with the support of an Evaluation Advisory Committee (EAC) that comprises a representative each of the evaluation units of the partner entities (DESA, ECA, ECE, ECLAC, ESCAP, ESCWA) and the Evaluation Officer with the CDPMO/DESA. The EAC primarily serves a quality assurance function and facilitates support to the Evaluation Team as necessary. An Evaluation Reference Group (ERG) consisting of a representative from each UN partner entity (e.g., the DA Focal Point for each entity) and the DA-PMT will review and contribute inputs to key steps in this evaluation such as the TOR and draft final report. Both the EAC and the ERG commit to submitting substantive comments on a timely basis, and comments will be invited on a ‘non-objection’ basis (no response = agree) so that the process is not delayed for an unnecessarily long time. An independent Evaluation Team will be convoked to undertake this assignment. The Evaluation Team (ET) is responsible for conducting the evaluation, applying the methodology as appropriate and for producing the evaluation report. All team members, including the Team Leader, will participate in briefing and debriefing meetings, discussions, and will contribute to the evaluation with written inputs for the draft and final report. A selected number of the evaluation team members will participate in the mission travel(s) if applicable. The ET and the Evaluation Manager will agree on the outline of the report, in consultation with

the EAC early in the evaluation process. The ET will develop its own evaluation tools and framework, within the available timeframe and resources. The team is fully responsible for its report, which may not reflect the views of any of the implementing entities of the project. The evaluation report is subject to quality control by the Evaluation Advisory Committee and clearance by the Evaluation Manager, as set out above. The Team Leader guides and coordinates the team member(s) in their specific work, discusses their findings, conclusions and recommendations and prepares the draft and the final report, consolidating the inputs from the team member(s) with his/her own. The members of the evaluation team should possess a mix of evaluation skills and technical or sectoral/thematic knowledge relevant to the evaluation. In putting together the team, adequate linguistic, geographic and gender representation will also be key considerations. The Evaluation Team will be provided full access to all project reports, documentation, and stakeholder lists and contact information. The Project Coordination Team are required to submit to the evaluation manager project documentation, including data and information residing with the other participating entities, in the last month of the project if possible, if not, immediately following the completion of the project, as well as support the evaluation process, including through facilitating the evaluators’ access to the project’s beneficiaries and other key stakeholders. The roles and responsibilities in the evaluation process are described below:

Evaluation Manager (UNCTAD) will: • Prepare the draft evaluation TOR and revise/finalize based on inputs received • Prepare the TOR for each member of the Evaluation Team (Team Leader, Team Member and Expert(s)) • Recruit and manage the Evaluation Team • Backstop the evaluation process, including supporting the development and administration of surveys, support outreach of the evaluation team to project stakeholders, and access to secondary data listed in Table 3. • Oversee/provide quality assurance to the evaluation and the development of the evaluation report • Facilitate the work of the Evaluation Advisory Committee and the Evaluation Reference Group • Be responsible for clearance of the evaluation report • Support the development of a management response to the evaluation report, including an implementation plan • Organise a virtual workshop on evaluation findings and lessons learned.

Evaluation Advisory Committee comprises a representative each of the evaluation units of the partner entities (ESCWA, ECA, ECE, ECLAC, ESCAP, DESA) and the Evaluation Officer with the CDPMO/DESA. The EAC primarily serves a support and quality assurance function. Specific responsibilities of the EAC include:

• Review and approve the evaluation TOR; • Advise on the selection of the evaluation consultant(s) to ensure that the selection is based on the required skills and qualifications; • Support facilitating access from their respective entities to relevant project documentation and stakeholders; • Review and comment on the inception and evaluation reports; and • Monitor and conduct periodic follow-ups on the implementation of evaluation recommendations addressed to the parties within their entities.

Evaluation Reference Group, consisting of a representative from each UN partner entity (e.g., the DA Focal Point for each entity) and the DA-PMT,170 will review and contribute inputs to key steps in this evaluation such as the TOR and draft final report. The ERG’s key function is to enhance the relevance, credibility and transparency of the evaluation process. Specific responsibilities include:

• To review the draft evaluation ToR and provide substantive feedback; • To facilitate access from their respective entities to relevant project documentation and stakeholders;

• To review the draft evaluation report and provide substantive feedback, including coordinating feedback from other sections, units and offices from headquarters and from the field to ensure quality and completeness; • To participate in the validation meeting of the final evaluation report; • To play a key role in disseminating the findings of the evaluation and implementation of the management response.

Project Coordination Team will: • Facilitate the Evaluation Team’s access to relevant Project documentation and stakeholders, including through:

o Collecting and compiling requested data and information from the participating entities, as requested by the Evaluation Manager o Providing an updated list of stakeholders, and facilitating access to the sample of stakeholders that the Evaluation Team may wish to interview o Facilitating the administration of questionnaires to workshop participants in the participating countries o Ensure the cooperation and contribution of the relevant staff of the implementing entities to the evaluation process, as requested

• Lead the preparation of a response to the recommendations directed to the participating entities, including an implementation plan

DA-PMT will: • Participate in the Evaluation Reference Group • Provide guidance on the allocation of the evaluation budget • Organize a virtual meeting with DA focal points to discuss the key lessons from this evaluation as well as from other COVID-19 joint project evaluations and how to incorporate them in future programming, implementation, monitoring and evaluation of DA-funded projects.

D2. EVALUATION TIME FRAME

The evaluation will be conducted from December 2022 to August 2023. The evaluation process will involve five phases with the tentative timelines as below in Table 5 (the timelines may be adjusted should any exigencies arise):

Table 5: Evaluation phases and tentative timelines Phase Timelines

1. Preparation

August 2022 – December 2022 • Preparation and finalization of evaluation TOR • Establishment of the Evaluation Reference Group • Recruitment of the Evaluation Team • Prepare package of documents required by the Evaluation Team

2. Inception

December 2022 -March 2023 • Preliminary documentation review and preparation of inception report by the Evaluation Team, including development of data collection instruments (questionnaires/ surveys, interview guides) • Brief visit to Geneva (3 days) to meet with UNCTAD Evaluation Unit and key project stakeholders (UNCTAD and ECE) • Draft inception report due: 10 February 2023

• Evaluation Manager review and inception report revision by Evaluation Team: 13 - 22 February 2023 • Reviews by Project Coordination Team, project focal points of implementing entities, and Evaluation Advisory Committee (in parallel): 23 February – 7 March 2023 • Draft final inception report due: 14 March 2023 • Final inception report approved: 20 March 2023

3. Data collection and analysis March- May 2023 • Desk review of remaining Project documents, including requesting additional documentation • Online surveys of stakeholders • Interviews with stakeholders • Data analysis and triangulation

4. Report preparation and reviews May – July 2023 • Data analysis and triangulation • First draft evaluation report due: 16 June 2023 • Evaluation Manager review and report revision by the Evaluation Team: 19 – 28 June 2023 • Reviews by Project Coordination Team, project focal points of implementing entities, Evaluation Advisory Committee and Evaluation Reference Group (in parallel): 29 June – 11 July 2023 • Revised draft evaluation report due: 18 July 2023 • Final evaluation report with annexes: 25 July 2023

5. Dissemination and follow-up August 2023 and onwards • Presentation to the Project Coordination Team, project teams of implementing entities and development and approval of a management response, including an implementation plan for recommendations • Virtual workshop on evaluation findings, lessons learned and follow-up with the DA Focal Points: April 2023

D3. EVALUATION TEAM DELIVERABLES The Evaluation Team will be composed of a team of three consultants (evaluators), namely Team Leader, Team Member and a Gender and Human Rights (HRGE) Expert who also plays the role of Team Member. The two Team Members will report functionally to the Team Leader. The Team Leader will report to the Evaluation Manager. Each of the Evaluation Team has a set of deliverables as described below: Deliverables for Team Leader and Team Member

o Initial review of key Project documents (preliminary document review) o Preparation of an inception report with a finalized evaluation scope and focus, evaluation questions and methodology, including information on data sources, sampling and key indicators, stakeholder mapping/analysis, selection of case study/ies, as well as survey design

o Desk review of remaining Project documents o Data collection and analysis based on the finalized methodology o Preparation of an evidence matrix presenting a summary of evidence collected through each data collection method by evaluation question o Development of a draft evaluation report, based on the template presented in Annex 1, for review by the Evaluation Manager, Project Coordination Team, project focal points of implementing entities, the EAC and the ERG o Revision/finalization of the evaluation report, including all annexes, based on comments received o Preparation of a 3-page summary of the evaluation report and a presentation (PPT) on key findings, conclusions and recommendations o Presentation of evaluation report and discussions with relevant stakeholders such as Project Coordination Team, project teams of implementing entities, DA focal points of participating entities and DA-PMT.

Deliverables for Gender and Human Rights Expert/Team Member

o Initial review of key Project documents (preliminary document review), including identifying gender equality, human rights and disability inclusion dimensions and issues for consideration; o Preparation of an inception report with a finalized evaluation scope and focus, evaluation questions and methodology, including information on data sources, sampling and key indicators, stakeholder mapping/analysis, selection of case study/ies, as well as survey design. Where applicable, gender equality, human rights and disability inclusion considerations will be integrated in the evaluation scope of analysis; evaluation criteria and questions design; methods and tools, and data analysis techniques; o Desk review of remaining Project documents; o Data collection and analysis based on the finalized methodology which would be gender sensitive; o Preparation of an evidence matrix presenting a summary of evidence collected through each data collection method by evaluation question; o Development of a draft evaluation report, based on the template presented in Annex 1 of the Terms of Reference for the evaluation, for review by the Evaluation Manager, Project Coordination Team, project focal points of implementation entities, the EAC and the ERG. The analysis of gender equality, human rights and disability inclusion dimensions and issues should be integrated in the report as an independent section and to the extent possible, these issues should be mainstreamed throughout the report, including in the evaluation findings, conclusions and recommendations. o Revision/finalization of the evaluation report, including all annexes, based on comments received; o Preparation of a 3-page summary of the evaluation report and a presentation (PPT) on key findings, conclusions and recommendations. o Presentation of evaluation report and discussions with relevant stakeholders such as Project Coordination Team, project teams of implementing entities, DA focal points of participating entities and DA-PMT.

E. COMMUNICATIONS AND DISSEMINATION PLAN: The results from the evaluation including key lessons learned, best practices and recommendations will be shared widely with participating entities, partners and stakeholders, and member States. In particular, the following modes of communication could be used:

e. A workshop with all relevant stakeholders to present the key findings, recommendations and lessons learned. The evaluation report will be presented at a workshop attended by the implementing entities, the DA-PMT and other relevant stakeholders for discussion and validation. The implementing

entities will be given the opportunity to present their management response, including an implementation plan for the recommendations; f. A separate virtual meeting will be held with the DA focal points to discuss the key lessons from the evaluation as well as from other COVID-19 joint project evaluations and how to incorporate them in future programming, implementation, monitoring and evaluation of DA-funded projects and projects. g. A copy of the final evaluation report will be published on UNCTAD’s website and the websites of the partner implementing entities, as appropriate; and h. The key findings from the evaluation report will also form a key input to the programme-level evaluation of the DA’s response to COVID-19 to be initiated by the CDPMO/DESA. i. Other communication briefs and products will be produced as appropriate.

1

United Nations Development Account

Terminal Evaluation of Project 2023W

“Global Initiative towards post-Covid-19 resurgence of the MSME sector”

(2020-2023)

Report completed: April 2024

Evaluation conducted by: Patrick Breard, Ariane Agnes Corradi, Jude Kallick

2

Evaluators: Patrick Breard, PhD, Team Leader

Ariane Agnes Corradi, PhD, Team Member Jude Kallick, MA, Team Member

Evaluation Manager: Nishta Keeble, Chief, Independent Evaluation Unit, Office of the Secretary-

General, UNCTAD

Contact: Thomas Callaghan, Associate Programme Management Officer, Independent Evaluation Unit, Office of the Secretary-General, UNCTAD. Tel: +41 22 917 5295 Email: [email protected] This report was commissioned by UNCTAD. The findings, conclusions and recommendations of this report are

those of the external evaluator and do not necessarily reflect the views of the UNCTAD.

3

Table of Contents

Lists of figures, tables, and boxes.............................................................................................................................................. 4

List of Acronyms and Abbreviations ........................................................................................................................................ 5

Executive summary ....................................................................................................................................................................... 6

1. Introduction .................................................................................................................................................................................11

2. Description of the Project........................................................................................................................................................11

2.1 Background ...........................................................................................................................................................................11

2.2 Project objectives and expected accomplishments ................................................................................................ 12

2.3 Project strategies and key activities .............................................................................................................................13

2.4 Beneficiaries and target countries................................................................................................................................. 14

2.5 Key partners and other key stakeholders .................................................................................................................. 16

2.6 Resources ........................................................................................................................................................................... 16

2.7 Link to the Sustainable Development Goals (SDGs) ................................................................................................ 17

2.8 Innovative elements ......................................................................................................................................................... 17

3. Evaluation objectives, scope, and questions .................................................................................................................... 17

3.1 Purpose and objectives .................................................................................................................................................... 17

3.2 Evaluation scope, criteria, and questions .................................................................................................................. 18

4. Methodology ............................................................................................................................................................................. 19

5. Findings ....................................................................................................................................................................................... 21

5.1 Relevance ............................................................................................................................................................................. 21

5.2 Coherence .......................................................................................................................................................................... 24

5.3 Efficiency ............................................................................................................................................................................ 29

5.4 Effectiveness ...................................................................................................................................................................... 32

5.4 Sustainability .................................................................................................................................................................... 59

5.5 Gender, Human Rights, and Leave No One Behind ................................................................................................ 64

6. Conclusions ............................................................................................................................................................................... 73

7. Recommendations .................................................................................................................................................................. 76

Annexes .......................................................................................................................................................................................... 78

Annex 1: Project’s Theory of Change ................................................................................................................................. 79

Annex 2: Evaluation matrix.................................................................................................................................................. 80

Annex 3: Data collection instruments .............................................................................................................................. 86

Annex 4: List of documents reviewed ............................................................................................................................... 97

Annex 5: List of individuals interviewed .......................................................................................................................... 99

Annex 6: Evaluation surveys ............................................................................................................................................. 100

Annex 7: Evaluation Terms of Reference ........................................................................................................................ 134

4

Lists of figures, tables, and boxes List of Figures Figure 1: Overview of project activities 13 Figure 2: Allocation of project funding 16 Figure 3: Perceived relevance of the project according to the survey 22 Figure 4: Project outputs per implementing UN entity and phase 27 Figure 5: Number of project outcomes covered by the implementing UN entities 28 Figure 6: Perceived complementarity of the project with other interventions 29 Figure 7: Surge Project outputs delivered per implementing entity and implementation phase 37 Figure 8: DEPAR number of users 38 Figure 9: Effectiveness results of outcome 1.A 38 Figure 10: Effectiveness results of outcome 1.B 40 Figure 11: Effectiveness results of outcome 3 45 Figure 12: Effectiveness results of outcome 4 47 Figure 13: Effectiveness results of outcome 5 48 Figure 14: Outputs per project outcome and implementing entity 53 Figure 15: Project contributions to formulate and implement enabling policies to MSMEs 54 Figure 16: Contribution of the project to improve the resilience and competitiveness of MSMEs in post-. COVID-19 resurgence in a sustainable way 60 Figure 17: Contribution of the project to better outcomes for women and youth entrepreneurs and other vulnerable groups in a sustainable way 61 Figure 18: Contribution of the project to formulate and implement enabling policies for MSME post-COVID-19 resurgence in a sustainable way 61 Figure 19: Contribution of the project to improve the resilience and competitiveness of MSMEs in post-COVID-19 resurgence in a sustainable way 62 Figure 20: Perceived accessibility of project activities for women, youth, people with disabilities or other vulnerable groups 69 Figure 21: Perceived contribution of the project to better outcomes for women and youth entrepreneurs and other vulnerable and marginalized groups 70 List of Tables Table 1: Project’s beneficiary countries 15 Table 2: Project clusters and lead and participating entities 16 Table 3: Project's implementation phases 17 Table 4: Evaluation questions 18 Table 5: Groups of informants and number of consultations 19 Table 6: Survey recipients and respondents 20 Table 7: Planned and delivered outputs per project phase 37 Table 8: Level of achievement of the Surge project indicators 50 List of Boxes Box 1: ECE’s guidelines and best practices – from a general approach to supporting country-specific responses 55 Box 2: UNCTAD Cross-border Trade and Gender Initiative 68

5

List of Acronyms and Abbreviations

ADB Asian Development Bank

AfDB African Development Bank

DA Development Account

DESA Department of Economic and Social Affairs

EAC Evaluation Advisory Committee

ECA Economic Commission for Africa

ECE Economic Commission for Europe

ECLAC Economic Commission for Latin America and the Caribbean

EMPRETEC Emprendedores (entrepreneurs) and tecnología (technology)

ERG Evaluation Reference Group

ESCAP Economic and Social Commission for Asia and the Pacific

ESCWA Economic and Social Commission for Western Asia

ET Evaluation Team

GDP Gross Domestic Product

GHRP Global Humanitarian Response Plan

HIC High-income Country

HRBA Human Rights-Based Approach

ICT Information and Communication Technology ILO International Labour Organization

IMF International Monetary Fund

LDC Least Developed Country

LIC Low-income Country

LLDC Landlocked Developing Country

LNOB Leaving No One Behind

MIC Middle-income Country

MNE Multinational Enterprise

MPTF Multi-Partner Trust Fund

MSME Micro-, Small and Medium Enterprises

PMT Project Management Team

PRODOC Project Document

RC Resident Coordinator

SDG Sustainable Development Goal

SIDS Small Island Developing States

SPRP Strategic Preparedness and Response Plan

SRO-SA Subregional office for Southern Africa

TCS Technical Cooperation Section

TOC Theory of Change TOR Terms of Reference

UN United Nations

UNCTAD United Nations Conference on Trade and Development

UNDP United Nations Development Programme

UNIDO United Nations Industrial Development Organization

UNOSSC United Nations Office of South-South Cooperation

UNSD United Nations Statistics Division

WHO World Health Organization

WTO World Trade Organization

6

Executive summary Project overview

The project “Global Initiative towards post-Covid-19 resurgence of the MSME sector” aimed at enhancing the resilience of Micro, Small, and Medium Enterprises (MSMEs) in developing countries and economies in transition, especially in the wake of the global COVID-19 crisis. The primary objective of the project was to develop and implement capacity-building tools for both governments and MSMEs. These tools were designed to mitigate the economic and social repercussions of the pandemic, focusing on MSMEs’ contribution to the SDGs. Operationally, the project was structured according to the Entrepreneurship Policy Framework (EPF) of UNCTAD. The EPF aims to support developing country policymakers in the design of initiatives, measures and institutions to promote entrepreneurship. The EPF features 6 key components: 1: Formulating a National Entrepreneurship Strategy; 2: Optimizing the Regulatory Environment; 3: Enhancing Entrepreneurship Education and Skills Development; 4: Facilitating Technology Exchange and Innovation; 5: Improving Access to Finance; and 6: Promoting Awareness and Networking. Building on these components1. The project was operated under five clusters, addressing critical areas of MSME recovery. These clusters included mobilizing entrepreneurial ecosystems, simplifying business registration processes, improving access to finance and financial literacy, increasing access to technology and innovation, and enhancing access to markets. The activities were designed to create an enabling entrepreneurship ecosystem by improving regulatory environments, providing access to innovation, technology, finance, and markets, and reaching out to vulnerable groups such as women and informal workers.

The project commenced in May 2020. The initial completion date was planned for March 2022 but this was extended to June 2022. It underwent three phases involving project startup and approval, initial project delivery and design of scaling up initiatives, and the launch of new activities based on previous results. Beneficiaries of this project included governments, policymakers, MSMEs, business associations, support services, and relevant technical staff in institutions and ministries. The project provided support to 96 beneficiary countries, employing various interventions such as advisory services, training, workshops, seminars, and online platforms. Jointly implemented by UNCTAD, DESA, and the UN regional commissions for Africa (ECA), Europe (ECE), Latin America and the Caribbean (ECLAC), Asia and the Pacific (ESCAP), and the Arab region (ESCWA), the project received financial allocations totalling $4,490,500, with each participating agency implementing its allocated component and managing day-to-day project operations. UNCTAD served as the lead coordinating entity. Key partners in this initiative included UN Development Partners, UN Resident Coordinators, UN Country Teams, UNITAR, UNDP, UNOSSC and ITC. Moreover, the project collaborated with EMPRETEC Centres and other stakeholders such as business incubators and support services. In alignment with the post-2015 development agenda, the project aimed to contribute directly to specific SDGs, particularly SDG 4 (Quality Education), SDG 8 (Decent Work and Economic Growth), and SDG 9 (Industry, Innovation, and Infrastructure).

Evaluation purpose and scope

This evaluation analysed the relevance, coherence, efficiency, effectiveness, and sustainability of the project and its mainstreaming of gender, human rights and disability considerations. The evaluation was carried out between February 2023 and September 2023 following a structured process of data collection and analysis, which included desk review, key informant interviews and online surveys. The evaluation conclusions and recommendations are intended to inform the work of UNCTAD and other UN implementing entities as well as the Development Account Programme Management Team (DA-PMT), which will conduct a synthesis from this evaluation and those of the other four DA Covid-19 projects.

1 According to the Project Document for Phase 2, interventions were selected based on several criteria including to target capacity building areas that are proven to have key impacts on entrepreneurship and MSMEs promotion as based on the formulation and implementation of UNCTAD’s Entrepreneurship Policy Framework (EPF): enabling ecosystem and facilitation of entrepreneurship competencies/motivation, access to finance, technology and markets, networking.

7

Conclusions

Relevance

The project responded to the needs of Governments and MSMEs. Interventions contributed to and were informed by needs and impact assessments. Recipients of project interventions were often involved in or contributed to the design and delivery of project activities and outputs. On various occasions, project outputs were tailored to the needs of target users and beneficiaries, being countries or vulnerable groups. The relevance of the interventions was somewhat mitigated by the mandates and capacities of the implementing UN entities. The latter are equipped for supporting medium- and long- term policy changes rather than immediate crisis responses. Other challenges included the restrictive Covid-19 measures and the short time frame for implementation of the project. Nonetheless, on the whole, the implementing UN entities exercised adaptive management to ensure high relevance of their interventions. The phased approach, in particular, provided the opportunity to introduce new activities not foreseen at an earlier stage.

Coherence

The project was anchored in the mandates and comparative advantages of the implementing UN entities. Selected interventions followed previous work and specific requests from member States and were often coordinated with or complementary to existing activities. At conception, the target of the project was to address specific constraints faced by MSMEs. It was designed around the Entrepreneurship Policy Framework to show the complementarities across interventions that each implementing UN entity would deliver in response to regional priorities and capacities. Some collaborations between implementing UN entities were pursued and complementarities established, but on a limited level. A few synergies in the form of joint outputs were integrated in the 3rd phase of the project. By design, technical task forces created through the project were expected to bridge the implementing UN entities. However, this proved difficult to launch and sustain, owing to several constraints, including a highly ambitious objective given the global crisis and competing priorities within each agency. Nevertheless, the coordination and complementarities established by the project among the implementing entities contributed to expanded dissemination and outreach of outputs, avoided overlaps between them, and allowed significant knowledge exchanges aligned with and supportive of a “One UN” system. On the other hand, collaboration of the implementing UN entities with other UN organisations, including UNCTs, was limited. Despite this shortcoming, at national level, target recipients of project outputs found the interventions complementary to those of other UN agencies.

Efficiency

Project coordination was complex and demanding, involving seven UN entities and the target delivery of around 100 outputs, of which some were composed of many activities. Nevertheless, the coordination of the project was found to be efficient overall given the circumstances and resources available. The coordination of the design phase of the response was actively supported by the DA-PMT. Coordination of project implementation was led by UNCTAD in collaboration with a Steering Committee that met on a regular basis, most often bi-monthly. UNCTAD’s Budget and Project Finance Section (BPFS) produced the project’s financial monitoring table every month, which included the expenditure data of UNCTAD as well as the other implementing entities. A monitoring dashboard was created by UNCTAD to facilitate the tracking of UNCTAD’s project delivery. Data from some of the other implementing UN entities was added, when provided, at two reporting points during the lifetime of the project.

The project would have benefited from the allocation of more resources towards the coordination and monitoring of overall implementation. Across the implementing UN entities, the project was considered complex and hard to follow outside of the interventions under their direct control. Some staff perceived that there were too many participants in the coordination meetings, that the project lacked a Theory of Change, and that it was primarily a compilation of activities rather than a cohesive and mutually reinforcing bundle of interventions supporting in depth any given country. Some staff in the UN Regional Commissions questioned the efficiency cost of a global or inter-regional response versus regional responses. The project was implemented in three phases, which were supportive of adaptive management and perceived by staff as the most rational approach considering the circumstances. However, it was also pointed out that clearer visibility from the onset on the funding available for future phases would have facilitated planning and the search for synergies.

8

Effectiveness

The Development Account programme, as a funding modality, was quick to respond and adapt to the COVID- 19 crisis. The prompt availability of resources and rapid orientation toward a global response contributed to the rapid launch of a wide range of interventions. However, questions remain as to its adequacy for addressing the immediate needs of a crisis in comparison with other UN agencies and mechanisms with a mandate for humanitarian response. The governance of this project, with DA-PMT’s oversight, the management structure with UNCTAD in the lead coordinating role, and each implementing entity leading the operational implementation at the regional level, ensured the global and regional scope of project delivery. At the operational level, implementing entities relied more on partnerships with local and regional institutions rather than coordination within the larger project management structure. The project governance and management mechanisms evolved in tandem with the COVID-19 crisis and the project’s emerging needs. The result was a high delivery rate but with limited inter-agency collaboration in the process. Yet, despite limited collaboration, this independent evaluation identified that the project was effective in producing identifiable results at the outcome level.

Overall, the project delivered 85% of the planned outputs. Phase 3, particularly, delivered the highest number of outputs but had the lowest outcome effectiveness rate, mostly because of budget cuts and some outputs still in progress at the time of this evaluation. There is evidence of the effective improvement of national capacities on formulating and implementing enabling policies on green, resilient and inclusive entrepreneurship and MSME promotion (outcome 1.A) [such as the adoption of a revised entrepreneurship strategy by South Africa, Uganda and Seychelles]; improved resilience and competitiveness of MSMEs (outcome 1.B) [for example with 64.5% of participants opening a new business after attending Empretec-based training]; facilitated MSME registration and formalization through e-platforms (outcome 2) in El Salvador, Benin, Cameroon and Mali; improved access to finance, including upskilling in financial literacy (outcome 3) [for example with 97% of the MSMEs in Latin America that attended trainings reporting improved financial literacy in accounting and reporting and also improved capacity to manage financial resources]; increased MSME access to innovation and technology (outcome 4) [including through capacity development on green technologies for SMEs in Southern Africa]; and increased access to local, regional and international markets through digitalization and non-tariff measures (outcome 5) [for instance with case studies on competition in Thailand, South Africa and Brazil, followed by webinars and the creation of an online course on SMEs and competition policy, primarily for government officials].

In relation to the project indicators, the assessment of their effectiveness was based on limited information, proxies, and expert judgement since they often lacked SMART criteria. It was possible to infer that 85% of them showed some level of progress, either through objective measures (e.g., sales increase) or proxies (e.g., post-training satisfaction assessments and survey responses). The high effectiveness results can be partly attributed to the conceptual framework provided by the EPF, as well as the cluster-based approach with each cluster addressing specific regional constraints and responding to demand from member States. In addition to clustering about 100 outputs from seven implementing entities into 5+1 outcomes, the EPF provided a coherent thread to relate outputs that cut across outcomes. Further analysis would be needed, however, to clearly identify which of those outputs are the most suitable or adaptable for a crisis response.

The project improved the capacity of policymakers in designing and implementing policies supportive of MSMEs, especially in terms of contributing to a country’s government responses, country-specific studies, technical assistance, training courses, and the development of digital tools and regional interventions. This improved capacity is reflected in beneficiaries’ reports of how they have been considering the needs of MSMEs, including those led by women and youth, in their daily work. These results could become more transformative and sustainable with more institutional support to networks of policymakers and communities of practice for mutual and regional learning on supportive policies for MSMEs, as well as by mainstreaming gender and Leaving No One Behind (LNOB) approaches in government officials’ capacity-building.

Sustainability

The project developed approaches, tools and capacities that are being transferred at multiple levels. At the policy level, the addition of more functionalities to e-government tools, expanding training courses to more countries, and the establishment of agreements to implement the project recommendations are all strong evidence of the sustainability of the project’s achievements. At the behavioural level, beneficiaries are applying new knowledge and skills to their daily work, be it by including MSMEs’ issues in policy making or by improving MSMEs’ management, performance, outputs and, by extension, resilience. Challenges to this transfer of

9

knowledge relate to the lack of financial resources, need for follow-up mentoring, and lack of a favourable business environment.

Gender, human rights, and disability

The overall project design was well aligned with the two key UN documents that provide clear direction for the integration of a human rights-based approach, which include addressing the aspect of gender equality and Leaving No One Behind. Although these aspects were well-articulated in the project documents, strong follow-through and documentation of their implementation was limited to a few project components. For those components that did include these aspects, gender was more commonly integrated and youth to a lesser degree. There was very little evidence of the outreach, participation, or impact of project activities for people with disabilities or other vulnerable groups. The UNCTAD component on Women and Cross-Border Trade targeting five countries in Africa was a good practice in how to design and implement this type of intervention with a strong reflection of the needs of some of the most vulnerable women across border areas.

Recommendations Recommendation 1: (i) UNCTAD should further leverage the experience gained through the project to map out how components of the Entrepreneurship Policy Framework relate to and can be best positioned to support MSMEs in their recovery from different types of crises and (ii) DESA and the Regional Commissions should add their analysis of how their work can contribute towards the objective. UNCTAD, DESA and the Regional Commissions could build on the project to map their areas of intervention in support of the MSME sector, as well as capacities and knowledge on the implementation of the EPF components, including in a crisis context, and opportunities for broadening the uptake of EPF components at the regional level based on national needs. UNCTAD, DESA and the Regional Commissions could further consider identifying areas of joint interventions that would trigger complementarities and synergies between the agencies. This could involve collaborating on the development of capacities of all national and sub-national actors, i.e. Governments, MSMEs, other partners (e.g. Chamber of Commerce, Business Incubators, etc.) in line with United Nations Sustainable Development Cooperation Frameworks (SDCF) in respective countries; strengthening or contributing to facilitate access to finance for the MSMEs (including seed money, grants, access to credit, etc.); improving the coordination of MSMEs related policies across ministries; increasing interventions at the local level, such as by supporting NGOs or MSMEs outside of the main cities. The application of a human rights- based approach, gender responsiveness and inclusion of other vulnerable groups (LNOB) should be ensured. Recommendation 2: UNCTAD should continue building on the momentum generated by the project to continue fostering knowledge exchanges and promoting the Entrepreneurship Policy Framework. UNCTAD, DESA and the Regional Commissions should identify means to more meaningfully continue to share good practices and lessons learned on the demand-driven support they provide to the MSME sector, including on areas such as green/circular economy; innovation policies and ecosystems; MSME formalization and strengthening MSME capacities for the empowerment and leadership of women, youth and groups in vulnerable situations. UNCTAD and the Regional Commissions could also propose organizing regional events and/or a global conference to promote entrepreneurship policy and further advance the goals and impact of the Surge project. Furthermore, learnings from the experiences with the Surge about external partnerships should encourage UNCTAD to promote the EPF and entrepreneurship development to other UN agencies also engaged in this area (e.g. ILO, FAO, UN Women, etc.), to development banks, or to related initiatives such as the recent Global Accelerator on Jobs and Social Protection. UNCTAD could also consider joining and supporting events that promote the achievement of SDG 8.3 as an avenue to further promote the EPF. Recommendation 3: The DA-PMT should develop a clear framework for assessing the costs and benefits of implementing a global or inter-regional project versus regional projects. As a global crisis can affect regions and countries differently, global or joint projects should be developed only when there are clear benefits of joint implementation. Thus, a global or inter-regional vs. regional response would not be a priori decision, but a result of a clear assessment. A specific guideline or framework should be developed for this. Some of the assessment criteria could include the need or demand from member States for an integrated response; the range of common versus entity-specific activities and added value of complementary interventions (i.e. in terms of geographical coverage, reach of the target population, capacity, and/or coverage of multiple reinforcing technical areas, or networks and partnerships); capability to execute joint interventions (e.g. in terms of time, resources, logistics, and flexibility to pivot interventions without bureaucratic procedures);

10

coordination costs (which increase with the number of participating UN entities); project inception modalities and governance and knowledge management requirements to facilitate collaboration and knowledge sharing; scaling and sustainability plans; expected results of spreading resources versus concentrating on fewer countries; and so forth. The time taken to conduct such assessments will increase project coherence and effectiveness and facilitate the identification of the financial and human resources needed for project implementation. The network of DA Focal Points is a key existing asset for this assessment. Recommendation 4: Implementing UN entities should ensure that they have a comprehensive Results Framework for the entire project as well as an adequate monitoring plan, with indicators that are designed to support the ongoing monitoring. The results framework should have one objective and ideally have one outcome per cluster/workstream. The objective should state the intended goal of the project, describe the overall achievement targeted by the project, involving a process of change aimed at meeting the needs of identified beneficiaries, and reflect the overall funding available to the project. Each objective should include reference to the project’s beneficiaries and its substantive focus. The objective should not attempt to explain the ways in which the project intends to achieve the objective (i.e. it should not include the word ‘through’ or describe the internal work of the UN using verbs such as ‘support’, ‘facilitate’ or ‘contribute’). The outcomes (OCs) should describe the changes that are expected to occur as a result of the completion of outputs. The OCs should be achievable within the project’s timeframe and budget, and should be specific enough to be measured by the associated indicators of achievement. The indicators of achievement (IAs) should provide measures for monitoring progress towards achieving the OCs and reporting on them after completion of the project. Every indicator needs to provide clearly defined baselines, units of measurement and targets, detailing the quantity, quality and timing of expected results. The monitoring, evaluation, accountability, and learning (MEAL) system should also be designed to capture HRBA, gender and LNOB aspects. In phased interventions or during project execution, any changes in the logframe if/when pivoting activities should be clearly explained to the wider team. Recommendation 5: Implementing UN entities should ensure that sufficient resources are allocated to project coordination, technical collaboration, and partnership building. The absorption capacity of implementing entities can be challenged by crisis response projects which add to the planned programme of work. This is further compounded by projects that come with an extensive UN partnership and a global scope. Sufficient resources should be dedicated to global coordination and to building global partnerships with strategic stakeholders (e.g., UN organisations engaged in supporting the MSME sector; development banks). Capacities should also be directed to supporting technical collaboration and the staff implementing interventions, including towards synergy or liaison with the UNCTs. When designing the project, implementing UN entities should consider featuring coordination and partnership-building in the Theory of Change or logframe of the project. Tools to support continuous connections and knowledge exchange, and to ensure institutional memory should be part of the response package, such as a project website, SharePoint space for all team members, and a Yammer network or Teams channel. The integration of cross-cutting aspects (HRBA, gender responsiveness, LNOB) also requires expertise with sufficient and dedicated time and resources. Guidance could be development to project managers on how to do this.

11

1. Introduction

1. This Evaluation Report presents the independent evaluation of the United Nations Development Account Project 2023W – “Global Initiative towards post-Covid-19 resurgence of the MSME sector” (the MSME Surge project or project). The objective of the project was to develop and implement capacity-building tools for governments and micro, small and medium enterprises (MSMEs) to facilitate their resurgence from the COVID-19 crisis and strengthen the resilience of MSMEs in developing countries and economies in transition. The project was implemented by UNCTAD, UN DESA and the UN regional commissions for Africa (ECA), Europe (ECE), Latin America and the Caribbean (ECLAC), Asia and the Pacific (ESCAP), and the Arab region (ESCWA). Project implementation started in May 2020 and ran until June 2022.

2. The final evaluation of the MSME Surge project aims to assess the results of the project and establish the link between the achievements and activities of the intervention; to assess the response delivery and external coordination of the project, including the extent of gender, human rights and disability mainstreaming; and to identify good practices and lessons learned from the project that could feed into and enhance the implementation of related interventions.

3. The evaluation started in February 2023 and concluded in September 2023. The evaluation conclusions and recommendations are intended to inform the work of UNCTAD and the implementing UN entities, as well as the UN Development Account Programme Management Team (DA-PMT).

2. Description of the Project

2.1 Background

4. The first diagnosed cases of ‘viral pneumonia’ became public on 31 December 2019. On 30 January 2020, WHO declared the coronavirus outbreak a public health emergency of international concern (i.e., WHO’s highest level of alarm) and soon after it launched a COVID-19 Strategic Response and Preparedness Plan (SPRP) and asked the UN Secretary-General to activate the UN crisis management policy2. COVID-19 was declared a “pandemic” on 11 March 2020. On 19 March 2020, the SG report “Shared Responsibility, Global solidarity: Responding to the socio-economic impacts of COVID-19”3 launched a call for action with a focus on people – women, youth, low-wage workers, small and medium enterprises, the informal sector and on vulnerable groups who were already at risk. The UN published the Global Humanitarian Response Plan (GHRP) for COVID-194 on 25 March 2020, initially calling for USD 2.01 billion. On 2 April 2020, the General Assembly adopted the Resolution on Global Solidarity to fight the coronavirus disease 2019 (COVID-19)5. On 5 April the COVID-19 Response and Recovery Multi-Partner Trust Fund (MPTF) was established to channel funds for a system-wide immediate socio-economic response to COVID-19. The UN Framework for the immediate socio-economic response to COVID-196 further set out the structure for the United Nations’ urgent socio-economic support to countries and societies in the face of COVID-19. The Framework presented five streams of work connected by a strong environmental sustainability and gender equality imperative to build back better. One pillar of the Framework focused on protecting jobs, supporting small and medium-sized enterprises, and informal sector workers through economic response and recovery programmes.

5. From its onset, the advent of COVID-19 confronted governments, the development and the humanitarian community and society with a deteriorating socio-economic context. COVID-19 affected poverty, trade and employment levels, and put at risk the achievement of the SDGs. The World Bank estimated the COVID-19 pandemic to have pushed an additional 119 million to 124 million people into extreme poverty

2 The UN Crisis Management Team brings together 23 UN entities to coordinate a comprehensive whole-of-UN response. https://www.who.int/emergencies/diseases/novel-coronavirus-2019/interactive-timeline/#event-47. 3 https://www.un.org/sites/un2.un.org/files/sg_report_socio-economic_impact_of_covid19.pdf 4 UN OCHA. 2020. Global Humanitarian Response Plan for COVID-19. Geneva. 5 United Nations. 2020. Resolution adopted by the General Assembly on 2 April 2020. A/RES/74/270. General Assembly. New York. Link: https://undocs.org/en/A/RES/74/270. 6 UN. 2020. A UN framework for the immediate socio-economic response to COVID-19. April 2020. New York.

12

in 20207. The IMF estimated a 3.5 percent contraction in global GDP in 20208. UNCTAD reported on the use of trade policy instruments with trade-restrictive effects in over 180 countries, most of them in developing countries9. ILO estimated that 8.8 per cent of global working hours were lost in 2020 relative to the fourth quarter of 2019, equivalent to 255 million full-time jobs10. At the beginning of the pandemic, most of these job losses affected women, and, by August 2020, the participation of women in the labour force was over 15 per cent lower than the baseline – male participation was about 10 per cent lower11. Annual unmet SDG financing needs before COVID-19 were USD 2.5 trillion. The pandemic created additional needs in 2020 for USD 1 trillion in COVID-19 spending while the drop in external private resources for developing countries was USD 700 million12.

6. The COVID-19 crisis plunged the global economy into a deep recession and micro, small and medium enterprises were amongst the hardest hit, with decreased liquidity or cash flow availability, decreased demand for products and services, and decreased supply of inputs13. According to UNCTAD’s analysis of the World Bank data in selected countries, on average 13% of small-size firms were temporarily closed due to COVID-19 pandemic, almost 75% of SMEs experienced a decrease in demand for their products and services, over two-thirds of SMEs experienced decrease in supply of their inputs14. A survey implemented by UNECA and IEC Ltd of African MSMEs at the start of Covid-19 found that four-fifths of the survey respondents indicated being significantly affected by the current COVID-19 crisis (rating the effect as highly severe or severe). MSMEs make up 90 per cent of the economic engine of developing countries and are a major source of employment and self-employment, including for vulnerable groups, such as women, elderly and youth. Trapped in economic stagnation due to large-scale lockdowns, millions of MSMEs became the most vulnerable to COVID-19 within the private sector. At the sectoral level, demand and supply stagnation was seen in many industries. MSMEs are major players in non-essential services,15 which became the most affected sectors by lockdown and other isolation measures. In addition, many micro and small businesses are in the informal economy, which was hit first and hardest by the pandemic.

2.2 Project objectives and expected accomplishments

7. The main objective of the project was to develop and implement capacity-building tools for governments and MSMEs to facilitate the resurgence and strengthen the resilience of MSMEs in developing countries and economies in transition. These capacity-building tools sought to mitigate the economic and social impact of the global COVID-19 crisis and to facilitate the contribution of MSMEs to SDG implementation.

8. The initiative brought together UNCTAD, UN DESA and the UN regional commissions for Africa (ECA), Europe (ECE), Latin America and the Caribbean (ECLAC), Asia and the Pacific (ESCAP), and the Arab region (ESCWA) with funding provided by the UN Development Account (DA). The project expected to ensure both global reach and regional presence, international cooperation, and exchange of knowledge and good practices from all over the world. The project also aimed to establish a coherent approach towards MSME resurgence reflecting the holistic and comprehensive nature of efforts needed to ensure and facilitate MSMEs’ green, resilient and inclusive recovery.

7 https://blogs.worldbank.org/opendata/updated-estimates-impact-covid-19-global-poverty-looking-back-2020- and-outlook-2021 8 IMF. 2021. World Economic Outlook Update. January 2021. Washington. 9 UNCTAD. 2021. Impact of the COVID-19 pandemic on trade and development: Recovering, but unevenly - Situation as at 31 March 2021. https://unctad.org/programme/covid-19-response/impact-on-trade-and-development- 2021#aTradePolicy 10 ILO. 2021. ILO Monitor: COVID-19 and the world of work. Seventh edition. 25 January 2021. Geneva. 11 UNCTAD. 2021. Impact of the COVID-19 pandemic on trade and development: Recovering, but unevenly - Situation as at 31 March 2021. https://unctad.org/programme/covid-19-response/impact-on-trade-and-development- 2021#aTradePolicy 12 OECD. 2020. Global Outlook on Financing for Sustainable Development 2021. Paris. 13 UNCTAD. 2021. Impact of the COVID-19 pandemic on trade and development: Recovering, but unevenly - Situation as at 31 March 2021. https://unctad.org/programme/covid-19-response/impact-on-trade-and-development- 2021#aTradePolicy 14 UNCTAD. 2021. Global Initiative towards post-Covid-19 resurgence of the MSME sector. MSME surge project - Addendum Phase 3 Project Proposal. Geneva. 15 The exact definition of these services varies by country, but they are generally recreational businesses, such as accommodation, catering, entertainment and tourism.

13

2.3 Project strategies and key activities

9. The project followed the conceptual approach of clustering interventions under the overall umbrella of the Entrepreneurship Policy Framework (EPF)16 of UNCTAD. The five clusters addressed the most critical areas of MSME recovery:

1- Mobilize entrepreneurial ecosystem and strengthen business skills 2- Simplify business registration and facilitate formalization 3- Improve access to finance/financial literacy 4- Increase access to technology and innovation 5- Enhance access to markets

10. Activities were to be delivered through coordinated and complementary efforts of participating agencies covering key interrelated components of an enabling entrepreneurship ecosystem, such as improving a regulatory entrepreneurship environment and mobilizing entrepreneurship potential, enhancing MSMEs’ access to innovation and technology, finance and markets, including at a policy level and at MSME level, and reaching out to most affected target groups, including women and informal workers. The project also included a number of activities to facilitate the green and sustainable recovery of MSMEs.

11. The project started its implementation in May 2020 and was scheduled to conclude on 31 March 2022, but received approval in February 2022 for an extension until 30 June 2022. The project was developed and implemented under three phases:

• Phase 1 (May-June 2020): The start-up of the project was approved by the DA Steering Committee based on a concept note presenting an overview of the project intended to be implemented over an 18-month period (to December 2021), the intended outcomes for Phases 1 and 2, and the detailed budget for phase 1 only.

• Phase 2 (July-December 2020): The Project Document submitted for approval of Phase 2 was expected to cover Phases 2 and 3, with the expectation that it would be revised in September 2020 to include the finalized Phase 3 activities and budget, and the project’s monitoring and evaluation plan, for Phase 3 approval.

• Phase 3 (initially January-December 2021, extended twice, first to March 2022 and then to June 2022): The addendum to the Project Document focused interventions on scaling up the implementation of the online outputs developed in Phases 1 and 2 and launching new activities in selected countries based on demand and results in the previous phases.

12. Over its 3 phases the project implemented more than 290 activities (Figure 1). Activities were sometimes multi-faceted, for example with reports being used to feed training materials, conferences, or webinars. Activities involved varying levels of effort, from one-time webinars to reports or online platforms requiring several months of development.

Figure 1: Overview of project activities (indicative17).

16 Entrepreneurship Policy Framework and Implementation Guidance | UNCTAD 17 Some activities were described by the UN entities with interchangeable terms, such as webinars and trainings, while some other activities were not necessarily described, such as disseminating a report through a workshop. The evaluation reconstructed the number of activities based on a review and interpretation of the project logframe.

14

Source: Independent evaluation, 2023.

13. The Project Document did not formulate a Theory of Change (TOC) but presented a results framework that the evaluation used jointly with inputs provided by informants to reconstruct a simplified (draft) TOC at evaluation (Annex 1).

14. According to the Project Document (Phase 2), the monitoring arrangements planned for the project included:

● Bi-monthly online meetings of the Project Steering Committee: Representatives from all seven entities were sought to monitor project delivery to ensure smooth and steady implementation. Representatives were also expected to meet on an as-need basis for important decisions that affected all seven entities.

● Inputs and feedback from the Project Design Team: Representatives from UNCTAD, DESA, ECA, ECE, ECLAC, ESCAP and ESCWA, who participated in the project design and its adjustments as needed, in accordance with the COVID-19 situation, were considered to be a key part of the project to evaluate and act upon stakeholder commentary.

● Task Forces: Task Forces comprised of experts from the partner entities were to be established to provide ongoing coordination of work of the five clusters.

● Annual progress reports: Annual or phase-based progress reports were to be provided to the DA- PMT on all aspects of project implementation.

● Success stories: based on request and template provided by the DA-PMT.

2.4 Beneficiaries and target countries

15. The main direct beneficiaries targeted by the project included: Governments, policy makers, Micro-, small and medium enterprises (MSMEs), Business or industry associations, Business support services, Commercial Registry Offices, Technical/experts staff in relevant institutions and ministries, and Multinational Enterprises (MNE).

7

101

29

90

40

17

8

2

0 20 40 60 80 100 120

Online systems and platforms (e-Registration systems, knowldge hubs, websites, search engine, etc.)

Trainings (Workshops, online trainings, online toolboxes, etc.)

Webinars (Global, regional, national; series or unique events)

Reports (Books, Assessments, Policy Reports, Guidelines, Case Studies, Maps, etc.)

Online training tools (e-Learning programmes, course materials, self-training, etc.)

Conferences (Meetings, roundtables, inter-governmental discussions, online events, policy dialogues, etc.)

Technical assistance (Entrepreneurship strategies, national policies, etc.)

Networks (Community of Practice, Policy Network)

The project conducted more than 290 activities

15

16. The beneficiary countries covered different geographical regions. The project compiled a list of 96 beneficiary countries in the final report, as shown in Table 1. These countries were supported by the Surge project, to varying degrees, either through physical/in-person activities or through online modalities. Some of the forms of interventions (in-person or online) included advisory services, trainings, workshops, seminars, conferences and events, platforms and systems. In many cases, implementing entities also provided direct technical assistance to countries through policy advice or in the form of assessment reports, research studies, or tailored guidelines18. Project partners organized online seminars targeting specific countries and also regional or global audiences in many cases.

Table 1: Project’s beneficiary countries.

Africa

1. Algeria 2. Angola 3. Benin 4. Botswana

5. Cameroon 6. Egypt 7. Eswatini 8. Ethiopia

9. Ghana 10. Kenya 11. Lesotho 12. Malawi

13. Mauritius 14. Mozambique 15. Namibia 16. Nigeria

17. Seychelles 18. South Africa 19. Tanzania 20. The Gambia

21. Uganda 22. Zambia 23. Zimbabwe

Americas

24. Argentina 25. Bahamas 26. Barbados 27. Belize

28. Bolivia 29. Brazil 30. Chile 31. Colombia

32. Costa Rica 33. Cuba 34. Dominica 35. Dominican Republic

36. Ecuador 37. El Salvador 38. Grenada 39. Guatemala

40. Guyana 41. Jamaica 42. Mexico 43. Nicaragua

44. Panama 45. Paraguay 46. Peru 47. Saint Kitts and Nevis

48. Saint Lucia 49. Trinidad and Tobago 50. Uruguay 51. Venezuela

Asia

52. Afghanistan 53. Armenia 54. Azerbaijan 55. Bangladesh

56. Bhutan 57. Brunei Darussalam 58. Cambodia 59. China

60. Georgia 61. India 62. Indonesia 63. Islamic Republic of Iran

64. Jordan 65. Kazakhstan 66. Kyrgyzstan 67. Lebanon

68. Lao PDR 69. Maldives 70. Malaysia 71. Mongolia

72. Myanmar 73. Nepal 74. Pakistan 75. Palestine

76. Philippines 77. Saudi Arabia 78. Singapore 79. Sri Lanka

80. Tajikistan 81. Thailand 82. Türkiye 83. United Arab Emirates

84. Viet Nam

Europe

85. Albania 86. Belarus 87. Bosnia and Herzegovina 88. Montenegro

89. North Macedonia 90. Republic of Moldova 91. Romania 92. Russia

93. Serbia

Oceania

94. Fiji 95. Papua New Guinea 96. Tuvalu

18 Some of the countries that received direct technical assistance include for example Armenia, Bangladesh, Benin, Bosnia and Herzegovina, Botswana, Brazil, Cameroon, El Salvador, Georgia, Kazakhstan, Kenya, Kyrgyzstan, Malawi, Mali, North Macedonia, Republic of Moldova, Serbia, Tajikistan, Tanzania, Thailand, The Gambia, Ukraine, Zambia.

16

2.5 Key partners and other key stakeholders

17. Jointly implemented by UNCTAD, DESA, ECA, ECE, ECLAC, ESCAP and ESCWA, the project built upon the comparative advantages and mandates of the participating agencies to provide immediate advice, capacity-building and support to governments and MSMEs during the ongoing global pandemic. Broadly, the roles and lead entities, based on the mandates and expertise of the participating agencies, for each of the five project clusters and workstreams were as presented in table 2.

Table 2: Project clusters and lead and participating entities. Source: Project document Phase 2.

Project cluster/workstream UNCTAD DESA ECA ECE ECLAC ESCAP ESCWA

Overall coordination19 O Project cluster/workstream 1 Entrepreneurship and business skills promotion

O O O O

Project cluster/workstream 2 Business facilitation/formalization

O O O

Project cluster/workstream 3 Access to finance/financial literacy

O O O

Project cluster/workstream 4 Access to technology and innovation

O

Project cluster/workstream 5 Access to markets

O O O O O O

O: Lead institution O: Participating institution

18. UNCTAD was the lead project coordinating entity20. Day to day management of the project was entrusted to each implementing agency for its own component. Each participating agency managed and implemented its component with funds allocated to it. Within UNCTAD, project oversight was ensured by UNCTAD’s Enterprise Branch within the Division on Investment and Enterprise.

19. Key project partners included the UN Development Partners with the UN Resident Coordinators (UNRCs), UN Country Teams (UNCTs), UNITAR, UNDP, UNOSSC, ITC. The project was also implemented with the EMPRETEC Centres21. Other key stakeholders included technical/expert staff in relevant institutions, ministries, and business incubators and support services.

2.6 Resources

20. Overall, the United Nations Development Account (UNDA) allocated a total of $4,490,500 under this project. Concretely, UNCTAD received $2,671,000, ECLAC received $134,000, ESCAP received $240,000, ECA received $467,000, ECE received $448,500, ESCWA received $370,000 and DESA received $160,000 (Figure 2).

Figure 2: Allocation of project funding.

19 Phase 1 of the project was co-led by UNCTAD and ESCWA. 20 UNCTAD and ESCWA co-led phase 1 of the project. 21 EMPRETEC is a flagship capacity-building programme of UNCTAD for the promotion of entrepreneurship and micro, small and medium-sized enterprises (MSMEs) to facilitate sustainable development and inclusive growth.

17

Source: Final report.

21. The project budget was approved and distributed by phase (Table 3). The budget for phase 1 was $325,000 and the budget for phase 2 was $1,310,000. Budget for phase 3 was reduced in November 2011 from a provisional $3,165,500 to $2,855,500 (see section 5.3).

Table 3: Project's implementation phases

Phases Implementation period Effective budget allocations

I May-June 2020 $325,000

II July-December 2020 $1,310,000 III January 2021-June 2022 $2,855,500

2.7 Link to the Sustainable Development Goals (SDGs)

22. The project aimed to contribute to the post-2015 development agenda by making direct contributions to the sustainable development goals (SDGs), in particular goal 4 on “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all” (target 4.4); goal 8 on “Promote development- oriented policies that support productive activities, decent job creation, entrepreneurship, creativity and innovation, and encourage the formalization and growth of micro-, small- and medium-sized enterprises, including through access to financial services” (target 8.3); and goal 9 on “Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation” (targets 9.3 and 9.5).

2.8 Innovative elements

23. The project demonstrated several innovative elements. At the institutional level, the project relied on increased consultative processes and a phased approach which enabled adaptive management. At the programmatic and technical levels, the project contributed to the introduction of MSMEs into policy discussions in response to the pandemic crisis, and e-government services. At the operational level, the project developed innovative approaches and tools to respond to country needs, such as online delivery of trainings and knowledge sharing events.

3. Evaluation objectives, scope, and questions

3.1 Purpose and objectives

$160,000

$370,000

$448,500

$467,000

$240,000

$134,000

$2,671,000

$0 $2,000,000 $4,000,000

DESA

ESCWA

ECE

ECA

ESCAP

ECLAC

UNCTAD

3.56%

8.24%

9.99%

10.40%

5.34%

2.98%

59.48%

DESA

ESCWA

ECE

ECA

ESCAP

ECLAC

UNCTAD

0% 20% 40% 60% 80%

18

24. As per the TOR (Annex 7), the final evaluation of the MSME Surge project has the following specific objectives:

● Assess the results and establish the link between achievements and activities of the intervention;

● Assess the response delivery and external coordination22, including the extent of gender, human rights and disability mainstreaming; and

● Identify good practices and lessons learned from the project that could feed into and enhance the implementation of related interventions.

3.2 Evaluation scope, criteria, and questions

25. The evaluation covers the duration of the project from April 2020 to June 2022. The evaluation focuses on the countries covered by the project. The evaluation analysed unintended positive (or negative) outcomes beyond the 96 countries identified by the project as beneficiary countries. The evaluation conclusions and recommendations are intended to inform the work of UNCTAD and other UN implementing entities as well as the Development Account Programme Management Team (DA-PMT), which will conduct a synthesis from this evaluation and those of the other four Covid-19 surge projects. The evaluation took a retrospective and forward-looking approach. It was summative but also formative aiming to support UNCTAD and UN partners in their next steps.

26. The evaluation addressed the questions in table 4. The evaluation questions were unpacked into an evaluation matrix with suggested measures (Annex 2) and guided data collection methods (Annex 3).

Table 4: Evaluation questions.

CRITERIA KEY QUESTIONS

RELEVANCE

1. To what extent was the project designed to target the new needs and priorities of participating countries as a result of COVID-19?

2. To what extent was the project aligned with the COVID-19 socio-economic responses of the participating countries (e.g., COVID-19 Socio-Economic Response Plan)?

COHERENCE

3. To what extent was the project complementary to, and coordinated with, other work undertaken by the implementing entities?

4. To what extent has the project been coordinated with, and complementary to, the response of other UN entities (Secretariat and non-Secretariat) to COVID-19 in delivering socio-economic support to Member States?

EFFICIENCY

5. How well coordinated was the process for the response among the entities implementing the joint project?

6. How did the three-phase budgeting and programming approaches impact the efficient delivery of the project?

EFFECTIVENESS

7. To what extent did the programme (Development Account) and project governance and management structures and processes enable, or hinder, the effective implementation of the joint project and the achievement of its results?

8. To what extent has the project contributed to the expected outcomes as enunciated in the project document?

9. How did the response contribute to the participating country Governments’ responses to COVID-19, especially in the area of MSME resurgence?

10. What innovative approaches or tools, if any, did the response use, and what were the outcomes and lessons learned from their application?

22 The OIOS COVID-19 response evaluation protocol identifies the following three cross-cutting focus areas: 1) response delivery; 2) external coordination (or “Delivering as one”); and 3) business continuity. “Response delivery” is further defined as consisting of delivery of: 1) the existing mandate needed to implement previously mandated activities in the new environment created by the pandemic; and 2) the COVID-19 specific response (health and non-health) needed to address the pandemic specifically. See OIOS (October 2020), “COVID-19 Response Evaluation Protocol”, para 3-4.

19

SUSTAINABILITY 11. What measures were adopted to ensure that outcomes of the response would

continue after the project ended? GENDER, HUMAN RIGHTS AND DISABILITY

12. To what extent were perspectives on gender equality, protection of human rights and reaching underserved groups integrated into design and implementation of the project? What results can be identified from these actions?

4. Methodology

27. The evaluation collected and analysed data from a range of sources to deepen understanding and triangulate the assessment. The following data collection instruments were used:

• Desk review: Study of secondary resources (Annex 4) as per the project management process and logframe in order to validate achievements, including UN GA resolutions, UN strategies and policies, documents/data related to project-level planning, implementation and results achievement (including post-training assessments, whenever data allowed23), project outputs and monitoring reports, and external reports (Annex 2).

• Interviews and focus groups: Interviews were conducted with a selected number of staff, partners, and stakeholders (Annex 5). Contact details of UN staff and partners were provided by UNCTAD (38 target informants), ECLAC (10), UNECE (10), UNECA (8), DESA (5), ESCAP (5), ESCWA (3). All prospective informants were invited to inform the evaluation, with at least one follow-up reminder message sent in absence of any response. The evaluation interviewed 58 informants. The canvas for semi-structured interviews was tailored to ensure specific relevancy to the selected stakeholders. To optimize time, a few interviews took the form of focus groups. Consultations were conducted virtually with Zoom or Teams. The following consultations per stakeholder group were conducted (Table 5):

Table 5: Groups of informants and number of consultations.

Informant groups Number of informants

Project Coordination Team 10 Project focal points of implementing entities (i.e. members of Task Forces) 12

DA-Project Management Team and DA focal points in participating entities 8

UN and Development Partners (UNCT members, UNDP) 3

Governments (Ministries, national institutions) and policy makers 6

MSMEs, Business or industry associations, Commercial Registry Offices 8

Technical/experts staff in relevant institutions, including EMPRETEC Centres 11

Total 58

• Survey: The evaluation carried out six external surveys (Annex 6) to inform the assessment of the relevance, coherence, and effectiveness of the project and its contribution to outcomes for women and youth entrepreneurs and other vulnerable and marginalized groups. The surveys targeted participants to project activities and beneficiaries. The lists of contacts were provided by UNCTAD, DESA, ESCAP, ECLAC, ECA, and ECE (Table 6)24. The sampling strategy was convenience sampling. In addition, in consultation with UNCTAD Independent Evaluation Unit, the evaluation used purposive sampling to exclude from the lists provided by the UN entities (i) the staff from the same UN entities who were part of those lists as having attended or contributed to the events; (ii) participants to Trainings of Trainers workshops as their role was to train project beneficiaries; (iii) participants to one-time only webinar sessions as having had very limited exposure to the project.

23 Not all post-assessment data could be used by the evaluation team, because some were responded by a very small number of training participants, in relation to the total number of participants in a given training. 24 ESCWA was requested to provide a list of survey recipients but did not share one.

20

The surveys were composed of questions that were common to all six questionnaires and some specific to each of the 5+1 project outcomes25. The questionnaires were made available in English. They were translated in Spanish for outcomes 3 and 5 as target recipients were primarily in the Latin America and Caribbean region. The surveys were anonymous and remained open for 2 weeks, from Thursday 20 July to Friday 4 August. Two reminder messages were sent to increase the response rate. The surveys were launched to a combined list of 1454 persons from at least 120 countries (information on the localisation of survey recipients was not available for all recipients), with 89 messages bouncing back. Altogether, the surveys compiled feedback from 133 respondents. Responses were received from at least 53 countries (some participants did not indicate their location). Responses were received from at least 62 males and 42 females (some participants did not indicate their gender). All survey questions were optional. Questionnaires partially completed were kept in the batch of results when they contained information that was judged credible and meaningful. Statistics were calculated on the basis of the number of valid responses per question and not on the basis of the overall number of respondents to the surveys. The overall response rate to the email surveys is circa 9.7%. The surveys presented the opinion of those who responded but not of the entire list of recipients of the questionnaires nor of all the beneficiaries of the project.

Table 6: Survey recipients and respondents.

Outcome Email addresses Bounced back Valid email Valid responses Percentage

1A 574 38 536 52 9.70% 1B 258 24 234 40 17.09% 2 21 0 21 4 19.05% 3 219 4 215 11 5.12% 4 98 1 97 13 13.40% 5 284 22 262 13 4.96%

Total 1454 89 1365 133 9.74%

• Cross-cutting thematic analysis on human rights, gender and inclusion: As an integral part of the above-mentioned methods of data collection, the evaluation team explored the extent to which gender equality, human rights and leaving no one behind perspectives were considered in the design and implementation of the project. The UN framework for the immediate socio-economic response to COVID-1926 guided the assessment as it outlines how the responses to the pandemic should be consistent with international human rights standards, ensure gender equality and special measures to protect the most vulnerable and marginalized groups, such as women, youth, and migrants.

28. The evaluation used a combination of complementary tools for analysis of the data collected.

● Qualitative analysis of data, secondary resources and interviews/focus groups. The evaluation used ATLAS.ti for coding qualitative inputs.

● Quantitative analysis including systematizing post-training assessments, indexing and cross- tabulations of survey data, and quantitative coding of survey qualitative inputs.

29. The evaluation confronted several constraints or limitations that were mitigated as follows:

● Remote data collection: The evaluation did not include country visits that would allow for face- to-face interviews and direct observations, which may have limited the collection of evidence

25 The project outcome 1 was subdivided into outcome 1A and 1B. 26 A UN framework for the immediate socio-economic response to COVID-19, April 2020 https://unsdg.un.org/sites/default/files/2020-04/UN-framework-for-the-immediate-socio-economic-response-to- COVID-19.pdf

21

at the outcome level. Mitigation: Use of complementary data collection methods to enrich and triangulate findings.

● Limited availability of informants: The interviews (individual and group) are a key instrument for this evaluation. Given turnover and time since the project was completed, some country staff were not available for interviews as they, for example, had retired or left the organization, did not recall exactly what happened, or had too much work or competing demands for their time. Mitigation: Support was requested from the Project Manager for securing interviews.

● Lack of generalisation of survey results: Convenience sampling cannot ascertain that survey recipients were statistically representative of the entire community of project beneficiaries. Furthermore, reminder messages mitigated but did not prevent any non-response bias. Respondents also required to have internet access. The surveys presented the opinion of those who responded but not of all the beneficiaries of the project. Surveys were used as one among other sources of findings.

● Limited evaluability of some indicators: Because some indicators do not comply with the SMART criteria and are elaborated in a broad way, it was difficult to identify the evidence that answers to them. Additionally, they were associated with the work of specific implementing entities, rather than the project as a whole. This posed an additional challenge, especially when the entity associated with a certain indicator was not clear. Mitigation: Targeted consultation about these indicators, corresponding evidence directed at the responsible entity, and an inductive approach to associate results achieved within an outcome to specific indicators.

● Too short a time between end of project and the evaluation to observe impact: Since most of the project’s activities take time to show impact, this evaluation reports on indications of possible future impact on beneficiaries, governments, and partners. Mitigation: Self-reported survey for beneficiaries, and data triangulation generated this evidence.

5. Findings

30. This section presents the evaluation findings per evaluation criteria and question.

5.1 Relevance

To what extent was the project designed to target the new needs and priorities of participating countries as a result of COVID-19?

31. The project was designed to respond to demands of Member States to mitigate the effects of the COVID-19 pandemic. The project was based on ownership by beneficiary countries and relevance towards national and sub-national sustainable development priorities. A demand-driven approach was mainstreamed throughout the project lifecycle. The project was designed based on the request to UN Secretariat entities for assistance for MSMEs from Member States, as well as intergovernmental demands and resolutions on COVID- 19. The Project Document for phase 2 reported requests for support received from more than 70 countries across the 5+1 project outcomes. For example, the Governments of El Salvador, Benin, Cameroon and Mali, requested UNCTAD to create online services that streamlined and simplified business registration. The Governments of Malawi, Tanzania and Zambia, requested UNCTAD to scale-up the format of the cross-border trade trainings piloted in these countries and to replicate them at other borders in the same countries. Georgia and North Macedonia expressed interest for ECE to develop country-specific reports based on the findings of the regional Guidelines and Best Practices for MSMEs in delivering energy efficient products and in providing renewable energy equipment. Several staff from the implementing UN entities also mentioned that in-country interventions were conditioned by the receipt of a Letter of Agreement from Member States. In a few cases, informants further illustrated the demand-based approach of the project by indicating that some initial plans for collaboration were parked sometimes due to changing country needs and priorities.

32. The project conducted or contributed to needs and impact assessments to inform interventions. Across all regions the project collaborated with Member States and national stakeholders to conduct surveys, needs assessments, or impact studies that informed project design and implementation as well as national responses. In South-east and Central Asia, and the Pacific, countries such as Samoa, Bangladesh, Nepal,

22

Cambodia and Kazakhstan participated in ESCAP in-depth country assessments. In the Southern African region, ECA carried out a survey across its 11 member states to assess the impact of Covid-19 and the war in Ukraine on MSMEs in association with national business organizations, The survey gauged policies needed to foster regional integration and technology and innovation in post Covid-19 recovery strategies. ECE produced a rapid impact assessment series on end-to-end supply chain activities to support long-term development efforts for five beneficiary countries and presented them for intergovernmental discussion. ECLAC ran diagnostics on competition policy and consumer protection, and on SME support policies, to identify needs for project activities or products in target countries, including the needs of women and vulnerable groups. In Kenya, DESA and the Kenya Micro and Small Enterprises Authority assessed how MSMEs were affected by the pandemic. The survey results were disseminated to stakeholders, who were brought together to participate in a validation workshop to discuss most needed support systems and formulate policy advice to be shared with the Ministry of Cooperatives and MSME Development. In several cases informants also recalled that needs assessments are part of the project design methodologies implemented by UN entities. For example, the first step of the EPF is to assess the status of the national entrepreneurial ecosystem. According to the evaluation survey, the vast majority of respondents reported that project activities have responded to their priorities and were relevant to their work (Figure 3).

Figure 3: Perceived relevance of the project according to the survey (n=133 respondents).

Source: Independent evaluation survey, 2023.

33. Stakeholders, including policymakers, MSME entrepreneurs and development partners, participated in project design and implementation. On various occasions, evaluation informants stressed that the involvement of Member States and stakeholders in designing and implementing interventions was also a factor contributing to addressing their needs and priorities. In Kenya for example, in addition to ensuring buy-in from the Government before implementing cross-border trade trainings, UNCTAD relied on national institutions to facilitate project delivery on the ground. UNCTAD asked the Government to identify training participants and the Government requested local associations to identify participants and to help with logistics. A representative of the Ministry of Trade of Kenya travelled to the border to introduce the workshops. This was the first instance of high-level government support to the project. Several Empretec centres also mentioned that the identification of workshop participants was done in collaboration with government agencies and departments. In Zimbabwe, the Empretec centre selected primarily businesswomen and youth whose businesses were declining or faltering. Another example involved the creation of training materials on Core Indicators for Sustainability and SDG Reporting (GCI). A series of trainings of trainers (TOT) were organized by UNCTAD and participants’ feedback was used to revise the materials before training entrepreneurs.

34. Project activities and products were tailored to Member States and target beneficiaries. Project activities and products were often tailored to the needs of target recipients. Many regional reports were localized through national studies. For instance, ECE developed guidelines and best practices for MSMEs to

41%

44%

11%

1% 1%

1% 1%

The project activities responded to my priorities and were relevant to my work

Strongly agree

Agree

Somewhat agree

Somewhat disagree

Disagree

Strongly disagree

Do not know

23

assure resiliency and progress towards a circular economy in sustainable resource management and critical raw material supply chain solutions, as well as guidelines and best practices for MSMEs in delivering energy- efficient products and in providing renewable energy equipment27. Four countries of the ECE region were chosen as pilot countries for which the guidelines were customized, before spanning to other countries. In Tanzania, the Empretec centre delivered a training on entrepreneurship skills development that was based on the UNCTAD-EMPRETEC methodology but was adjusted to participants with limited literacy skills. UNECA delivered an online training course on the role of technology and innovation for MSMEs under Phase 2 and feedback from the training was used to develop a second course on role of green technologies in MSME development under Phase 3. For many activities and products, UN entities also ensured the translation of materials in official UN and local languages to facilitate access and outreach at global, regional and national levels. In the early months of the response, ECE partnered with UNDP (which had a Zoom license) to deliver workshops with simultaneous interpretation as this could not be accommodated by ECE online platforms.

35. Implementing UN entities adapted to maximize the relevance of interventions aimed for longer term change. While some Surge project outputs were found to be more crisis-response driven and with a shorter path to outcomes, such as the implementation of e-registration systems, other activities such as the design of a national entrepreneurship strategy were referred to as yielding an impact after a longer period of time. This is typical of DA projects and other socio-economic UN initiatives. Several interviewees were of the opinion that the UN socio-economic pillar did not generally have many crisis response products. Despite the achievements of the project (see section on Effectiveness), the capacity of the UN to respond to the needs and priorities of participating countries as a result of COVID-19 depended on institutional factors that were not always under their direct control. It was further indicated that at the onset of the pandemic, countries needed immediate financial support to help their companies and MSMEs to stay alive. Several survey respondents reported that complementing capacity-building activities for entrepreneurs with grants would have been beneficial28. Although the EPF and cluster-based approach proved an appropriate conceptual framework to articulate the response (see section on Coherence), the latter could have benefited from some prior analysis of the relief-recovery-development pathway to facilitate the identification of appropriate crisis mitigating interventions. Implementing UN entities therefore used adaptive management to ensure adequacy of the services in addressing demand-orientation and longer-term outcomes.

To what extent was the project aligned with the COVID-19 socio-economic responses of the participating countries (e.g., COVID-19 Socio-Economic Response Plan)?

36. The project was aligned with the COVID-19 Socio-Economic Response Plans (SERPs) in many participating countries, but the design occurred before the SERPs were finalized. The project was part of the general strategic plan of the United Nations called “A UN framework for the immediate socio-economic response to COVID-19”29 and specifically contributed to Pillar 3 “Economic Response and Recovery: Protecting Jobs, Small and Medium-Sized Enterprises, and Informal Sector Workers” of this system-wide framework30. This UN framework also served to guide the COVID-19 Socio-Economic Response Plans (SERPs) which were developed by UN Country Teams and Member States and stakeholders. However, while the concept note of the project was released in April 2020 and the Project Document for phase 2 in June 2020, many SERPs were published later, i.e. in or after August 202031. The design of Phase 3 of the project was finalized in early 2021 but most outputs were a continuation of previous work. Several informants also mentioned that the implementing UN entities have limited country presence, reducing the capability to mainstream the project in the SERPs. Despite these limitations, the evaluation reviewed a purposeful sample of 10 SERPs32 showing their systematic coverage of pillar 3 of the UN framework and some frequent references

27 UN Development Account project | ECE 28 See for example: Training and grant support for potential entrepreneurs and start-ups in the Sea of Azov region, Ukraine (ilo.org) 29 https://unsdg.un.org/sites/default/files/2020-04/UN-Framework-for-the-immediate-socio-economic-response-to- COVID-19.pdf 30 The UN framework built upon the UN General Assembly’s resolution on “Global solidarity to fight the coronavirus disease 2019 (COVID-19) (A/RES/74/270)” and on the Secretary-General’s report on “Shared responsibility, global solidarity: Responding to the socio-economic impacts of COVID-19”. 31 UNSDG | Resources Library 32 The evaluation selected 2 SERPs per region and countries where the project implemented interventions, i.e. Argentina, Bangladesh, Bhutan, BiH, Cameroon, Jordan, Moldova, Peru, Thailand, and Zimbabwe.

24

to or links with the outcome areas of the project, with a prevalence of connections with outcomes 4, 3 and 1A. The evaluation also found references sometimes to the project outputs. For example, the Socio-Economic Response and Recovery Plan for the Republic of Moldova (June 2020)33 was informed by the ECE’s survey- based assessment of the COVID-19-induced economic crisis and changes in non-tariff measures to contain the pandemic on MSMEs (May 2020) and by the Assessment on the impact of the crisis on female-owned enterprises (June 2020).

37. Independently of the SERPs, implementing UN entities considered the COVID-19 socio-economic responses of participating countries during project design and implementation. As noted in the earlier section, the project delivered many regional and national needs assessments and impact studies, which were a source for aligning project activities with the COVID-19 socio-economic responses of the participating countries. For example, UNECA organised, at the request of the Government of Mauritius and in association with the SADC Business Council, a regional seminar on the impact of Covid-19 on MSMEs in Southern Africa and the recommendations were intended to support the development of a roadmap for technical assistance at a regional level. In Latin America, ECLAC analysed the policies implemented by governments to face the MSME crisis generated by COVID-19. Experiences in countries having MSME fostering institutions were prioritized. Good practices were systematized and shared among public institutions in charge of MSMEs policies. Virtual meetings were organized to share implementation experiences about measures to help MSMEs in order to enhance the coordination of public policies for economic recovery. In the Gambia and Kenya, DESA ensured that the design and implementation of project activities were aligned with the national agendas to contribute to the relevance of the interventions and strengthen ownership by the countries. In the Gambia, work was carried out with the Ministry of Trade, Industry, Regional Integration and Employment (MOTIE), UNDP and the UNRCO. Project implementation served to strengthen the capacity of policy makers and other stakeholders to implement new policies and to pursue coherence among the already formulated policies in the country. Several evaluation informants indicated the project was relevant for bringing MSMEs to the policy debate where they were often absent.

5.2 Coherence

To what extent was the project complementary to, and coordinated with, other work undertaken by the implementing entities?

38. The project was anchored in the mandates, comparative advantages, and programme budgets of the implementing UN entities, paving the way to complementarities with previous interventions. The initial concept note and Project Document presented a detailed review of the respective mandates and comparative advantages of the implementing UN entities as a rationale for their contribution to the project34. Among these

33 MDA_Socioeconomic-Response-Plan_2020.pdf (un.org) 34 Many factors were put forward to stress the institutional coherence and rationale for the project. Some highlights include: (i) UNCTAD is a focal point in the UN on entrepreneurship and MSME policy as formally mandated by two United Nations General Assembly resolutions on Entrepreneurship for Sustainable Development. UNGA resolutions /RES/71/221 and A/RES/73/225 both call on “the United Nations system, and in particular the United Nations Conference on Trade and Development, to continue to provide support to and assist member States, at their request, to identify, formulate, implement and assess coherent policy measures on entrepreneurship and the promotion of micro-, small and medium-sized enterprises”. UNCTAD also provides technical assistance and capacity-building support to a number of developing economies, including advice and training to policy makers to inform and guide the development and implementation of national entrepreneurship policies based on UNCTAD’s Entrepreneurship Policy Framework (EPF), which was endorsed by member states through a series of intergovernmental discussions and formally launched at the Ministerial Conference UNCTAD XIV in 2012. (ii) DESA brings relevant expertise from implementing the project “Enhancing national capacities for enhancing potentials of MSMEs in achieving SDGs in developing countries (MSME project)” funded by the UN Peace and Development Fund (PDF). (iii) ESCWA has developed and launched an SME information portal for the region with an aim to support entrepreneurs and small businesses access information needed to support their business. (iv) ECE has conducted rapid impact assessment of the COVID-19 pandemic on end-to-end supply chain activities, with a view to supporting national and regional recovery and long-term development efforts. (v) The ECA sub-regional office for Southern Africa leverages its mandate and implementation of a UNDA 13th tranche project for integrating a component to build capacities of MSMEs in Southern Africa to harness technological applications to address the impact of the pandemic. (vi) ECLAC has a long experience in studying the MSME performance in Latin America and in supporting public institutions in developing, implementing, and evaluating MSME fostering policies. (vii) ESCAP responds to ESCAP Resolution 70/5: Strengthening regional cooperation and capacity for enhanced trade and investment in support of sustainable development, and has planned to work with existing initiatives and organizations to support MSMEs in the region.

25

enabling institutional factors, references were primarily made to supporting policy makers on MSME development as well as to building capacities of MSMEs and entrepreneurs. Significant evidence was found showing that the project was consistent with and sometimes a continuation of previous activities. UNCTAD for instance relied on its network of Empretec Centres to deliver activities on the capacity-building of entrepreneurs. In collaboration with centres and experienced Empretec trainers, UNCTAD delivered on-line trainings and developed demand-driven, customized tools to assist the centres in coping with short-term and medium-term pandemic-induced challenges to their business operations. UNCTAD also built on earlier contacts to support the establishment of e-Registration platforms in El Salvador and Benin, which was accelerated by the pandemic. ESCWA grounded its proposed contribution to the 1st phase of the project in previous meetings and surveys. ECLAC delivered courses that were the result of prior studies and a long preparatory phase of getting to know the issues of SMEs in e-commerce. ESCAP’s “Bangladesh Startup Ecosystem Assessment Report”35 delivered by the Trade, Investment and Innovation Division followed earlier work from another division.

39. Interventions were often coordinated with and complementary to other internal activities that were underway in the implementing UN entities. Within each implementing UN entity, internal complementarity was sought with other projects. ECE for example used the impact assessments on the novel COVID-19 pandemic on end-to-end supply chain activities, conducted under UNDA 10 “Strengthening the national capacities of selected ECE countries for evidence-based regulatory and procedural trade policies to achieve SDGs”, to further draw two national assessments in Armenia and Moldova focusing on female-owned enterprises. DESA leveraged its expertise and understanding of country needs gained with the project “Enhancing national capacities for enhancing potentials of MSMEs in achieving SDGs in developing countries (MSME project)”36 funded by the UN Peace and Development Fund (PDF), DESA complemented the PDF Kenya and The Gambia to improve contributions of MSMEs to social and economic resurgence after the pandemic and strengthen their roles as enduring institutions for jobs and entrepreneurship to reach those further behind. ESCAP took advantage of the Asia-Pacific MSME Policymakers Network established with the project to present and launch the 2nd edition of the “Policy Guidebook for (M)SME Development in Asia and the Pacific”37, and its accompanying “Online Resource Centre for MSME Development”. Several knowledge products from ESCAP were also co-financed by the project and other sources. For instance, the publication on MSME Access to Finance: The Role of Digital Payments38 was funded from the Regular Programme of Technical Cooperation for the studies on Cambodia and Nepal, and from the Government of Canada (through Global Affairs Canada, in the context of ESCAP’s Catalysing Women’s Entrepreneurship Programme), for the studies on Bangladesh and Samoa. The UNECA Sub-regional office for Southern Africa leveraged the project to build the foundations of a UNDA 12th Tranche project on innovation and technology for SMEs, which had been approved. ESCWA presented the DEPAR portal39 and MSME toolbox40 at the first Arab SME Summit41, organized in Jordan from 30 October to 1 November 2022 by another ESCWA project.

40. The Surge project increased internal collaborations between departments in the implementing UN entities. Evaluation informants often shared the perspective that the project has contributed to increased communication between internal departments that were used to working separately. Several project outputs were developed on the basis of internal cross-collaborations between teams. In UNCTAD for example, the SME Entrepreneurship Branch and the Empretec centres collaborated with the Trade and Gender team, the Competition Branch, and the Accounting team. Staff from these sections participated in coordination meetings, exchanged tools and new materials, and were consulted to identify beneficiaries. The UNCTAD Business registration team also commended the project for providing an opportunity to have an open channel of communication with the Entrepreneurship section. This contributed to sharing experiences and good practices and for these colleagues to be better positioned to promote e-Registration. In ECE, project activities helped to break down silos and foster collaboration between different teams working in the area of sustainable trade, innovation, circular economy, rational use of natural resources, PPPs and women’s empowerment. In ESCWA, the work on the MSME toolbox involved 7 divisions, reflective of the 7 foci of the toolbox. Each specific team

35 Bangladesh startup ecosystem assessment report | ESCAP (unescap.org) 36 UNPDF | Enhancing national capacities for unleashing full potentials | United Nations 37 Policy guidebook for MSME development in Asia and the Pacific, 2nd edition | ESCAP (unescap.org) 38 ESCAP. 2022. MSME Access to Finance: The Role of Digital Payments, MSME Financing Series No.7. Bangkok. Available at: https://www.unescap.org/kp/2022/msme-financing-series-role-digital-payments. 39 Communities | DEPAR (unescwa.org) 40 MSME Toolbox Dashboard | DEPAR (unescwa.org) 41 Arab SMEs Summit | DEPAR (unescwa.org)

26

helped with research, data collection, and finalising materials. Weekly meetings were organized with the coordinators under each division. The teams used Trello to see what others were doing. In ECLAC, the International Trade and Integration Division and the Production, Productivity and Management Division jointly developed a course promoting digital economy and digital trade. In ECA, the ECA subregional office for Southern Africa (SRO-SA) and the ECA Digital Center of Excellence (DCE) collaborated to jointly produce a study on the “Role of Digitalisation in Strengthening Capacities of Micro, Small and Medium-size Enterprises (MSMEs) in Southern Africa”42. The ECA Subregional Office for Southern Africa collaborated with the ECA Africa Institute for Economic Development and Planning (IDEP) to deliver two training courses on role of technology and innovation for MSMEs. In some cases, informants mentioned that the Surge coordination meetings organized by UNCTAD helped identify other units from the same implementing UN entity involved in the project. Despite such examples, other staff also stressed that the overall emergency context meant there was little time to pursue internal synergies as teams concentrated most often on activities under their direct responsibility and on achieving immediate results.

41. The project enabled some level of technical collaboration and complementarities between implementing UN entities. According to the concept note, the project was conceived to propose a coherent approach towards MSME resurgence reflecting the holistic and comprehensive nature of efforts needed to ensure and facilitate the MSMEs’ green, resilient and inclusive recovery. Activities sought complementary and synergies in impact among participating agencies covering key interrelated components of an enabling entrepreneurship ecosystem. Inter-agency collaborations expanded the dissemination of agency-specific products while coordination and communications helped to avoid overlapping activities. Some joint products were also developed by the implementing UN entities, most often involving UNCTAD as one of the partners (see also next paragraph). UNCTAD and ESCWA, for example, cooperated to inform and build awareness on the e-registration platform for MSMEs. ESCWA leveraged US$ 30,000 from the project to organize a capacity development workshop with UNCTAD to familiarize member states in the Arab region with the e-registration system. This generated interest and ESCWA was later able to mobilise US$ 600,000 from different sources for two e-registration projects in Jordan and Syria43. UNCTAD also produced a global report on market access and competition policy44 to which ESCWA and the other Regional Economic Commissions contributed through advice, inputs, data, and review of the document. In Europe, ECE’s COVID-19 impact assessment targeting MSMEs in Georgia45 spurred discussions with the Ministry of Economy and Sustainable Development leading to the integration of Georgia’s non-tariff measures (NTM) information into UNCTAD’s NTM database46,. In Romania, ECE took up the Empretec methodology and provided grants for the Asociatia pentru Antreprenoriat din Romania (APAR) to conduct two in-person workshops of 9 days each which were attended by 60 MSMEs. In coordination with the Regional Economic Commissions, UNCTAD organized a series of regional capacity-building events to raise the awareness of policy makers on the role of competition-related policies for access to markets in the post-COVID-19 resurgence of MSMEs. UNCTAD, ESCAP and the Trade and Competition Commission of Thailand (TCCT) organized, for example, a conference in June 2022 on the Contribution of Competition Policy to the Resurgence of MSMEs post-COVID-1947, to which DESA and ESCWA also contributed. UNECA and UNCTAD co-organised a Regional Policy Dialogue on “The role of Competition Policy in supporting MSMEs economic recovery in the post COVID-19 crisis” and collaborated on a case study on South Africa for the UNCTAD global report on “How Covid-19 affects MSME access to markets and competition: a review of key issues and recommendations for future action”. Evaluation informants also mentioned that the project provided a unique and successful opportunity to learn about the work of the other implementing UN entities in the area of entrepreneurship and MSME development, therefore contributing to the “One UN” agenda. The project website48, developed to gather outputs and increase their visibility and sustainability, was also commended by many informants. However, most of these collaborations had a

42 The role of digitalisation in strengthening capacities of Micro, Small and Medium-Size Enterprises (MSMEs in Southern Africa to take advantage of the AfCFTA): ECA Sub-Regional Office for Southern Africa (SRO-SA) & ECA Digital Centre of Excellence (uneca.org) 43 ESCWA also mobilized an additional US$300.000 for an e-registration project in Sudan but the political context in the country did not allow for implementation. 44 The covid-19 pandemic impact on micro, small and medium sized enterprises: Market access challenges and competition policy (unctad.org) 45 Impact_COVID-19_Georgia.pdf (unece.org) 46 UNCTAD TRAINS 47 Conference on the Contribution of Competition Policy to the Resurgence of MSMEs post-COVID-19 | UNCTAD 48 Home | UNCTAD

27

bilateral character and the evaluation noted the difficulties for Regional Commissions to collaborate with one another.

42. Several factors limited opportunities for amplifying synergies between interventions and for establishing a stronger global coherence. From a total of 100 project outputs formulated across the three phases of the project logframe, only 6 were identified by the evaluation as joint outputs from 2 implementing UN entities (Figure 4, in yellow), out of which just 4 were effectively delivered (i.e., UNCTAD and ECLAC on outcome 5). Joint outputs were formulated for phase 3 of the project, owing to several factors such as the short duration of phases 1 and 2 and the need for a sufficient amount of time to design joint interventions identified after a national agenda, the number of implementing partners progressively increasing between phases 1 and 2, or the limited cross-institutional awareness of each partner’s expertise on and capacities for supporting MSMEs.

Figure 4: Project outputs per implementing UN entity and phase.

Note: Joint outputs are referred to in yellow. Source: PRODOC.

43. Staff from different Regional Commissions attempted to establish synergies with other Regional Commissions for this project but with limited success. ESCWA for example reached out to ECE and ECA to seek their prior experience in supporting MSMEs with a view to uploading their materials on the DEPAR portal to be used in the region and globally. Despite several meetings and some exchange of materials, there was no concerted effort to populate the DEPAR portal. Several evaluation informants in UNCTAD and the Regional Commissions seconded the perspective of an ECE staff member that “everybody was too busy and there was no time to make the effort of cooperation”. According to a staff member, if “capacity development by Regional Commissions is to be jointly done, this is a long process cycle” not suited to a crisis situation. Evaluation informants also mentioned that the geographical focus and mandates of the Regional Commissions created various challenges to collaboration as the Regional Commissions come with different areas of expertise and niches. Furthermore, needs of Member States also varied across and within regions. In addition, although the EPF was found to be a comprehensive and flexible framework, none of the implementing UN entities outside of UNCTAD had the capacity to cover all pillars. On average, UN entities covered about 2 out of 5+1 project outcomes (Figure 5), i.e., they remained geographically and technically focused. Accordingly, several informants shared the perspective that the project was a compilation of activities and outputs based on a unifying and valid conceptual framework (i.e., the EPF) and cluster-based approach but with limited synergies, moderate coherence, and high overall complexity. According to a staff member at ESCAP, “The project did not install joint indicators and reporting. If there was to be a joint event, it would indicate specifically who would deliver what. It was not about the project producing a pool of papers without referring to one or another agency; products were assigned by entity. From then on, it was clear agencies would have different streams of work.” Some evaluation informants questioned the cost and benefit of a global project in a crisis context compared to several inter-regional projects (see section on Efficiency), despite this project enabling strong knowledge exchange between participants.

Figure 5: Number of project outcomes covered by the implementing UN entities.

28

Source: Final report. To what extent has the project been coordinated with, and complementary to, the response of other UN entities (Secretariat and non-Secretariat) to COVID-19 in delivering socio-economic support to Member States?

44. The project triggered some collaboration with other UN entities at national, regional, and global levels. At global level, UNCTAD partnered with UNITAR to provide the e-learning course “Policymaking: Entrepreneurship for Sustainable Development”49, which was followed by 317 policymakers from over 104 countries. ESCAP also published a series of knowledge products on MSME Financing50 that benefitted from a partnership between ESCAP and the United Nations Capital Development Fund (UNCDF). Complementarities with other UN organisations were also at national level. In several cases, the UN RCOs and UNCTs were referred to as project partners. In the Gambia, the Ministry of Trade, Industry, Regional Integration and Employment together with DSDG/DESA and the UN Country Team organized a high-level policy implementation forum in December 2020 on supporting MSMEs in the post-Covid-19 pandemic era, focusing on promoting MSME formalization51. In Europe and Central Asia, UNDP was often involved substantively and/or logistically in the delivery of online or hybrid workshops and dissemination events. In Armenia for example, ECE, UNDP, and other partners organized in March 2022 a conference on “Unveiling Market Opportunities for Boosting Residential Energy Efficiency and Alleviating Energy Poverty”52. UN Women also contributed to the development of a roadmap in Moldova based on the recommendations outlined in a study on women-led MSMEs, conducted by the project. Country office staff from several UN organisations including FAO, UNDP, OHCHR, and UNHCR also attended some of the online trainings or events delivered by the project. Despite such examples, evaluation informants were often of the opinion that the project had not maximised partnerships with other UN entities owing to several constraints or limitations, including the respective crisis response priorities and challenges in UN development agencies, limited staffing capacities from the project to focus on partnership building at global level, limited funds directed towards creating synergies with other agencies, lack of dedicated outputs and indicators referring to joint work with other agencies, and the lack of country presence of the implementing UN entities (see also §46).

45. The project was implemented with a range of partners that complemented the responses of the UN entities. Besides UN entities, project implementation involved a range of other partners at international, regional and national levels that helped to complement the responses from other UN organizations. Expert speakers in ECE events included, for example, important players in greening the MSMEs agenda, including the International Chamber of Commerce (ICC), World Trade Organization (WTO), Organisation of Economic Co- operation and Development (OECD), European Bank for Reconstruction and Development (EBRD), and a number of national authorities and private sector players. In Southern Africa, the project contributed to strengthening the ECA SRO SA partnership with the SADC Business Council and national chambers of commerce and industry in all 11 Southern African countries. Joint events were organized by ECA and the SADC Business Council at the SADC Industrialisation Week in Malawi and the Dubai Expo 2020. ECA also developed a digital platform on technology and innovation for MSMEs to be hosted by the SADC Business Council. According to the evaluation survey, the majority of respondents were of the opinion that the project complemented interventions implemented by other international organizations working in their

49 E-Learning Course on Policymaking: Entrepreneurship for Sustainable Development | UNCTAD 50 ESCAP’s MSME Financing Series | ESCAP (unescap.org) 51 High Level Policy Forum on MSMEs | United Nations Development Programme (undp.org) 52 From buildings to small companies, UNECE supports countries to reap benefits of improving energy efficiency | UNECE

1 1 1

5

1

2

3

2

1 1

6

1 1

2 2

5

1

0

2

4

6

UNCTAD DESA UNECA ESCAP UNECE ESCWA ECLAC

Phase 1 Phase 2 Phase 3

29

country/region (Figure 6). As illustrated by a private sector representative in North Macedonia, “A lot of organizations were conducting projects regarding COVID. However, this one was the only one providing actual information on practical ways to mitigate some challenges arising from the pandemic”.

Figure 6: Perceived complementarity of the project with other interventions according to the survey (n=130 respondents).

Source: Independent evaluation survey, 2023.

46. Partnerships with other UN entities and organisations proved challenging to harness. Despite the above and some other cases, the evaluation found that the project was not able to establish a cohesive eco- system of strategic partnerships. The initial concept note of the project considered that project governance would establish working groups on each outcome area, including “Digital economy solutions for advancing SME resilience (ITC, ITU, UNCDF, UN/CEFACT)”, “Resource Efficient and Cleaner production (led by UNEP and UNIDO)”, “Zoning/urban issues (led by UN-Habitat)”, “Employment (led by ILO)”. The Project Document for Phase 2 of the project also mentioned that “Close cooperation and partnerships should also be sought with World Bank, IFC.” This proved over-ambitious. In practice, the evaluation found limited global collaboration with UN entities also engaged in responding to the crisis and providing support to MSMEs, such as the ILO53, UN Women54, or FAO55. The evaluation also did not find evidence of the project attempting to leverage funding from other UN COVID-19 response mechanisms, such as the UN COVID-19 Response & Recovery Multi- Partner Trust Fund (MPTF) which financed projects at the national or subnational level in some of the countries where the project was implemented. Some informants stressed that Member States were looking for financial resources to support MSME resurgence, which is rather in the realm of the development banks. However, connections with the World Bank and the regional banks were limited. Partnerships appeared, therefore, primarily based on a bottom-up approach, relying on legacy collaborations or immediate opportunities, but not necessarily globally strategized and pursued. It should be noted though that the project did not dedicate specific resources to such activities (see next section).

5.3 Efficiency

How well coordinated was the process for the response among the entities implementing the joint project?

47. Despite unprecedented circumstances with the Covid-19 pandemic, and the lack of clear dedicated resources, the administrative, management and financial processes for the response were well coordinated overall. According to the initial concept note, Phase 1 of the project aimed to implement two components.

53 wcms_854253.pdf (ilo.org) 54 WEA-ActionBrief3-SMEs.pdf (weps.org) 55 COVID-19 Response and Recovery Programme | Resource Mobilization | Food and Agriculture Organization of the United Nations (fao.org)

28%

43%

18%

3% 2% 1%

5%

The project complemented interventions by other international organizations working in my country/region

Strongly agree

Agree

Somewhat agree

Somewhat disagree

Disagree

Strongly disagree

Do not know

30

Component 1 was to be led by ESCWA and planned to establish a virtual knowledge hub. Component 2 was to be led by UNCTAD and targeted the provision of hands-on assistance to countries on several sub- components grouped as 1+5 thematic clusters56. Although the narrative of the concept note did not identify an agency in charge of the overall coordination of the project, the budget featured the output “Coordination and proposal preparation” for which UNCTAD was to be provided US$ 27,000. This lack of clarity in the initial setup of the coordination of the project was reflected in the evaluation interviews, informants from the various implementing UN entities, including UNCTAD and ESCWA, having different perspectives on the scope of work and responsibilities for the coordination of the process during the first phase of the project (see also section on Effectiveness). The Project Document for Phase 2 clarified that UNCTAD would be the lead agency for the overall coordination of the project. This role was reflected in the budget of the project, with UNCTAD receiving US$ 40,000 for “Output –.1 - Overall coordination”, but not included in the project logframe. A Project Steering Committee was to monitor project delivery and make necessary adjustments to ensure smooth implementation 57. In practice, evaluation informants distinguished three different periods regarding the coordination process. The first period coincided largely with the preparation and implementation of Phase 1 of the project (from April to June 2020) during which the DA-PMT was reported by UN entities’ informants as having a very active coordination role. The second period spanned from August 2020 until April 2021, covering Phase 2 and part of Phase 3 of the project. It had regular (i.e., about bi-monthly) Project Steering Committee meetings coordinated by UNCTAD. Then, from April 2021 until the closure of the project, a third period saw the Project Steering Committee meetings attended by a larger number of participants (up to 40 participants). Project Steering Committee meetings held during the 1st, 2nd and 3rd phases were not technical but provided room to discuss and align activities. According to an informant who attended the 3rd phase meetings, “they were also used by UNCTAD to propose participants to build synergies on topics, but the project was so broad that this seldom happened (sic)”. Most often, the evaluation informants reported that these 3rd phase meetings were useful to learn about the activities of other implementing entities. However, the complexity of the project also made these 3rd phase meetings at times quite long and difficult to follow. Furthermore, the different time zones made it challenging for all staff to participate. In terms of administrative and financial management, project staff in UNCTAD reported an efficient and supportive DA team. This was valued as, according to a staff in UNCTAD, “the UN requirements to obtain the funds are onerous. There is a need for expertise for administrative management and administrative rules to prepare requests and reporting.” Monthly financial statements were provided to project management in UNCTAD. The DA also helped in understanding who among the UN entities was behind schedule and who was on track. According to an informant in UNCTAD, “In the project management meeting, one UN entity could have a narrative saying that all was going great but then, with the DA, we would see the financial delivery was only 12%. Then we knew we had to push delivery. The DA provided the implementation rate. Very helpful.”58 To support its internal coordination, UNCTAD specifically designed in April 2021 a new product, the project monitoring dashboard, consolidating information on UNCTAD’s planned outputs and how each intervention was progressing. Several informants from UNCTAD reported this as a very useful, centralized tool. The dashboard was regularly updated, despite being demanding to maintain. Collecting information from staff was time-consuming. Furthermore, the dashboard was very detailed, recording information at the activity level (some outputs could entail a dozen activities) and across 11 data fields. In preparation for the progress report for Phases 1 & 2 in May 2021, the dashboard was used to compile information from all the implementing UN entities. Despite most informants reporting appropriate and efficient project coordination considering the circumstances, informants in UNCTAD, and across the partner entities, shared the perspective that the project could have benefited from additional staffing capacities dedicated to global coordination as well as to coordination with countries and UNCTs, and to partnership building.

48. The technical coordination of the response proved sometimes difficult to install and sustain. According to the concept note (Phase 1), working groups and task forces were to be formed around thematic clusters to support a coordinated approach and complementary efforts among participating agencies. It was

56 The concept note indicated the creation of the following clusters: A- Access to Resources (Component 1); B- Entrepreneurship and business skills facilitation; C - Business facilitation; D- Access to Finance/financial literacy; E- Access to Technology; F- Access to Markets. 57 As per the Project Document for Phase 2: “Project Steering Committee - Representatives from UNCTAD, DESA, ECA. ECE, ECLAC, ESCAP and ESCWA will have online meetings once in two months to monitor project delivery to ensure smooth and steady implementation. Representatives from all seven entities will meet on an as-need basis for important decisions that affect all seven entities.” (p. 53). 58 As a side note, DA Focal Points reported that meetings of the DA Focal Points network became more dynamic and substantive with the response to the Covid crisis.

31

initially envisaged that members of these task forces would include the implementing UN entities as well as other UN partners. The Project Document (Phase 2) narrowed down the prospective setup for the clusters’ technical coordination by referring to Task Forces comprised of experts from the implementing entities. According to an informant, “The discontinuation of the original concept note was positive because it was unrealistic. The problem is that staff members were on other jobs, no one exclusively working on the project. It was time-consuming to coordinate and staff had different agendas and there were different priorities between partners.” In practice, this cluster-based thematic coordination was loosely implemented but fluid, sometimes involving sharing of information between several implementing agencies, though more often one- on-one bilateral and direct exchanges. As noted earlier, interviewees reported information sharing with UNCTAD and sometimes directly between the Regional Commissions “but then entities were doing their own part”. One of the cluster leads for example proposed that the Regional Commissions undertake a global analysis on market access and COVID-19 to develop a joint policy paper, but this did not work: “Colleagues wanted it but there were so many things to do. Project managers did not approve it and it was not in the logframe.” In that sense, the technical coordination of the project and its contribution to the efficiency of the response was eventually more closely aligned with the aim stated in the initial concept note as “(t)he effort will help to enhance coordination in the area of the MSME promotion in the UN system, and avoid confusion, duplication, and waste of resources”. Informants in UNCTAD and the Regional Commissions shared the perspective that directing adequate staffing capacities specifically to technical coordination may have facilitated the establishment of stronger synergies between the implementing entities. Furthermore, despite most UN informants being supportive of the global architecture of the project, including as a modality to enlarge partnerships and seize the benefits of a “One UN” system approach, a few staff from the Regional Commissions suggested to commit more time in the future to analyse the added value of global/inter-regional versus regional projects. It was argued that coordination costs increase with the complexity of the project and with the number of actors involved. As indicated by a staff in ECE, options to balance would include designing a global project or inter-regional or regional projects “and then coming together for dissemination at global level". Therefore, “the added value of a global project should be thoroughly assessed before project design. But during Covid it was a global challenge. […] Regional projects might have been an option. If going through inter-regional projects, their added value should also be well assessed. But in crisis, with the short timing and need to bring everybody under the project, one needs to cut some corners.”

How did the three-phase budgeting and programming approaches impact the efficient delivery of the project?

49. The phased approach was found to be a valid mechanism to cope with the uncertainty of the crisis. The project was developed and implemented under a three-phase budgeting and programming approach: Phase 1 spanned over 2 months (May-June 2020), Phase 2 was implemented in 6 months (July-December 2020), and Phase 3 was provided a total of 18 months (initially January-December 2021, extended twice, first to March 2022 and then to June 2022). On the programming side, during Phase 1 various entities were able to leverage existing products to deliver some immediate responses to countries that had requested assistance. In general, during Phase 2, the project contributed to develop tools, deliver technical support, and customizing materials required by countries. Phase 3 enabled implementation at country level of the tools developed during phase 2 and, building on country experience, to collect good practices and lessons learned that further informed policy makers. Technical staff interviewed by the evaluation were largely positive about the 3- phased approach. Interviewees recalled that this was a very difficult period and that it was unclear when the pandemic would end. Although a clear project plan for the entire period would have been beneficial, the context was too unstable to warrant it. According to a staff member in one of the Regional Commissions, the 3-phased approach “was the optimal and rational thing to do”. It was well aligned with the uncertain circumstances and supportive of a learning process. According to staff at UNCTAD, “Phases helped adaptive management” as “each phase was an innovative stage that gave a chance to introduce new things not foreseen at the beginning”. Incidentally, as put forward by a staff member in one of the Regional Commissions, it shows that the UN had an agile way of thinking to cope with the challenge. On a few occasions, a more nuanced assessment was conveyed by interviewees. Several informants reported that the phased approach added to the complexity and uncertainty of the project and made it more difficult to understand its design, and that a Theory of Change was missing.

50. The project would have benefited from providing clearer visibility on the funding available. From a financial standpoint, the five Development Account COVID-19 projects, including this project, were funded from various DA funding sources. This entailed relying on interest savings, residual balances from the 10th and other closed tranches, and savings from the 11th tranche. Accordingly, the overall available funding was

32

confirmed by phase. It was determined in late 2021 that the combined proposed phase 3 budgets submitted by the five projects exceeded the funding that became available by 1 million USD. As a result, the projects were requested in November 2021 to reduce their proposed phase 3 budgets, and for this project, the budget was reduced by $310,000, from a total of $3,165,500 to $2,855,500. UNCTAD received the final instalment for phase 3 in December 2021. Due to the requirement to reduce the budget and revise the planned activities for phase 3, all five projects were extended by three months to March 2022. However, according to informants at UNCTAD, this initial 3-month extension did not fully account for the late final instalment in the context of a global crisis. A UN staff member mentioned, for example, that if implementing UN entities “want to recruit a consultant, they need funds, then to identify candidates, then 2-3 weeks to issue the contract, then the consultant needs to do the work. That can take 3 months in normal circumstances”. However, the context of the pandemic imposed additional delays. Some activities that were planned to be delivered in person had to be delivered online instead in 2022, which required time for reprogramming based on the new budget. Furthermore, counterparts were also challenged as there were uneven capacities in beneficiary countries to put in place some of the interventions. Accordingly, activities for which disbursements were made shortly before the closure of the project in June 2022 were still underway several months later. Informants indicated that this contributed to the delayed completion of the final report of the project, together with limited resources for project monitoring and reporting in the Regional Commissions, particularly when the DA Focal Points were not associated with project implementation (see next section). Some staff also felt that the phased approach created a lack of visibility on the funds that would be mobilized by the DA. This was sometimes found to be a challenge when designing the project, as it prevented setting clear expectations and a clear design. Better visibility on the resources in the pipeline could have helped to construct a different logframe, which would have been designed at phase 2 and used at phase 3 with little revision. According to a UN staff member, a clearer view on the funding available for the entire duration of the project could have helped the teams to consider additional scenarios, such as focusing some of the activities on a more limited number of countries with greater depth.

5.4 Effectiveness

To what extent did the programme (Development Account) and project governance and management structures and processes enable, or hinder, the effective implementation of the joint project and the achievement of its results?

51. The Development Account programme, as a funding modality, was an effective tool in supporting the design of a global response to support the resurgence of MSMEs post-COVID-19, with remaining questions about it being the most adequate tool in comparison with other UN agencies and mechanisms with a mandate for humanitarian response. DA is a regular budget capacity development programme. It funds, among others, operational workshops, advisory services, and travels to countries to deliver capacity-building activities. Projects are approved by the General Assembly as part of the programme budget for the first year in the biennium. The project implementation period (from receipt of funds by implementing entities to the financial closure of the project) usually is 4 years, following a design or planning period that takes about 1.5 years. 59 Given project specificities and circumstances, and, possibly, limited knowledge of the DA rules and procedures, evaluation participants reported different views on the duration of DA-funded project cycles, the requirements for partnerships and indicators, and the DA governance structure. The converging perception is that a regular DA-funded project cycle is long and onerous to plan for. Nevertheless, to provide a swift and global answer to the COVID-19 crisis, an adaptive management approach was adopted. These procedures were made flexible, the design phase was significantly shortened, and a phased planning was introduced. For the five COVID-19 projects, the Surge project included, the GA approval of project concepts was skipped, while the endorsement by the DA Steering Committee remained and the use of residual balances from closed and closing projects to fund the five projects was retrospectively reported in the fascicle for 2022, which was submitted to the GA in 2021.60 Hence, the concept note of the Surge project and 1.5 years of implementation

59 Under regular tranches of DA, the design or planning phase of projects starts with the launch of a DA new tranche by the USG for Economic and Social Affairs and the development of concept notes by 10 DA entities, and ends with the approval of the project documents by the DA Programme Management Team and the endorsement of DA budget and funded projects by the General Assembly. 60 A report commentator referred that “this has happened in the past when developing projects using residual balances from the tranche budget, but has never happened, nor is allowed to happen, for regular tranche projects.”

33

period was endorsed to initially respond as quickly as possible to the urgent requests from Member States to address the socio-economic fallouts of the pandemic. It was widely reported by informants that this was the best approach to the circumstances, as no one knew how long the crisis would last, with expectations that it could be just a few months. An advantage of this alternative path to access residual balances from closed and closing projects, reported to this evaluation by a DA representative, is the almost immediate availability of resources. This promptness is praised by stakeholders in many implementing entities as a ‘unique’ achievement.

52. Nevertheless, when the DA was compared with other UN agencies and mechanisms with mandate and resources to provide humanitarian assistance in a crisis context, informants referred to the difficulties to be as prepared as these other agencies. The strategy, then, was to focus on post-crisis recovery and development, as a suitable DA approach to actively support countries and, specifically to the Surge project, MSMEs. Yet, informants from the regions referred that adaptations to the project document guidelines kept the highly demanding requirements of the fund and this was assessed by them as an issue to be reviewed if the fund is to be used in future crises. A better crisis-fit approach would have involved an even more agile track, and a focus on deliverables of immediate effect to mitigate the crisis’ impact on MSMEs and government policies (e.g., enhanced access to emergency finance and digital tools for trade). Informants shared expectations about the DA-PMT’s review of the five COVID-19 evaluations toward helping to inform the DA’s role in dealing with future shocks.61 Among members of the project coordination team, for instance, there are ideas about the usefulness of the Entrepreneurship Policy Framework as an overarching tool with relevant potential outputs for crisis response, as discussed later. Considering that the EPF is the key framework of UNCTAD to fulfil its role as the UN focal point for entrepreneurship and MSME policy for sustainable development (RES/71/221 and A/RES/73/225), EPF-related ideas can be considered as a tool to support crisis response funded by the UNDA.

53. Project governance and management structures and processes, overseen by the DA-PMT, supported a quick and overall effective response in light of the UNDA standards, identifying available resources and potential partnerships that ensured the delivery of an impressive number of outputs and achievements at the outcome level, even if joint delivery between implementing entities took time to come to fruition and some DA-FPs were ambivalent about the extent to which they were involved. Addressing a crisis of global scope such as the COVID-19 pandemic was new to the DA’s management structure. Seeking to find quick and effective ways to deal with it, the Head of the Capacity Development Programme Management Office of DESA, in his role of assisting the DA Programme Manager, sent out a memo to the DA focal points in the 10 implementing entities on 18 March 2020, inviting them to a Skype call to discuss the DA's COVID-19 support to Member States, in particular the possible reorientation of the ongoing 10th, 11th and 12th tranche projects, on 20 March 2020. The meeting minutes indicate that the participants were informed of two proposals for potential response initiatives from ESCWA (on SMEs) and from DESA (on statistics). A follow-up meeting was convened on 26 March, where five other project proposals were discussed, and other implementing entities were requested to provide comments on those proposals, as well as submit any additional proposals. After these extensive consultations with DA-implementing entities, DA-PMT selected five topics considered as the most relevant to a socio-economic response to COVID-19 and built on comparative advantages of DA- implementing entities. This included strengthening the capacity and resilience of MSMEs to mitigate the economic and social impact of the pandemic – ESCWA’s proposal. The preliminary idea of this project, as reflected in its 1-page concept note, focused on digital tools to support SMEs. UNCTAD further enriched it with additional ideas, such as the e-registration platform and entrepreneurship for SMEs. The project concept note finally took shape with the inputs of more entities, including DESA and other four regional commissions. As more implementing entities joined the project, UNCTAD supported the clustering of activities in 5+1 outcomes, with implementing entities taking outcome leadership according to their expertise and capacities. This structuring of the project governance, with DA-PMT’s oversight, and the management structure with UNCTAD in the overall coordinating role and each implementing entity leading the operational implementation at regional level ensured the global and regional scope of the project delivery. At the level of the ongoing processes, implementing entities relied more on partnerships with local and regional institutions rather than coordination within the larger project management structure. The online meetings of the steering committee

61 An updated information about this review received in feedback to this evaluation report refers that “this programme- level evaluation had to be cancelled due to the delay with the completion of four of the five project evaluations (only one has been completed to date, and two are expected to spill into 2024). A meeting of the DA focal points to discuss the findings of the five project evaluations will be organized by DA-PMT as soon as they are completed; the DA's role in dealing with future shocks would certainly be discussed at this meeting.”

34

and the smaller-scale coordinating role of the task forces to weave efforts worked to a limited extent, as these meetings were not as regular as planned in the project document. The result was a high delivery rate with limited inter-agency coordination in the process. Yet, this independent evaluation identified that the higher- level governance and management structures and processes were effective in bringing the project together with identifiable results at the outcome level.

54. Given the uncertainties about how the pandemic would evolve, and considering the pressing immediate needs, a first phase of the project was launched with a limited number of outputs (n=7), in May 2020. It resulted from meetings of the DA Network in March and April 2020. From April 2020, these became bi-weekly meetings of the DA focal points. It is noteworthy that this process was not recollected as such by the informants of this evaluation. There is a perception that not all DA-FPs were involved from the start, that this inclusion was gradual and followed advances in the project. Informants acknowledged that these meetings also included staff from the implementing entities to discuss the next steps to be taken, the different country and regional needs, and the installed regional expertise to propose responsive actions62. This trickled down into several smaller-scale bilateral meetings at the regional and local levels. It became clear that the DA’s mandate on development had to be taken into consideration when planning the project outputs and adapting activities associated with crisis response (e.g., providing direct health support was out of the scope of possible answers). The DA’s development mandate, instead, could support the strengthening of institutions that needed preparedness and resilience to design supportive measures in a post-pandemic recovery perspective. Hence, out of five projects, the Surge was one with the objective of developing and implementing capacity-building tools for governments and MSMEs to facilitate the resurgence and strengthen the resilience of MSMEs in developing countries and economies in transition. There was a need to bring the relevant expertise of different entities quickly on board. Because of its global mandate and expertise on entrepreneurship and MSME policy, UNCTAD became the lead agency of the Surge project, and coordinated the elaboration of a project document to deliver a global response with more entities. For the first time, the DA-PMT was intensively involved in the oversight and monitoring of project implementation, with more direct contacts with the technical cooperation staff – rather than going through organizational-level DA focal points. “The DA PMT was in contact with the DA-FPs but had more contacts with the project managers than it would normally have.” (DA-PMT representative). This was made possible by the small number of projects responding to the COVID-19 crisis. This centralized approach allowed for a swift response at a global level, with a record time endorsement of the concept note and the Phase 1 budget by the DA Steering Committee. DA-PMT’s oversight and direct contact with technical staff resulted in the institutionalization of periodic meetings with DA-FPs and other stakeholders. UNCTAD DA-FP, represented by the Technical Cooperation Section (TCS), served as a bridge between the DA-PMT and UNCTAD project teams, especially at the design stage. UNCTAD TCS also provided substantive support to the UNCTAD project coordinator in the draft of the concept note of the MSME Surge project. During the implementation stage, UNCTAD DA-FP participated in several project Steering Committee meetings and addressed questions from project teams. UNCTAD, as the leading entity, and under the guidance of the DA-PMT, communicated directly with the project focal points to monitor progress. This monitoring was supported by UNCTAD Budget and Project Finance Section through the sharing of financial information on implementation rate by the implementing entity. This, and the evolution of the pandemic, brought more clarity about the needs and the implementing capacity of UN entities and member states in responding to the crisis. It supported the design and implementation of phases 2 and 3. There is consensus between informants that the introduction of the phased planning was consistent with the uncertainties of the pandemic crisis, and they would not recommend another way to respond to it, despite reporting some bottlenecks.

55. There are different perspectives on how well the DA procedures were adapted for crisis response. Following the requirements for joint proposals as much as possible, according to informants, felt like squeezing a 4-year long implementation period into 1.5 years. It was suggested by some informants that, instead of a concept note and a PRODOC, a crisis response could have required only one of them. Outcomes were over- ambitious for a 2-year project, but project planners felt they had to comply with the DA requirements, even if this would imply challenges to deliver at the outcome level. Indeed, some outputs were delivered after the end of the project (see next section). There was and internal quality control of the PRODOC with multiple inputs, oversaw by the DA-PMT to ensure a ‘good enough quality’ level, however, a higher-level involvement of all relevant DA-FPs in dialogues and feedback loops from the start and the gender review and other quality assurance steps (i.e., internal clearance processes of the implementing entities of interim documents) could

62 Other informants referred to these meetings as more about sharing information than coordination between implementing entities.

35

have improved the quality of the PRODOC further. For instance, it could have ensured that project indicators cover the SMART63 criteria and be better aligned with a crisis response context – some were the same Empretec-based indicators in use for the past 10 years. In one example suggested by an informant, instead of having an indicator on increased sales of MSMEs, the project could have one about ensuring that a certain percentage of MSMEs that received project support remain in business. Informants confirmed that designing the logframe of the project was challenging because of the scale of the COVID-19 crisis, the number of implementing entities involved, and the uncertainty about the level of funding associated with the phased approach. The high number of planned activities and countries would imply a thin distribution of financial resources. What some project teams did to manage the uncertainty of resource availability and ensure delivery in the Surge project was saving resources from regular projects. Informants acknowledged that these managerial and processual arrangements were the best that could have been done during the crisis and the uncertainties associated with it, but also acknowledged that, even with some trimming in the proposed activities, the end result was less focused than would usually be the case. A lesson learned is the need to assess what crises the DA programme is equipped to respond to (e.g., economic, financial, market, health, climate, national, regional, global, etc.) and what tools can be immediately made available to countries in need. For instance, the e-registration platform was mentioned as an existing technology for e-government that was rapidly scaled up to respond to the needs of MSMEs in the COVID-19 crisis.

56. The project governance and management structures and processes were effective in supporting bilateral collaborations between UNCTAD and individual Regional Commissions in delivering project outputs, but there were limited project-level management mechanisms. Nevertheless, the operating mechanisms supported communication and seeded exchanges of information between regions with increased joint deliveries toward the end of the project. From the perspective of the DA-PMT, according to interview data, one of the purposes of bringing all implementing entities together in bi-weekly meetings was to foster dialogue and help collaboration for a quick response to the crisis. It seems that this did not represent an inter-entity coordination function of these meetings, but a sharing of knowledge to be taken up by other coordination structures. A project coordination team, with two representatives of UNCTAD and one of each of the other implementing entities, was set up. The actual coordination was mostly centralized in UNCTAD, as the project lead. The perception of some members of the PCT is diverse in relation to how the PCT worked, with some of them referring to themselves as focal points rather than members of the PCT, and others not providing substantive evidence on how the PCT, as a whole-of-project coordination structure, worked. Evidence from deliverables indicates that collaboration was bilateral, mostly between UNCTAD and individual Regional Commissions, and built on previous successful collaborations. Some informants referred to parallel streams of work without coordination with the work of other Regional Commissions. Others saw the Surge project as an opportunity to learn about what other Regional Commissions were doing and to set the stage for future collaboration. For instance, ESCAP, ESCWA and ECA exchanged information about the DEPAR portal, exchanged materials, and participated in meetings; ECE and ESCAP acknowledged that mutual cooperation can be facilitated by them having member States being in both commissions. Interviewees from all implementing entities referred to attending events organized by other entities. An advanced level of cooperation was achieved by ECLAC and ECA (through its Africa Trade Policy Center (ATPC) and IDEP), on the online course material on competition policies delivered in Portuguese. They engaged in initial conversations and, because the pace of implementing this specific output was different in each entity, this was an asynchronous cooperation, mostly involving the sharing of course materials. Yet, this is reported by ECLAC as a successful South-South cooperation during a turbulent crisis context.

57. UNCTAD, as the lead agency, focused on bilateral cooperation according to regional expertise and planned outputs. UNCTAD cooperated with ECLAC, ESCAP, ESCWA, ECA, and ECE in the delivery of outputs such as “The COVID-19 pandemic impact on micro, small and medium-sized enterprises – Market access challenges and competition policy” (phase 1, output F7). With ECLAC, UNCTAD collaborated in delivering a training course on competition law and policy, an online course on competition and SMEs and cross-border trade, outputs on competition policy, and technical assistance for projects on Asia exports in the LAC region. With ESCAP, UNCTAD collaborated with the creation of the ASEAN SME policymakers’ network, and a discussion paper and meetings proceedings of this network. With ESCWA, UNCTAD collaborated in actions on topics such as e-registration, and the organization of an event on digitalization. This partnership grew stronger and new partnerships emerged in the area of competition. With ECE, UNCTAD collaborated with a report on SMEs, and counted on ECE speakers contributing to its workshops. This is consistent with UNCTAD’s traditional approach of engaging in bilateral cooperation, rather than interregional. A factor associated with

63 Specific, Measurable, Achievable, Relevant, and Time-Bound.

36

this approach is the source of demands for assistance, which normally are country- or regionally based. In addition to collaboration between implementing entities, the Surge project also promoted intra-agency cooperation. This was reported within UNCTAD, involving Empretec, the competition and consumer protection policies branch and the investment branch, within ESCAP, on the Bangladesh-related outputs, including the UNCTAD Voluntary Peer Review of Competition law and policy of Bangladesh, with a focus on SME, and within ECLAC, between the economic affairs and international trade divisions. This independent evaluation identified that, toward the end of the project, in phase 3, there were more joint deliverables involving different Regional Commissions (e.g., the online events on competition policy and MSMEs held in phase 3), which suggests that cooperation increased as the project advanced. A lesson learned from these collaborations is that cooperation depends on a combination of countries’ needs, sources of resources (e.g., donors’ requirements, modalities of funding and what they allow to be funded), continuous presence of staff and time for coordination. This makes planning for cooperation a challenge, especially under a global crisis of the COVID-19 scope. Nevertheless, higher collaboration could have been fostered had the project coordination team worked more effectively and planned joint outputs. Hence, despite Surge’s opportunities for cooperation, actual results of these initial efforts are at an early stage.

To what extent has the project contributed to the expected outcomes as enunciated in the project document?

58. Eighty-six outputs were delivered across the world’s regions, built on the expertise of UNCTAD and the Regional Commissions, and some degree of inter-entity collaboration. These deliverables composed a mosaic of actions that, on a global scale, contributed to achieving the outcomes enunciated in the project document. Implementing entities adapted resources and expertise that they already had and expanded (mostly bilateral) inter-entity partnerships to quickly attend to the most urgent needs of entrepreneurs and policymakers to support MSMEs in the first 2-3 months of the project. These actions comprised making information digitally available64, converting face-to-face training into online, developing digital platforms to support MSMEs and governments in responding to the crisis, and producing and disseminating assessment reports on the impact of COVID-19 on MSMEs. UNCTAD, ESCWA, and ECE delivered the eight outputs of phase 1, with some collaboration with the other entities to collect data for the development of these outputs. For the delivery of the Competition and Consumer Protection assessment report (output F7 of phase 1), under UNCTAD lead, for instance, all regional commissions were involved. Outputs in phase 1 were delivered not knowing how long the crisis would last but considering it could extend to 2021. As successive COVID-19 infection waves hit the world, the end of sanitary measures was not foreseeable. Phases 2 (planned to last until the end of 2020) and 3 (planned to last until the end of 2021) expanded the project to support more countries and MSMEs, and increasingly relied on inter-entity collaboration to ensure a high rate and quality of delivery. The PRODOC of these phases, then, involved seven implementing entities and shifted from general responses (phase 1) to the delivery of a wide range of country- and region-based interventions65. The project’s responses to these needs depended on the locally installed capacities to develop more resilient institutions and supportive policies to MSMEs. Some of these deliverables, mostly those related to online awareness- raising, policy dialogue and workshops, relied on inter-entity partnerships to be successfully delivered. Examples from phases 2 and 3 are the UNCTAD-ESCAP Regional Dialogue held in cooperation with the ASEAN Secretariat, the series of Online regional capacity-building events on the role of competition policies for access to markets in post COVID-19 resurgence of MSMEs (UNCTAD-ECE, UNCTAD-ECLAC, UNCTAD-ESCAP), the UNCTAD-ESCWA-ECA Regional Policy Dialogue: The role of Competition Policy in supporting MSMEs economic recovery in the post COVID-19 crisis, the Sub-regional workshop: Global initiative towards post- Covid-19 resurgence of the MSME sector – Women in informal and small-scale cross-border trade (UNCTAD- ECA), the online course Oportunidades y retos para Mipymes de América Latina en el e-commerce transfronterizo66 (ECLAC-UNCTAD), and the Global Policy Dialogue: Post COVID-19 Resurgence of MSMEs and Competition Policy (all seven implementing entities), The Surge project, therefore, effectively delivered a wide range of interventions (i.e., advisory services, trainings, workshops, seminars, digital portal, events, and

64 Smart search engine: Manara Portal (unescwa.org); MSME knowledge portal: https://msme-resurgence.unctad.org/ 65 From a project management perspective, phase 2 was to develop intervention tools and phase 3 was to implement them at country level. The analysis of the deliverables, however, shows a mixed picture, determined by the local capacities to move from the development of a tool to its implementation. Many outputs in both phases 2 and 3 combine, for instance, course development and delivery (e.g., OP 1.5 and 1.6, OP 3.4 and 3.5, and OP 5.10 in phase 2, and OP 1.14, and 5.3 in phase 3). 66 Translation: Opportunities and challenges for MSMEs in Latin America in the cross-border trade.

37

direct technical assistance through policy advice, assessment reports, research studies, and tailored guidelines), directly assisting 96 countries in specific ways. A good practice that comes from this approach is that a global response can be made of multiple local and regional actions that are the most relevant and impactful for its recipients during a crisis. Figure 6 shows the distribution of project-planned activities per phase and implementing entity. The number of outputs planned increased from 7 in phase 1, to 38 in phase 2 and 60 in phase 3 (+57.9% from phase 2), totalling 105 outputs, even under a budget cut of USD 310,000. UNCTAD was responsible for 50% of the outputs in phase 2 and 43% in phase 3.

Figure 7: Surge Project outputs delivered per implementing entity and implementation phase67.

Source: Project’s final report, 2022.

59. It is noteworthy that the number of outputs slightly changed between the PRODOC and the project final report. Additionally, this independent evaluation received evidence of deliverables completed after the conclusion of the final report, in 2022 (e.g., five outputs delivered by UNECA in phase III), and others that do not clearly fit in any of the project planned outputs (e.g., ESCAP’s papers on MSME Financing Series, delivered in 2022). Table 7 summarizes these changes and updates and presents the percentage of outputs delivered.

Table 7: Planned and delivered outputs per project phase. Phases PRODOC

outputs Final report

outputs Delivered outputs*

% delivered Not delivered**

1 7 8 8 100 2 38 39 36 92.3 4.3, 5.9, 5.13

3 60 54 42 77.8 1.25, 2.2, 2.3, 3.2, 3.3, 3.4, 4.1, 4.3, 4.5, 4.7, 5.5, 5.6

Total 105 101 86 85.2 * Based on evidence collected in this independent evaluation. **These outputs fall into the following situations: not delivered due to budget cuts, in progress (i.e., reports in the inception phase by the time of this evaluation), or missing information.

60. Overall, 85% of all project outputs were delivered. Phase 3 has the highest number of outputs and the lowest percentage of effectiveness, around 78%. It was impacted by the budget cuts, the longer time needed to conclude some outputs and the lack of information for this independent evaluation. In cases of budget cuts, this evaluation counted as delivered those outputs with at least 50% of the activities concluded. It is noteworthy that the delivery of some of these outputs was concluded after the end of the project68, given factors such as the pandemic barriers to implementation, coordination difficulties, personnel turnover, and

67 Figure 7 presents the number of outputs fully delivered as stated in the project’s final report. In addition, 6 outputs were still in progress when the final report was submitted (ECLAC 1 output, UNCTAD 1 output, UNECA 4 outputs in progress). 68 Interviewees used the project’s final report (2022) as reference for the end of the project, since some of their deliverables, pending then, were concluded afterwards.

Phase 1

Phase 2

Phase 3

0

5

10

15

20

UNCTAD DESA UNECA ESCAP UNECE ESCWA ECLAC

3

0 0 0 2 2

0

19

2 4 4 4

1 2

19

1 3 2

9 7

4

Number of project outputs delivered per phase

38

limited capacities of member states. Considering all these barriers, this independent evaluation considers the level of effectiveness high, since these rates of delivery contributed to the achievement of the main project objective through significant achievements in each project outcome, as described next.

61. National capacities on formulating and implementing enabling policies on green, resilient and inclusive entrepreneurship and MSME promotion in post-COVID-19 resurgence (outcome 1.A) improved through e-learning courses, awareness-raising activities, technical assistance in designing and implementing national entrepreneurship policies, the creation of regional networks of policymakers, and knowledge production and hubs on MSME’s policies. Deliverables to support national capacities started in phase 1, with the launch of the DEPAR portal by ESCWA. This portal provides a wide range of tools and services to MSMEs, such as the MSME toolbox69, with business guides, inspiration stories, start-up tools, external tools and news from partners on innovation, technology, policy-making, etc. It allows visitors to become part of a community of practice, in which participants can share content, build their own community, create events, develop courses, and announce opportunities. One of the ideas for the portal was to provide a one-stop shop for materials from other Regional Commissions too, but, later, it became clear that this would lengthen the path70 for MSMEs and policymakers to access materials from their respective regions and this was not implemented. Figure 8 indicates a growing number of users to the portal, with about 2000 users four months after launch, and about 3000 users one year after launch. Most of these users are from Jordan (59%, according to data from 20 June 2023). There was no qualitative information to explain the quantitative trend.

Figure 8: DEPAR number of users.

Source: ESCWA.

62. Other deliverables under outcome 1.A included e-learning courses based on the EPF; a series of webinars to policymakers on entrepreneurship policies for MSME resurgence; technical assistance in designing and implementing national entrepreneurship policies; the creation of the Asia-Pacific MSME policymakers’ network and follow-up workshops; the creation of a knowledge hub on policy measures, tools and practices on MSMEs’ resurgence; regional workshops to promote dialogue on women’s role in small-scale and informal cross-border trade and tailored recovery; development of the ‘Compendium of Renewable Energy Technologies’ with an online toolbox and training materials on green recovery of MSMEs; policy roundtables with communities and SMEs; and design and implementation of entrepreneurship policies and strategies based on regional assessment. Survey evidence from this independent evaluation corroborates the positive assessments observed for individual deliverables, with approval rates above 74% across project dimensions (Figure 9). It is noteworthy that, given the small response rate, survey results need to be interpreted with caution, keeping in mind that these findings reflect the perceptions of those who responded to the survey questions, and are not representative of all participants (See the methodology section for details).

Figure 9: Effectiveness results of outcome 1.A (n=51 respondents).

69 https://depar.unescwa.org/msmetoolbox. 70 This would increase the need to liaise with ESCWA to upload information according to regional needs (i.e., more time to have information available online), and could impose a barrier to MSMEs and policymakers from other regions to access their regional information (i.e., by adding ESCWA as an intermediary institution between local stakeholders and their respective RECs).

39

Source: Independent evaluation survey, 2023.

63. In the second half of 2021, the 5-week UNCTAD/UNITAR EPF e-course launched with the objective of enhancing the capacity of policy makers to provide policy response and support measures on MSMEs for post-COVID-19 resurgence and develop longer-term strategies for a resilient and competitive MSME sector. One of the course contents explicitly addressed how the environment for and role of entrepreneurship evolved during and in response to the COVID-19 pandemic. It was developed in English, French and Spanish, and covers all five pillars of the EPF. It was preceded by the launching of webinars with the participation of UNCTAD officers, UNITAR representatives, policymakers, and thematic experts. The UNCTAD / UNITAR 2021 Summary Report of this EPF online course presents data for each of its three versions. Results indicate a gender balance between the number of male (n=221) and female (n=223) registrants to the course in the English version, a higher number of female registrants (female=65, male=48) in the Spanish version, and a higher number of male registrants (male=78, female=52) in the French version. Between 5-7% of registrants indicated another gender in the three course versions. The majority of survey respondents, in the three versions, indicated that the course presented new information (>75%) and was relevant to their jobs/occupations (100%). The likelihood of survey respondents applying the information acquired from the course is above 96%, and all respondents would recommend the course to a friend or colleague. This e-course supported UNCTAD’s assistance to Uganda, Seychelles and South Africa in designing and implementing national entrepreneurship policies (NES) based on the EPF. The NES of South Africa focuses on Youth Entrepreneurship, the NES of Uganda on migrant and refugee entrepreneurs, and the NES of Seychelles on the marine biotechnology sector with reference to youth entrepreneurs. Although there was an impact assessment of the NES implemented before the COVID-19 pandemic in The Gambia, this was not delivered due to the budget cuts in phase 3.

64. A set of deliverables by ESCAP included the launch of the terms of reference of the Asia-Pacific MSME Policymakers’ Network (February 2021), followed by a concept note. Three online regional dialogues of the Network were organized by ESCAP in partnership with UNCTAD and ASEAN secretariats. They covered the digital economy (including how to support women entrepreneurs through ICT, business skills training and innovative financing), the role of competition policy in strengthening the business environment, both in March 2021, and the launch of the Policy Guidebook for MSME Development in Asia and the Pacific71, in May 2022. Documents produced to support the Network include a paper on the digital economy72, UNCTAD’s broad survey on the impacts of the COVID-19 on the business environment, particularly in relation to competition issues (2020)73, and the regional report ‘The Role of Competition Policy in Strengthening the Business

71 This book describes the MSME contribution to achieving the SDGs, with comprehensive and practical information to policymakers on MSME development. 72 Callo-Müller, M. V. (2020). Micro, small and medium enterprises (MSMEs) and the digital economy. 73 UNCTAD (2020). Covid-19: Firmer Action Needed to Better Protect Consumers. [online] https://unctad.org/news/covid-19-firmer-action-needed-better-protect-consumers.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

provided access to information and best practices on government support measures for post COVID-19

recovery

improved my knowledge on entrepreneurship/MSMEs policies design and implementation in the context of

post COVID-19 recovery

contributed to identifying new/improved policy measures for MSME promotion in post COVID-19

resurgence

The project...

Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

40

Environment for MSMEs in the ASEAN Region’ (2021)74. Following up on the second meeting, there was a report with recommendations to governments, competition authorities, development agencies, and multiple stakeholders on competition policy measures in a pandemic recovery context. Assessment data is available for the two first meetings and shows that more than 75% of the respondents indicated enhanced knowledge and skills, allowing them to effectively receive from and share information with others in the region regarding MSME policies (>67%), and providing opportunities to enhance regional cooperation in competition policy and law (>69%). Comments from participants referred to the advantages of sharing experiences with other countries, learning how they try to overcome MSMEs’ problems during COVID-19, and providing ideas on how to promote digitalization to MSMEs via education. Interview data indicates that the Network became inactive afterwards, since ESCAP dropped the MSME agenda after the project to focus on other demands.

65. In September 2021, the ‘UN-ESCWA Entrepreneurship Community Workshops and Policy Roundtables – How to improve existing government and non-government programs’ took place, with a set of interactive discussions on existing government programs, existing non-government programs, needed government programs, and needed non-government programs, with follow-up recommendations on these areas. Specifically on the health economy, a set of online conferences on ‘Entrepreneurship in the new health economy’ were delivered in October 2021. One targeted the Asia-Pacific region75, with 45 attendees, another targeted Latin America76, with 25 attendees, and another, Africa77, with 33 attendees. They were organized in partnership with private sector organizations in the health sector for policymakers. It is noteworthy that, except in the African event, these conferences did not include policymakers among their panellists. These conferences generated inputs to the publication ‘Entrepreneurship & innovation in the new health economy’78, launched by UNCTAD in 2022. This document highlights the role of SMEs and start-ups in the new health economy in developing countries, with main findings on the role of regulation in the e-health sector, and the role of public institutions in building bridges between different stakeholders.

66. The resilience and competitiveness of MSMEs in post-COVID-19 resurgence (outcome 1.B) improved through a large set of Empretec-related activities (including trainings tailored to low literate entrepreneurs), cross-border trade workshops for female traders, adaption of the Farming as a Business program, and country-tailored guidelines and best practices on circular economy and sustainable resource management. Outcome 1.B is the one with the highest number of deliverables (n=28), each encompassing sets of activities and supporting MSMEs in multiple countries. They can be grouped in four packages: Empretec (responsible for most of these deliverables and demonstrating the effectiveness of UNCTAD’s pre-existing capacity to deliver trainings, including onsite, during the COVID-19 pandemic), informal and small-scale cross-border trade for women traders, UNCTAD’s Farming as Business program, and Guidelines and best practices on circular economy and sustainable resource management79. Survey evidence from this independent evaluation corroborates the positive assessments observed for individual deliverables, with approval rates above 68% across project dimensions (Figure 10).

Figure 10: Effectiveness results of outcome 1.B (n=39 respondents).

74 ESCAP (2021). The Role of Competition Policy in Strengthening the Business Environment for MSMEs in the ASEAN Region. 11 February 2021. NOTE: there is no authorship in the document. This reference may need revising. 75 Scaling innovation in the new health economy – lessons learned from the Asia-Pacific region. 76 The rise of medtech in Latin America. 77 Unlocking digital health innovation in Africa. 78 UNCTAD (2022). Entrepreneurship & innovation in the new health economy. Geneva: UNCTAD. 79 Given the higher emphasis on government advice, more details on these guidelines are in the highlight box on the contribution of the Surge project to the country’s government responses to the COVID-19 crisis regarding the resurgence of MSMEs.

41

Source: Independent evaluation survey, 2023.

67. Effectiveness evidence of the Empretec package of deliverables indicates that development of the updated online training course (E6DM) was a strategy to facilitate post-COVID-19 entrepreneurship recovery. Ten Empretec webinars, jointly organized and delivered by UNCTAD and Empretec Centres between April and December 2020, were attended by Jordan, Malaysia, Ghana, Nigeria, Russia, South Africa, The Gambia, Argentina, Benin and Zimbabwe. In tandem, 5-day E6DM TOTs were delivered in seven batches with participants from 15 countries80, between June and October 2020. In total, 65 men and 32 women participated in these trainings. Training assessments show positive results, with more than 85% of the respondents rating the course content as good or very good, and as a useful opportunity to progress as an Empretec trainer or trainee trainer. The 21-day TOT, delivered in English, French and Spanish in November 2020, had 35 participants from African countries81 and Romania, and 35 participants from Latin America82. The content was assessed as good or very good by more than 83% of the respondents (16 in Africa and Romania, and 25 in Latin America), and useful for their progression as Empretec facilitators (>91%). Training survey comments highlighted, for instance, the online approach, the follow-up and participant-centered approach, and the possibility of interacting and sharing opinions with colleagues as the most interesting aspects of the training. In sequence, ten 6-day Entrepreneurship Training Workshops (E6DM) were delivered in seven countries83 between September 2021 and July 2022. The total number of participants across countries was around 30084, with about 50% of women (estimated on the data available) and some countries reporting the participation of youth entrepreneurs up to 54% (i.e., Ghana). When post-training assessment is available, results are positive with more than 88% rating the overall workshop as effective/useful/good or very good/excellent85. More than 84% rated the workshop content as good-excellent/relevant. Good-very good improvements in entrepreneurial skills ranged between 43% and 100%, good-very good relevance of the training to run the business was above 80%, and estimated good-very good impact on the future of their businesses was between 33% and 100% in a set of 10 possible impacts. Subsequently, UNCTAD delivered an additional online 1-day E6DM TOTs in September and October 2022, as a pioneer effort to homogenize the application of new versions of the Empretec method to different Empretec Centres and trainers. Participants totalled 109 (64 male, 45 female) from 19 countries86.

80 Argentina, Benin, Cameroon, Ethiopia, Ghana, Jordan, Kenya, Peru, Romania, Russia, Tanzania, The Gambia, South Africa, Venezuela, and Zambia. 81 Benin, Botswana, Ghana, Mauritius, Romania, South Africa and Tanzania. 82 Argentina, Chile, Colombia, Ecuador, Panama, Peru and Venezuela. 83 Benin, Nigeria, Cameroon, Zimbabwe, The Gambia, Ghana, and Malaysia. 84 Note: Some country reports do not inform the total number of participants. 85 Measurement scales could be different between countries. 86 Angola, Argentina, Botswana, Colombia, Ghana, Ecuador, El Salvador, Jordan, Kenya, Malaysia, Mauritius, Mozambique, Namíbia, Panama, Romania, Saudi Arabia, South Africa, Tanzania, and Uganda.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

provided information useful for starting a new business or expanding a business venture

improved my knowledge on entrepreneurship and business development

contributed to identifying new/improved measures to increase business sales

The project...

Do not know Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

42

68. Other innovations to the Empretec programme implemented in 2021 were the development of a version for low-literate people and a full online training. Between April and May 2021, UNCTAD delivered three sessions of the TOTs for low-literate people, in English, French and Spanish, for 62 male and 58 female participants. Trained trainers, then, delivered three 8-day workshops for low-literate people, between March and July 2022, in Benin, Malaysia and Uruguay. The total number of participants was 72, 57% female. The assessment was positive87 in relation to the workshop in general, its content, and its relevance for the participants’ businesses. Regarding the development of the online Empretec training course using gamification and other IT-based tools88, there is evidence of three pilot workshops with entrepreneurs, delivered in hybrid format and in partnership with SEBRAE (Empretec host institution in Brazil) between December 2021 and August 2022. Each pilot tested the effectiveness of the digital tool and led to the identification of, and solutions to, technical issues. Interviews for this independent evaluation conducted with Empretec centers and UNCTAD staff indicated that the performance of Empretecos in the online course was below face-to-face training. An informant said that “workshops are better when people share in the classroom, with safe environment, with facilitator. Online does not facilitate that.” However, even if the online format is not suitable for a workshop that requires six full days of commitment, some procedures, e.g., tracking participants’ progress, and resources, such as web platform for Empretec centers, can be online. Another challenge of delivering the Empretec training online was conciliating the objective of reaching out to vulnerable beneficiaries with no access to the necessary technology. The solution, then, was to keep Empretec workshops offline. Only the TOTs remained online, to ensure there would be trainers when the presential workshops could be resumed.

69. The Surge project delivered a series of webinars on the role of Empretec on enhancing entrepreneurship promotion in post-COVID-19 resurgence. At the global level, it supported the virtual 7th Empretec Global Summit, ‘Boosting Entrepreneurship: The Contribution of Empretec into the Post-Covid-19 Resurgence of the MSME Sector’, in April 2021. The Summit was attended by over 570 participants from 64 countries and was interpreted in seven languages89. “It was also followed by 783 viewers on UNCTAD Facebook Live and almost 2,000 views were recorded on UN WebTV from 94 countries.” Another global event was the Global Entrepreneurship Congress, in Riyadh, in March 2022. UNCTAD contributed to the Ministerial Roundtable on Entrepreneurship. Regional webinars were held face-to-face in Colombia, in November 2021, with Empretec center’s representatives from Colombia, Argentina and Ecuador and entrepreneurs; and with the francophone Empretec centres, with 68 participants from Benin, Cameroon and Mauritius, in May 2022. Country webinars included an online event with NAWEN in Malaysia, in December 2021.

70. To expand knowledge on relevant topics of the modernized version of Empretec, UNCTAD delivered a series of ad-hoc online courses between August and October 2021. They covered business model generation (64 attendants from nine countries90), design thinking (83 participants from 11 countries91), entrepreneurship and marketing (Seychelles, no attendance data available), capacity-building for women in cross border trade (59 women participants in Kenya and Tanzania), and a 4-day TOT on sustainable businesses and SDGs (62 participants from 10 countries92). Post-training survey data (22 respondents) is available for the last course, indicating 90% of participants were satisfied with the training content and 86% agreed that sustainability as per the 2030 Agenda should be included in their Empretec training activities.

71. An impact survey conducted by UNCTAD in March 2023 collected data from a representative sample of Empretecos in Benin, Cameroon, Ghana, Nigeria, The Gambia and Zimbabwe. The Empretec programme was positively assessed by respondents (8.75 points in 10) and was considered better than the average of other business training tools by 54%. Most respondents reported sales growth after the ETW (67%), and 94% believed sales will increase in 2023-24. Among non-entrepreneurs, 60% reported an increase in their income levels after the workshop, 88% felt more motivated to venture after the ETW, 67% foresaw opening a business in the coming three months, and 91% of the Empretecos believed their employability level had increased after the ETW, a trend even stronger among women. Interviews with Empretec centers for this independent

87 100% of good-excellent ratings in Benin, and an average range between 4.5 and 4.7 in a scale of 5 in Uruguay. The training report for Malaysia did not present specific assessment data. 88 LMBInterativa (2022). Empretec Online. Delivery report. 89 Arabic, Chinese, English, French, Portuguese, Russian and Spanish. 90 Ecuador, Ethiopia, Ghana, Jordan, Kenya, South Africa, Tanzania, Zambia and Zimbabwe. 91 Ecuador, Ethiopia, Jordan, Kenya, Malaysia, Romenia, Saudi Arabia, South Africa, Tanzania, Zambia and Zimbabwe. 92 Angola, Argentina, Brazil, Colombia, Dominican Republic, Ecuador, Guatemala, Peru, Uruguay, and Venezuela.

43

evaluation indicated that, in Zimbabwe, for instance, women and youth facing business difficulties were the majority of those trained. Two of these participants are the founders of the Women in Agriculture Union. They learned how to keep records of funds received from members, and the advantages of doing so. Young entrepreneurs in agribusiness improved product processing knowledge, leading to expansion and new markets, in Zimbabwe. In Argentina, course participants prefer the online course format, even when their access to the internet is hard, because of the flexibility of time and schedule and the possibility to connect with people across the country. The number of participants is much higher in the online version in Argentina. In Jordan, the online TOT targeted vulnerabilities, such as internet connection and digital literacy. The online delivery was a challenge also from the Empretec centres’ perspective. More than three hours of online training was not possible; and the programme is based on learning by doing, i.e., it requires observation of behaviours. The new online tools helped to mitigate this challenge, but they did not always work because of poor IT literacy, people’s commitment, and internet connection.

72. The outputs related to the UNCTAD component on women and cross-border trade, were delivered in phases 2 and 3 of the Surge project. Evidence indicates 6-day presential workshops93 delivered in Zambia (36 participants), Malawi (32 participants), Kenya (64 participants) and Tanzania (64 participants), between February and November 2021. They tackled two main critical issues: lack of knowledge about trade rules and customs procedures, and lack of entrepreneurial skills. Workshop beneficiaries were mainly small- scale/informal cross-border traders, especially women, representatives of cross-border trade associations, representatives from the Revenue Authorities, and other authorities based at the borders. Workshop assessments were positive, with more than 45% of the participants rating that their knowledge about trade rules and customs procedures had improved very much or extremely. Most of them acquired confidence in crossing the border through normal channels (>61%) and will be able to formalize within the next 12 months (>54%). Assessment data for Kenya and Tanzania indicated that more than 80% of the participants rated the programme as excellent, the training content as very relevant for running their businesses, and more than 90% indicated the training will help them improve professional behaviour, management skills, business planning skills, establish a network with other business owners, learn about business opportunities and help learn about funding possibilities. The workshop’s methodology, content and results are compiled in the activity report94 commissioned by UNCTAD. Country-specific guides on ‘Women in informal cross-border trade: A small- scale trader’s guide to trade rules and procedures’ developed by UNCTAD provided tailored information for female traders in Kenya, Botswana, Tanzania, Malawi and Zambia. These guides, except for Malawi and Zambia, contain traders’ obligations regarding COVID-19 measures related to crossing the border and a checklist that includes COVID-19 certificates.

73. A 2022 impact assessment on project activities carried out by UNCTAD on women and cross-border trade95 indicated that 100% of the participants became very or extremely familiar with their rights as cross- border traders, 95% became very or extremely familiar with the obligations of cross-border traders, and 43% acquired considerable knowledge on the use of COMESA and EAC STRs and the SADC Trade Protocols. The enhanced knowledge about the use of official border points was very or extremely helpful for 92% of the respondents. “Some traders reported positive results in terms of increased profits, a larger clientele, and the ability to offer a more diversified range of products” associated with the enhanced use of official border points. Almost all participants (96%) reported that they had adapted their business model. Also, 80% of the respondents considered registering their businesses. Responses were positive on acquiring extremely good or very good understanding of the COVID-related rules (65%). Qualitative data indicated that the training helped trainees to be proactive in protective measures, such as in encouraging customers to wear masks and wash their hands. Regarding new strategies, “some traders turned to digital tools to continue their businesses, including Facebook, Instagram, WhatsApp groups, and online platforms.” Interviews with trainers conducted by this independent evaluation indicated that the results of the training for low literacy beneficiaries in cross- border trade were ‘amazing’, based on a follow-up four months after the training. Proper understanding of the trade rules and procedures, rights and obligations of the traders coupled with upgrading of their entrepreneurship skills helped traders to overcome the adverse trade environment caused by the Covid-19

93 They built on training activities completed by UNCTAD in 2019 in the framework of the project ‘Informal cross-border trade for the empowerment of women, economic development and regional integration in Eastern and Southern Africa’. 94 Tigere, F. (December 2021). Phase III Final Report – Global initiative towards post-COVID-19 micro, small and medium enterprises (MSME) sector: Kenya and the Republic of Tanzania. 95 UNCTAD (2022). Cross-border Trade in the Pre- and Post-Pandemic Environment Evidence from Malawi, the United Republic of Tanzania and Zambia. Geneva.

44

pandemic. Some were able to bounce back and build back their businesses. More information on female traders in cross-border trade is in the specific section on gender and LNOB.

74. The delivery of the Farming as Business (FaaB) programme took place in phases 2 and 3 of the Surge project. Adaptation of this programme referred to the online delivery of the TOT, between February and June 2021. Training material does not refer to the pandemic crisis96. The English version of the TOT trained 25 male and 28 female small farmers from 13 countries97. The French version trained 25 male and 28 female participants from Angola, Benin and Cameroon. The Spanish version trained 9 male and 7 female participants from eight countries98. Post-training assessments, available for the English (15 respondents) and French (4 respondents) versions, were positive. On average, 79%, rated the course content as good or very good, 84% found it a useful learning opportunity, and 95% considered the course useful for them to progress as Empretec trainer or trainee trainer. Following the TOT, a grant agreement between Empretec Ghana Foundation and UNCTAD allowed for training of 65 vegetable farmers in Ghana, in November 2021. A report on this training99 indicated that 90% of the participants were male and 10% female. There is no reference to resilience in post-COVID-19 recovery.

75. The Surge project implemented tools that facilitated MSMEs’ registration and formalization (outcome 2) through an e-registration platform, policy toolkits for governments on formalization, and virtual workshops for knowledge dissemination. Under this outcome, UNCTAD delivered the e-registration platform, implemented it initially in El Salvador (cuentamype100), and expanded implementation to Benin, Cameroon and Mali. DESA developed policy toolkits for the governments of Kenya and The Gambia in streamlining MSME formalization and inclusive delivery of formalization services. The dissemination of these toolkits was with virtual workshops in the two countries. Survey evidence from this independent evaluation corroborates the positive assessments observed for individual deliverables, but, because the number of respondents was between 3 and 4, no further conclusions can be drawn from our survey.

76. Interview data indicated that the pilot implementation of Cuentamype in El Salvador involved private and public institutions, and the signing of an inter-institutional agreement, considered one of the biggest project achievements in the country. It brought together, for instance, social security institutions, the Ministry of Finance, and municipalities, and attracted the interest of financial institutions in relation to providing credit to MSMEs. The preparation of the platform included the organization of sectoral working tables with these institutions, and a pilot, in 2020, with a group of 20-25 entrepreneurs for feedback, for around 3-5 months. After adjustments, the platform launched in a forum where entrepreneurs could learn about the tool. The project trained government technicians as trainers for the end-users of the platform – 3-4 training days of about 15 nationwide professionals. During technical visits to MSMEs (e.g., women’s owned beauty salons), specific needs of beneficiaries were identified, such as basic knowledge in accounting and individual monitoring, mostly for women. The answer to these needs included the development of a printed simplified booklet on accounting (i.e., a paper version of cuentamype), follow-up visits for six months, and the creation of a WhatsApp group for direct contact with the registry office (CONAMYPE) to get direct support in interpreting financial analysis outputs and suggest improvements to the platform. Out of about 400 trained entrepreneurs101, CONAMYPE estimates that at least half of them use the tool regularly (about 60% of them are women and covering entrepreneurs from 18 to 70+ years old). The e-registration platform was, then, expanded to Benin, Cameroon (guichet electronique) and Mali (guichet unique) in 2021. The online platform increased MSMEs’ registration 91% in Benin (84% women, 181% youth), and 40% in Mali (49% women, 110% youth). Replicating the case of El Salvador, these e-registration platforms are incorporating additional services, such as modifications in business registry and grant applications. Real-time data generated by these platforms have been used to design better entrepreneurship policies.

96 The trainer’s guides (i.e., coffee and dairy farming), in English and French, were produced before the pandemic and their content was not updated to the new situation. The guide on dairy farming in Spanish was issued in 2021 and reproduces the content of previous guides, also not referring to the COVID-19 pandemic situation. 97 Angola, Benin, Botswana, Ghana, Kenya, Malaysia, Myanmar, Panama, Saudi Arabia, South Africa, Tanzania, Zambia and Zimbabwe. 98 Argentina, Chile, Colombia, Ecuador, El Salvador, Peru, Uruguay and Venezuela. 99 Empretec Ghana (Feb.2021). Report on training of farmers on the UNCTAD Farming as a Business (FaaB) programme - Organized for vegetable growers in the Ayawaso West municipaliy – Greater Accra region. Ghana. 100 https://cuentamype.org/ 101 This included entrepreneurs who could not read or write and were not tech savvy. In these cases, entrepreneurs indicated collaborators to take part in the trainings.

45

77. Policy toolkits issued by DESA considered regional and country-based approaches. A regional toolkit was ‘Best Practices: Formalization of Micro-, Small and Medium-Sized Enterprises (MSMEs) in Africa’102, a document that covers a wide range of best practices associated with formalization (e.g., institutional frameworks, regulatory environment, supply and value chains, access to finance, human capital development, and social dialogue). It assembles data from African countries’ best practices, such as Kenya, Nigeria, Rwanda, South Africa, Tanzania, and Zimbabwe, and benchmarks on examples from countries in other regions, such as India, Sri Lanka, China, Argentina and Colombia. Country-based policy toolkits focused on Kenya and The Gambia and included policy documents and virtual or hybrid workshops for knowledge dissemination. In Kenya, the focus was on leveraging policies governing cooperatives of MSMEs, with the issuing of a report103, a toolkit with policy recommendations for the formalization and growth of MSMEs through a cooperative model104, policy guidelines for the formalization of MSMEs in Kenya, modelled on the guidelines for the formalization of cooperatives105, and the toolkit promotion in a virtual workshop in March 2021. A workshop assessment (responded by 60 participants, 41 male, 19 female from African, Asian and Europe countries106) indicated that 98% of the participants were satisfied or very satisfied with the event, and 100% considered the event relevant to their work. Later, in 2022, DESA supported an assessment of the impact of COVID-19 on MSMEs led by women and youth in Kenya 107 and stakeholder consultations, based on these assessments, in a hybrid workshop. For instance, 100% of the 22 respondents (14 male, 8 female) of the assessment of the workshop on the Assessment of the Covid-19 Impact on Women and Youth-led MSMEs in Nairobi, were satisfied or very satisfied with the workshop, and considered it relevant to their work.

78. MSMEs’ access to finance (outcome 3) improved through online training to policymakers and MSMEs (e.g., financial literacy, accounting, SDG reporting), publications, case studies, training of trainers, and national and regional workshops for knowledge dissemination. Outputs under outcome 3 were carried out by UNCTAD and ESCAP. They produced online training material for policymakers on MSMEs’ access to finance, developed training courses for MSMEs for financial literacy and accounting and reporting, including on the SDGs – supported by a guidance –, TOT’s courses on accounting and reporting for MSMEs, published a book on MSME finance, developed case studies on accounting and reporting for MSMEs, and delivered national and regional workshops based on the online materials produced. Survey evidence from this independent evaluation corroborates the positive assessments observed for individual deliverables, with approval rates above 70% across project dimensions, keeping in mind that the number of respondents is low for robust conclusions and this result needs to be interpreted in qualitative terms (Figure 11).

Figure 11: Effectiveness results of outcome 3 (n=10 respondents).

102 DESA (2021). Best Practices: Formalization of Micro-, Small and Medium-Sized Entreprises (MSMEs) in Africa. 103 DESA (2021). Leveraging policies governing cooperatives to encourage the formalization of micro-, small and mediu- sized enterprises (MSMEs) in Kenya. 104 DESA (2021). Promoting MSME Formalization through the Cooperative Enterprise Model. 105 DESA (2021?). Policy guidelines for the formalization of micro-, small and medium enterprises in Kenya. 106 Kenya, South Africa, Burundi, Tanzania, The Gambia, Lesotho, Morocco, Philippines, Indonesia, Germany. 107 MSEA (2022). Assessment of the impact of COVID-19 among micro, small and medium-sized enterprises in Kenya and their resilience mechanism. Nairobi: Micro and Small Enterprises Authority (MSEA).

46

Source: Independent evaluation survey, 2023.

79. ESCAP led the delivery of a book that set out a post-crisis policy agenda for MSME finance, in 2021108. It addresses MSMEs’ finance gaps and the financial landscape for MSMEs before, during and after the pandemic crisis. The book draws policy recommendations and practical suggestions in MSME finance in relation to, for instance, business services ecosystem, regulatory sandboxes, supply chain and trade finance, financing women-led MSMEs, multi-stakeholder collaboration for mutual benefits, and regulatory framework for fintech. Two thematic studies followed-up, in 2022. One was on the role of banks in Asia in lending to MSMEs109, and the other on the role of digital payments110, including a topic on women-led MSMEs and SDG 5.

80. UNCTAD developed the guidance on core indicators (GCI) ‘Implementation of core SDG indicators for sustainability reporting by companies’ in English, Spanish, French, Arabic, Chinese and Russian, and corresponding training material in partnership with UNITAR. This guidance supported GCI workshops on accounting and reporting for MSMEs in Kenya (23 participants), China (150 participants), Latin America (55 participants, 14 countries), Francophone countries (122 participants, 11 countries) and the MENA (79 participants, 4 countries) and Eurasian (200 participants, 10 countries) regions, between February and November 2021. Across workshop sessions, assessments responded by 246 participants (78 male, 47 female111) indicated that more than 90% of them rated the overall course content as good-excellent/satisfied-very satisfied.

81. MSME’s access to technology and innovation (outcome 4) increased through a combination of survey-based reports assessing the impacts of Covid-19 and the Ukraine crisis on MSMEs at country and regional levels, regional seminars for knowledge dissemination and sensitization of policymakers on needs of MSMEs, online courses on the role of technology and innovation for entrepreneurs and policymakers, including on harnessing green technologies, and knowledge products on a range of areas including building linkages between MSMEs and multinational companies. Activities under outcome 4 were led by ECA, through its Subregional office for Southern Africa, and included surveys circulated to more than 3,000 MSMEs across its 11 member states in Southern Africa, to assess the impacts of the Covid-19 and Ukraine crises on MSMEs112, the development of an online training course for entrepreneurs and policymakers on the role of technology and innovation in addressing this impact (December 2021), an online training course on harnessing green

108 ESCAP (2021). Rethinking MSME Finance in Asia and the Pacific: A Post-Crisis Policy Agenda. Bangkok. 109 ESCAP (2022). Small and Medium-Sized Enterprises Lending Approaches: The Role of Banks in Asia. MSME Financing Series No. 6. Bangkok. 110 ESCAP (2022). MSME Access to Finance: The Role of Digital Payments, MSME Financing Series No.7. Bangkok: United Nations. 111 No gender disaggregated information for the Eurasian workshop. 112 The result was 11 country level reports which, at the time of the evaluation, were being finalized and not yet published.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

provided useful information on financial inclusion and MSMEs access to finance

improved my knowledge on accounting and reporting, including on the SDG reporting, and

improved my capacity to manage financial resources

contributed to identifying new/improved measures for financial inclusion and MSMEs access to finance

The project...

Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

47

technologies to enhance MSME competitiveness in Southern Africa (August 2022), two side events at the margins on the SADC Industrialisation Week on strengthening linkages between African MSMEs and Southern led MNCs and on the role of technology incubators in MSME development in collaboration with the SADC Business Council Southern Africa113. An additional unplanned output, at the request of the Government of Mauritius, was a regional seminar on “The impact of Covid-19 on MSMEs in Southern Africa: Country Experiences and a Focus on Building Back Better in Mauritius” was held at the margins of Dubai Expo 2020, in collaboration with the SADC Business Council and the Ministry of Industrial Development, SMEs and Cooperatives of Mauritius. Survey evidence from this independent evaluation corroborates the positive assessments observed for individual deliverables, with approval rates above 83% across project dimensions, keeping in mind that the number of respondents is low for robust conclusions and this result needs be interpreted in qualitative terms (Figure 12).

Figure 12: Effectiveness results of outcome 4 (n=12 respondents).

Source: Independent evaluation survey, 2023.

82. Evidence from the technical report114 of the 4-week online training on the role of technology and innovation for African businesses in addressing the impact of COVID-19 indicated that 41 participants attended the course. Out of those who responded to the post-training assessment, 95% noted that the course modules were relevant to their professional needs and 90% felt satisfied with the relevance of the topic and the modules. Regarding the course on green technologies for African SMEs, the assessment report indicates that 30% of the 155 registrants completed the course successfully by passing the quizzes. Out of these, 45 responded to the post-course assessment. Results indicated that 93% of respondents rated the quality of the course as good or very good, 93% indicated that the content of the course is relevant to their work, and 100% indicated that they will use the knowledge acquired in this course.

83. MSME’s access to markets (outcome 5) increased through activities that combined the production of national and regional assessments, studies and reports with online events for knowledge dissemination of findings, and online trainings for policymakers and MSMEs at the regional level on topics such as competition policies and access to markets, agricultural quality and food loss, and the promotion of export capacity. Outcome 5 is the second in number of outputs planned (n=25) and the first in collaborative deliveries between UNCTAD and the Regional Commissions. Deliverables cover a wide range of activities on competition policies to regulate access to markets during the pandemic crisis and promote the resurgence of MSMEs in post- pandemic recovery. Survey evidence from this independent evaluation corroborates the positive assessments observed for individual deliverables, with approval rates above 66% across project dimensions, keeping in

113 Updated information, received after the finalization of this independent evaluation, indicated that, as of October 2023, the two publications on strengthening linkages between African SMEs and Southern led MNCs and the role of technology incubators for MSMEs development in Southern Africa are completed and being professionally edited. Two other publications, finalized in 2023, are “Opportunities for MSMEs within the Blue and Green Economy: the case of SADC” and “The role of digitalization in MSMEs development in the context of the AfCFTA”. 114 IDEP, ECA (2022). The role of technology and innovation for African businesses in addressing the impact of COVID-19 – Technical report.

0% 20% 40% 60% 80% 100% 120%

provided useful information on MSMEs access to innovation and technology

improved my knowledge on how to increase MSMEs access to innovation and technology

contributed to identifying new/improved measures to increase MSMEs access to technology

The project...

Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

48

mind that the number of respondents is low for robust conclusions and this result needs to be interpreted in qualitative terms (Figure 13).

Figure 13: Effectiveness results of outcome 5 (n=12-13 respondents).

Source: Independent evaluation survey, 2023.

84. A good illustration of coordinated outputs to enhance access to markets to MSMEs is the joint work produced by UNCTAD’s competition and consumer protection policies branch with ECLAC, ESCAP, ESCWA, ECA and ECE. During 2020, multiple regional preparatory meetings between UNCTAD, the Regional Commissions and regional stakeholders discussed how to mobilize participation and build collaboration in designing and implementing these dialogues to raise awareness of the role of competition policies for post- pandemic access to markets by MSMEs. The resulting five online regional dialogues took place between March and April 2021 and had approximately 650 attendees across regions115. This inter-institutional partnership supported the delivery of a global report on the articulation between competition policy and MSMEs (produced in phase I, between June and September 2021)116. This global report provided a snapshot of the challenges MSMEs faced around the world to deal with the impact of the COVID-19 pandemic and drew key recommendations centered around levelling the playing field for MSMEs. It was disseminated with a global policy dialogue, organized by UNCTAD in partnership with ECLAC, ESCAP and ESCWA, in December 2021 (phase 3). This global dialogue explored how competition policy could support MSMEs post-COVID-19, with the presence of representatives and officials of government MSME bodies and competition agency representatives from across the world. It was divided into three sections, one with a focus on digitalization, another on access to finance and government support, and the third on inter-agency coordination between MSME agencies and competition authorities. It was delivered in English, French, Spanish, Chinese, Russian, and Arabic.

115 The UNCTAD-ESCAP-ASEAN dialogue was delivered in English, Thai, Burmese, Bahasa Indonesian and Bahasa Malay and had 149 participants. The UNCTAD-ECA dialogue was delivered in English and French and had 97 registrants. The UNCTAD-UNECE dialogue was delivered in English and Russian and had 140 attendees, 30% from outside the UNECE region. The UNCTAD-ECLAC dialogue was delivered in English, Spanish and Portuguese and had 115 registrants. The UNCTAD-ESCWA dialogue was delivered in English and Arabic and had 150 attendees. 116 UNCTAD (2021). How COVID-19 affects MSME access to markets and competition: A review of key issues and recommendations for future action. NOTE: the previous title is in a press release by UNCTAD and the next refers to the report shared with the evaluation as output F7. Alternate title and reference: UNCTAD (2022). The COVID-19 pandemic impact on micro, small and medium-sized enterprises. Market access challenges and competition policy. Geneva.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

activities and publications provided useful information on MSMEs access to markets

improved my knowledge on how to increase access to markets, including export and integration into value

chains

activities contributed to identifying new/improved measures on consumer protection and competition

activities contributed to identifying new/improved measures on agricultural quality and food loss reduction

The project...

Do not know Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

49

85. Other cooperation-based and replicated outputs were the three country case studies on the impact of COVID-19 on specific SME sectors in Brazil117, Thailand118 and South Africa119, delivered between March and December 2021. They were implemented with the support of national institutions to ensure a grounded approach in understanding what competition policy is, how it is framed in the country, and how it can impact MSMEs in the context of the COVID-19 crisis. Each case study provided recommendations to competition authorities and representatives and support entities for MSMEs. A global event in June 2021, ‘DA COVID-19 technical cooperation project on Competition Policy & MSMEs’ resurgence’, disseminated the findings of the case studies and promoted policy advocacy on this subject to 238 attendees. Also, five regional webinars with competition authorities and MSMEs’ institutions delivered on how the recommendations of the global and the three national reports could support the resurgence of MSMEs. A follow-up hybrid global workshop, held in Bangkok in 27-28 June 2022, to discuss ‘The Contribution of Competition Policy to the Resurgence of MSMEs post-COVID-19’ had panellists from UNCTAD, ESCAP, DESA, ECLAC, ESCWA and African countries. This event covered recommended practices for digital platforms for MSMEs120 and presented the guidance document for Thai competition and SME institutions121 for 534 virtual participants. It was assessed as relevant (>95% of participants) and effective in enhancing skills, regional cooperation and methods for integrating MSMEs into regional value chains (>95%). This workshop presented the newly designed online course on SMEs and competition policy, primarily for government officials. The country case studies are cited as having inspired the course showing what capacities SMEs needed to engage in better trade practices. This 5-module course was delivered between November and December 2022 to 141 participants (73 male, 68 female) from 41 countries122. The post-course survey indicated that the experience was very/extremely valuable to 98% of the 72 respondents, 81% felt that the course provided them with learning opportunities not available otherwise, 82% felt able to serve the community with what they have learnt, and 68% considered the course relevant to advance in their career.

86. At the level of the indicators of achievement, despite somewhat lacking in specificity, and the high level of ambition and long-term-orientation for a crisis response, considerable progress is observed. The elaboration of the project indicators was reported by informants as one of the challenges in the Surge’s design phase, given the uncertain pandemic context, the lack of clarity on the resources that would be available for each phase of the project, an understanding that the DA requirements for indicators had to reflect development goals (adequate for longer projects123), and time pressure to deliver as soon as possible. As a result, the management decision was to focus on what would be right for MSMEs’ support from each entity’s perspective (i.e., separate streams of work), beneficial for countries, and already available as standard indicators for the entities. Reviews of the initial indicators followed advances in the project and the evolution of the pandemic situation. From phase 2 to phase 3, some indicators were revised to become more ambitious, since there was more time and resources available. According to members of the project coordination team, there was awareness of the broad scope of these indicators, that some of them were not crisis-driven, and it would be difficult to achieve them within a year. Nevertheless, there is also the shared perspective that they are needed to track longer-term effects of the project. A mitigation monitoring strategy by project managers used proxies or partial reporting from specific delivered outputs when short-term impact could not be measured. This is reflected in the Surge project’s final report, concluded in November 2022. Adding to this

117 UNCTAD (2021). Competition and Market Access Policies in the Resurgence of MSMEs in Post-Covid-19 in Brazil. 118 Charoenrat, T., Harvie, C. (2021). Analysis of the impact of COVID-19 on micro, small and medium-sized enterprises (MSMEs) in Thailand from competition policy and market access perspectives. 119 Nsomba, G., Tshabalala, N., Vilakazi, T. (2021). Analysis of the impact of COVID-19 on micro, small and medium-sized enterprises in South Africa. 120 UNCTAD (2022). Harnessing the interaction between digital platforms and MSMEs: Recommended best practices for digital platforms (RDP). 121 UNCTAD (2022). Guidance for action: Supporting MSMEs’ recovery from the COVID-19 pandemic. Prepared for the trade competition commission of Thailand. 122 Kenya, St. Kitts, Botswana, Philippines, Indonesia, El Salvador, Azerbaijan, Trinidad and Tobago, Kuwait, Belize, Brazil, Barbados, Suriname, Eswatini, Burkina Faso, Peru, Malawi. Côte d'Ivoire, Bahrain, Colombia, Russia, Malaysia, Zambia, Ghana, Jamaica, India, Tanzania, Spain, Grenada, Switzerland, Ethiopia, San Marino, South Africa, Nigeria, Senegal, Montserrat, Egypt, Saint Lucia, Zimbabwe, Madagascar, Liberia. 123 It is noteworthy that the DA guidance on the expected outcomes for PRODOCs of joint COVID-19 response projects states that “The outcomes (OCs) describe the changes that are expected to occur as a result of the completion of outputs. The OCs should be achievable within the project’s timeframe and budget and should be specific enough to be measured by the associated indicators of achievement.” Nevertheless, implementers seem to have been driven by an understanding that the PRODOC approval was conditional to a longer-term perspective on outcomes – an understanding that reflected on their indicators too.

50

evidence, this independent evaluation conducted additional desk review with the support of interviewees who shared the results of the latest outputs delivered, and a survey with relevant stakeholders. This evaluation corroborated comments on the project’s final report in relation to finding mismatches between indicators, project phases and some deliverables, as implementing agencies seem to have shifted activities around, repeated the same activities in different outputs or created new ones since the PRODOC. Other activities in the evidence pool were associated with more than one output and are reported here in the output where their content makes more sense. Also, this independent evaluation pooled together evidence from different implementing entities to make an expert judgement of the progress achieved per indicator, not being restricted to the implementing entity associated with specific indicators. This promoted a better global perspective of the results achieved. Importantly, since most quantitative indicators lack a baseline, the baseline used was the number of countries/participants engaged in the activities under that indicator. At a later stage, more evidence on sustainability and impact indicators will be reported in the UNCTAD Division on Investment and Enterprise’s Annual Research and Impact Report. Table 8 summarizes the findings per indicator at the time of this independent evaluation.124 Out of the 26 indicators of achievement, this independent evaluation located evidence of good progress for 11 of them. Some evidence of progress, from some of the implementing entities involved, but not all, or from a limited number of the beneficiary countries/participants engaged in activities under a specific indicator, was found for other 11 indicators. One indicator showed limited progress and activities related to it are continuing in another project. This independent evaluation could not locate evidence for other three indicators. Overall, this evaluation identified levels of progress for 84.6% of the performance indicators. Indicators under outcome 1A are the ones with more progress reported. Outcome 5 is the one with the most uneven performance in terms of evidence of progress identified by this independent evaluation.

Table 8: Level of achievement of the Surge project indicators. Expected Outcomes Indicators

(Important: each indicator was planned to reflect the work of one implementing entity and

was not designed to cover all participating countries)*

SMART* * criteria covered

Level of progress identified

Source of evidence

OC 1A: To improve national capacities on formulating and implementing enabling polices on green, resilient and inclusive entrepreneurship MSME promotion in post COVID- 19 resurgence

IA 1.1. At least 75% of the countries that participated in the project, have adopted and/or implemented a revised/improved policies aimed at building a resilient and competitive MSME sector in post COVID-19 resurgence

MR Significant progress.

DA 2023W Final report, data from UNCTAD, ESCAP and ESCWA

IA 1.2. At least 75% of policymakers that participated in the interventions of this component indicated their enhanced capacity on entrepreneurship/MSMEs policies design and implementation in the context of post COVID-19 recovery

MAR Significant progress.

DA 2023W Final report, data from UNCTAD, ESCAP, ESCWA, survey

IA 1.3 At least 75% of policymakers and other stakeholders of the ESCWA network indicated their improved access to information on best practices on government support measures for post COVID- 19 recovery

SMAR Significant progress.

DA 2023W Final report, data from ESCWA, survey

IA 1.4. At least 75% of policymakers, MSMEs and other key stakeholders indicated their improved access to the ESCWA knowledge hub on issues on the MSME post-COVID-19 resurgence

SMAR Significant progress.

DA 2023W Final report, data from ESCWA

IA 1.5. At least 75% of users of training materials developed under this component indicated their usefulness for their improved capacities in policy- making for enabling MSME environment

MAR Significant progress.

DA 2023W Final report, data from UNCTAD and ESCWA

IA 1.6. Positive feedback of users of publications under this component (assessment reports and other publications)

AR Some progress.

DA 2023W Final report, data from UNCTAD and ESCWA, survey

124 Importantly, the baseline for each indicator was the activity or activities associated with it, rather than the project beneficiaries as a whole.

51

OC 1B: To improve resilience and competitiveness of MSMEs in post-COVID-19 resurgence

IA 1.7. % of Empretec125 trainees expanding or starting a business venture, disaggregated by country and gender, including by people with low literacy

MAR Some progress.

DA 2023W Final report, data from UNCTAD and ECE

IA 1.8. % of Empretec-supported MSMEs able to increase sales in one year, disaggregated by country and gender

MART Some progress. Gender disaggregated data not available

DA 2023W Final report, data from UNCTAD and ECE

IA 1.9. Number of new jobs created by MSMEs which participated in the Empretec programme, disaggregated by country and gender

MAR Some progress. Gender disaggregated data not available

DA 2023W Final report, data from UNCTAD and ECE, interviews

IA.1.10. At least 75% of participants to the entrepreneurship training courses under this component indicated usefulness of the activities for increasing their entrepreneurship and business skills in the related areas by country and gender

MAR Significant progress. Lacking data by gender for some of the training courses.

DA 2023W Final report, data from UNCTAD and ECE, survey

OC2: To facilitate MSME registration and formalization

IA 2.1. Increased number of new businesses registering, disaggregated by country and gender of business owner

R Some progress.

DA 2023W Final report, data from UNCTAD

IA 2.2. Decreased administrative costs to start a business, disaggregated by country

R Some progress.

DA 2023W Final report, data from UNCTAD and interviews

IA 2.3. At least 75% of participants of the workshops on policy guidelines for MSME formalization stated their increased capacity to facilitate MSMEs formalization

MAR Significant progress.

DA 2023W Final report, data from DESA, survey

OC 3: To improve MSMEs access to finance

IA 3.1. Number of participating MSMEs with improved financial literacy and ability to prepare financial statements, disaggregated by country, and gender

MAR Some progress.

DA 2023W Final report, data from UNCTAD and ESCAP, survey

IA 3.2. At least 75% of MSME participants of online training courses state their improved financial literacy in accounting and reporting, including on the SDG reporting, and improved capacity to manage financial resources, by country and gender

MAR Some progress. Gender disaggregated data not available

DA 2023W Final report, data from UNCTAD and ESCAP, survey

IA 3.3. At least 75% of policymakers that participated in the workshop state their increased capacity to facilitate financial inclusion and MSMES access to finance

MAR Significant progress.

DA 2023W Final report, data from UNCTAD and ESCAP

IA 3.4. Positive feedback of users of the publication prepared under this component

AR Significant progress.

DA 2023W Final report, data from UNCTAD, survey

IA 4.1. Number and type of new/improved adopted measures to increase MSMEs access to technology

MR Data not available

-

125 Empretec is a flagship capacity-building programme of UNCTAD for the promotion of entrepreneurship and micro, small and medium sized enterprises (MSMEs) to facilitate sustainable development and inclusive growth.

52

OC4: To increase MSMEs access to Innovation and technology

IA 4.2 Digital community of practice on use of technology and innovation to build resilience to shocks is created with at least 100 members

MAR Limited progress.126

Data from ECA, interviews

IA 4.3. Positive feedback from users of online training materials and assessment report on their usefulness for increased access to innovation and technology

AR Significant progress.

DA 2023W Final report, data from ECA, survey

IA 4.4. At least 75% of participants of training activities under this component indicated usefulness for their increased access to innovation and technology, by country and gender

MAR Significant progress. Lacking data by gender on usefulness for training activities.

DA 2023W Final report, data from UNCTAD, survey

OC5: To enhance MSMEs access to markets

IA 5.1 Number and type of new/improved measures endorsed by policymakers on consumer protection and competition disaggregated by country

MR Data not available127

DA 2023W Final report, data from ECE

IA 5.2 Number and type of adopted new/improved measures on agricultural quality and food loss disaggregated by country

MR Data not available

-

IA 5.3. Increased number of MSMEs in regions targeted by interventions with increased access to markets, including export and integration into value chains, disaggregated by country and gender

R Limited progress. Gender disaggregated data not available

Anecdotal data from UNCTAD

IA 5.4. At least 75% of policymakers that participated in the workshops indicated their increased capacity to improve MSMEs access to markets, including by integration into the value chains

MAR Significant progress.

DA 2023W Final report, data from ECE and ESCAP, survey

IA 5.5. Positive feedback on usefulness of publications under this component (studies, guidelines, videos, recommendation) by users of these publications

AR Some progress.

DA 2023W Final report, data from UNCTAD and ECE, survey

Source: Evaluation team analysis of available evidence. *Given that each indicator refers to the work planned by a specific implementing entity, this independent evaluation was bound by the coverage of activities of the corresponding entity. Nevertheless, whenever possible, evidence from other implementing entities on the same content of the indicator was considered as evidence for that indicator. **Specific, Measurable, Achievable, Relevant, and Time-Bound.

87. The EPF and the cluster-based approach formed an effective framework to organise outputs delivered by different implementing entities under specific outcomes in the Surge Project, and, also, to support the inter-relatedness between outputs whose content cut across different outcomes. Nevertheless, questions remain on the extent to which this approach is suitable for a crisis response (see para. 89). The EPF and the cluster-based approach were introduced in phase 2 of the Surge project as a strategy to provide a global narrative and weave opportunities for collaboration, mostly evidenced by bilateral partnerships between UNCTAD and respective Regional Commissions, and by collaborations between different UNCTAD divisions (e.g., registration and trade). According to members of the project coordination team, UNCTAD already had the EPF as an agreed menu of best practices to support enterprise development, supported by latest resolutions of the UNGA128. Therefore, as the project moved from phase 1 to phases 2 and 3, the EPF components and the cluster-based approach became the design template for the logframe and structured activities around the five outcomes described previously. Activities delivered by different Regional

126 Update received from ECA in feedback to this evaluation report indicates that this work is continuing under the

UNDA13th Tranche project “Innovative approaches for MSME competitiveness to promote trade and inclusive

industrialization in Southern Africa in the Post-Covid context”. 127 There is evidence of national case studies and regional studies collecting measures taken by governments, but these measures do not result from interventions implemented by the project. 128 UNGA resolutions /RES/71/221 and A/RES/73/225.

53

Commissions and UNCTAD’s divisions, then, became conceptually connected, providing the UN with a clear picture of how country-level deliverables fit the global framework. As stated by a member of the PCT, “The original idea was to put together the areas of expertise of Regional Commissions and divisions within UNCTAD under the pillars of the EPF, but each pillar is a different type of product.” It is acknowledged that only UNCTAD has the capacity to deliver a full EPF project at a global level, as this is its flagship intervention model acknowledged in resolutions of the UNGA. Regional Commissions do not have this global mandate on entrepreneurship policy, so the project benefited from their mandates and expertise on MSME development129. Nevertheless, the implementation of the project by the Regional Commissions was constrained by their work programme for the year – previously agreed with governments –, limited staffing resources and online delivery. Figure 14 reflects how the EPF structured the delivery of the project’s outputs by different implementing entities.

Figure 14: Outputs per project outcome and implementing entity.

Source: Final report.

88. An example of delivery that cuts across the EPF pillars is ECE’s series of guidelines and best practices (located under outcome 1.B) that follow a template that covers topics across the spectrum of the EPF: policy, legal and regulations (outcome 1.A), business facilitation and business registration (outcome 2), access to data, information and knowledge, entrepreneurship skills facilitation, access to finance (outcome 3), access to technology (outcome 4), and market access and logistics and supply chains (outcome 5). Other examples indicate the potential for replicability provided by this framework, such as the national entrepreneurship strategies (outcome 1.A), the e-registration platform (outcome 2), and the case studies on competition and access to markets (outcome 5).

89. Interview data with members of the PCT indicates that the Surge’s implementation was consistent with the way UNCTAD implements the EPF, by assessing what exists at country level, making practical considerations on what can be done, and responding to countries’ demands. A risk of this EPF-centred approach in a global crisis response situation is not building enough synergy with partners and facing implementation difficulties imposed by competing local or regional priorities and uneven competencies on MSMEs. Interview findings corroborate this, with informants referring to challenges in balancing regional priorities and competencies with the delivery of outputs and some focal points in the Regional Commissions reporting a lack of knowledge on the role of the EPF in structuring the project. In these cases, project focal points reported the delivery of what was agreed with UNCTAD on specific outputs and outcomes without a coherent conceptual perspective. Some of them felt that decisions on the project were top-down to create a global response, and delivering under the given circumstances was challenging. Informants from different groups of stakeholders shared the perspective that, for a crisis response, most EPF outputs would be low to medium impact, since they follow a long-term development perspective (e.g., the development of national

129 The Subregional office for Southern Africa, for instance, has a mandate to work on MSME development as contained in the outcome statements of its Intergovernmental Committee of Senior Officials and experts Meetings.

0

10

20

30

40

Outcome 1A

Outcome 1B

Outcome 2 Outcome 3

Outcome 4 Outcome 5

Number of outputs per project outcome

UNCTAD DESA UNECA ESCAP UNECE ESCWA ECLAC

54

entrepreneurship strategies). So, in future crises, it would be an advantage to know from the start what EPF and MSME development-related outputs are fit for immediate results and how to implement them accordingly.

How did the response contribute to the participating country Governments’ responses to COVID-19, especially in the area of MSME resurgence?

90. The Surge Project contributed to the participating country governments’ responses to the COVID-19 pandemic crisis by improving policymakers’ capacity to design and implement enabling policies to MSME resiliency and post-COVID-19 resurgence through studies, technical assistance, training, digital tools, and interventions at the regional level. Survey results from this independent evaluation indicated that 71% of respondents agree or strongly agree that the project contributed to formulate and implement enabling polices on green, resilient, and inclusive entrepreneurship for MSME promotion in post COVID-19 resurgence (Figure 15). Given the small response rate, these findings cannot be generalisable to all project participants and represent the perceptions of those who responded to the question.

Figure 15: Survey responses on the project contributions to formulate and implement enabling policies to MSMEs (n=41 respondents).

Source: Independent evaluation survey, 2023.

91. This was achieved through higher level outputs, such as the Entrepreneurship National Strategies resulting from technical assistance of UNCTAD to the governments of Seychelles, South Africa and Uganda (outcome 1.A) and country-specific case studies with tailored recommendations and follow-up workshops for knowledge dissemination in the Asia and Pacific region (outcome 1.B, see Box 1 below) and Latin America, Southeast Asia and Africa (on the role of competition and consumer policies in Brazil, Thailand and South Africa, outcome 5). The production of assessments on the impacts of COVID-19 on MSMEs at the regional level (Southern Africa in outcome 4, multiple studies in Latin America in outcome 5), followed by policy advocacy supporting inter-institutional partnerships between different areas of government to align policies in efficient and effective ways and learning from exchanges with other countries represent a great contribution to institutional strengthening (e.g., Europe, Latin America). Informants referred to previously ‘unthinkable’ dialogues, collaborations, and policy alignment between different ministries, with policies for the MSMEs becoming part of the bigger inter-institutional government agenda. In Latin America, interview data referred to how SME policy capacity was initially poorly coordinated with other productive development policies. During the pandemic, however, the same institutions adopted a different working methodology by coordinating with other ministries, such as health, economy, and other productive ministries. The intention was to make the most of the money and avoid dispersed actions. The modalities of policy implementation were a) localized, with targets per sector, or certain geographical locations, b) made flexible and adaptable on a case-by-case basis, according to set targets and sectors most affected, and c) with support from local governments. This was confirmed by government officials in El Salvador and Peru. In Kenya, the survey results supported government action in facilitating access to finance to MSMEs through, for instance, the creation of a targeted fund to increase the access of MSMEs to digital platforms. Now, the new government is focused on MSMEs and DESA is supporting the design of a strategic plan aligned with that. In Zambia, the Ministry of Small and Medium Enterprises improved the participation of women in business through various cooperatives.

The project contributed to formulate and implement enabling polices on green, resilient and inclusive

entrepreneurship for MSME promotion in post COVID-19 resurgence

0% 10% 20% 30% 40% 50% 60% 70% 80%

Do not know Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

55

Learning from other countries in webinars and regional dialogues is reported as an additional contribution to building and strengthening government capacities to respond to the crisis.

92. In relation to providing innovative tools to government, the development and implementation of specific tools, such as the e-registration platforms (outcome 2) in El Salvador, Benin, Cameroon, and Mali improved the delivery of public services to MSMEs through this digital technology. Real-time data on the number of businesses, categorization by type of business, locations, main business activity, gender, age, etc., from the e-government platforms, is reported to be used in the design of better entrepreneurship policies (outcome 4). The e-registration platform increased the number of MSMEs formalized by about 90% in El Salvador130 and Benin131, and 40% in Mali132, supporting mostly women and youth. Benin became the fastest country in the world to register a business simultaneously with 5 different public agencies (in less than 2 hours). These positive results led to requests from these countries to expand the functionalities of the platform, e.g., by providing analytical financial statements, modifications in the business registry, grant applications to access financial resources (outcome 3), and access to social security benefits for employers and employees. The Prime Minister in Cameroon is signing a new Decree on entrepreneurship making online business registration mandatory.

93. Another type of contribution, regarding capacity-building, refers to the development of the online training course on accounting and reporting on the SDGs (outcome 3), with a specific guidance in French, Russian, Arabic, and Chinese and training workshops for MSMEs in English, Spanish, French, Russian, Arabic and Chinese. This approach expanded the outreach of these courses to over 600 participants in 39 countries, indirectly contributing to countries’ advances on the sustainable development agenda. Having these relevant materials available in more than the three official UN working languages represents an additional contribution to governments, since it may expand opportunities for knowledge uptake through other initiatives in the country.

94. It is noteworthy that contributions to governments often take more time than the project duration, since deeper changes require rounds of consultation and chains of approvals. Hence, challenges to make transformational contributions to governments are often beyond the project’s managerial control, such as government’s capacities, institutional changes in governments due to shifts in power or political priorities (e.g., Georgia, Moldova, The Gambia, Brazil).

Box 1: ECE’s guidelines and best practices – from a general approach to supporting country-specific responses. In phase 1, ECE developed two general guidelines and best practices for MSMEs. One of them was on circular economy and sustainable resource management and critical raw material supply chain solutions133 and the other on delivering energy-efficient products and in providing renewable energy equipment134. They refer to the UNFC and UNRMS as a global framework that can help MSMEs to build innovative business models and gain from the new opportunities created by the pandemic crisis. Under outcome 1.B, this expanded to cover a wide set of studies and interventions aimed at supporting governments to develop an enabling environment for MSMEs (outcome 1.A) in the areas of circular economy and critical raw materials, and energy efficiency and renewable energy. Overall, these country-specific guidelines replicate the same structure of the original documents, allowing for country-specific analysis and recommendations, while providing information comparable between countries and sub-regions within the ECE region. They cover opportunities for entrepreneurs, business development managers, technical experts in existing and potential MSMEs, and financiers on the following topics: business facilitation and business registration (outcome 2), policy, legal and regulations (outcome 1.A), access to data, information and knowledge, entrepreneurship skills facilitation (outcome 1.B), access to finance (outcome 3), access to technology (including digitalization, outcome 4), market access, and logistics and supply chains (outcome 5). Recommendations to governments include providing immediate financial support to MSMEs in the event of a crisis (e.g., temporary tax waivers, temporary tax breaks, national financial programs to support MSMEs, offer payment delays, wage subsidies, line of credit and guarantee free loans to MSME), supporting MSMEs in networking and information gathering, and adopting policies to avoid workforce layoffs (e.g., offer employee development

130 In El Salvador, 12,000 new businesses formalized (40% women-led). 131 In Benin, 52,000 businesses formalized between 2019 and 2021, mostly by young entrepreneurs. 132 In Mali, business creations increased from 6,087 in 2019 to 14,796 MSMEs in 2021. 133 UNECE (2020). Guidelines and Best Practices for MSMEs to assure resiliency and progress towards a circular economy in sustainable resource management and critical raw material supply chain solutions. August 2020. 134 UNECE (2020). Guidelines and best practices for micro-, small and medium enterprises in delivering energy-efficient products and in providing renewable energy equipment.

56

programs, such as in digitalization, wage subsidies to cover temporal downsizing and financial support to laid-off workforce). Following the OECD strategic directions for SME development, these recommendations include the development and promotion of 1) an institutional framework and operational environment (e.g., coordination mechanisms for SME policy, collect relevant SME statistics and disseminate the results, improve business closure and insolvency procedures, use targeted public procurement for MSMEs, and a specialized clean energy financial institution to enable MSMEs as a driving force to clean and green economic recovery); 2) access to finance (e.g., public grants, especially for MSMEs working on low-carbon technologies, supply-side financial skills of banks, demand-side financial education for entrepreneurs, credit guarantee scheme, other non-bank financing for SMEs, with a clear governmental guidance); 3) skills and entrepreneurial culture (e.g., training needs assessments, implementation of vocation training policies, access to non-formal training, women’s entrepreneurship); 4) internationalization (e.g., export promotion activities and export-related skills, SMEs to adapt to DCFTA requirements, financial support to exporting SMEs, and SME integration into global value chains); and 5) innovation and R&D (e.g., legal framework for innovation, collaboration between industry and academia, and financial support to SMEs for innovation). Country-specific guidelines on delivering energy-efficient products and in providing renewable energy equipment were produced for Georgia and North Macedonia in phase 2, and expanded to Armenia Albania, Kyrgyzstan, and Republic of Moldova in phase 3. Likewise, customized guidelines on circular economy in sustainable resource management and critical raw material supply chain solutions were produced for Tajikistan and Ukraine in phase 2 and expanded to Bosnia and Herzegovina, Kyrgyzstan, Kazakhstan, and Serbia, in phase 3. Once the guidelines for a country were done, there were regular contacts with the government’s agency responsible for that piece, for dissemination and organization of the online training. These were about half-a-day training sessions, sometimes with presentations from the WB, or UNDP. There were a couple of events by the end of the project delivered face- to-face. Also in phase 3, an updated version of the guidelines and best practices for MSMEs in delivering energy-efficient products and renewable energy was delivered. This update responded to changes in the work environment of MSMEs in this sector in the ECE region, where countries laid out economic recovery through ‘green’ approaches. A follow-up on the original guidelines’ recommendations indicated that “Most of the countries in the ECE region have adopted various economic and operational tools for recovery from the COVID-19 crisis, helping MSMEs in the energy efficiency and renewable energy equipment sector to boost their growth and to bring the clean energy transition across the region. Intervention measures like low-interest financing, innovation grants and funds, operational relief funds, provision of subsidized/common manufacturing units and workplaces, etc. are being issued for supporting MSMEs during the COVID-19 crisis.” The assisted countries can now request technical assistance from the regular programme on technical cooperation to implement the recommendations in these guidelines.

95. The absence of strong policymakers’ networks on policies for MSMEs and of mainstreaming gender- and minority-inclusive approaches hindered the effectiveness of the Surge project in supporting governments’ responses to assist MSMEs recover from the COVID-19 crisis. The two initiatives on building policymakers’ networks, one by ESCAP and another by ECA, showed high relevance and appreciation by country stakeholders interviewed. In the case of ESCAP, after three meetings and a set of associated outputs, such as input studies and follow-up meeting reports, interest on the topic of MSMEs faded away and this network seems to be inactive now. In relation to ECA, the plan was to deliver a ‘Digital community of practice’ (Techni Africa portal) on technology and innovation for SMEs in Southern Africa. Progress was made in developing and validating the digital platform (TechniAfrica) that will host the community of practice, but it had not been operationalized by the closing of the Surge project.135 Another tool with potential to support strategic networks of policymakers is the DEPAR portal. Although it has the digital structure and input materials in place (e.g., the ecosystem maps), there seems to be a lack of institutional effort to build and maintain an active network, as a community of practice. Networks of policymakers could sustain mutual and regional learning on policies to support MSMEs in different sectors, including in cross-border trade.

96. The project could have improved its support to government capacities in cross-border trade by involving government officials, i.e., immigration officers, customs and police with crucial roles on CBT, in the training courses, as trainees, or having a dedicated training session for them. Interviewees referred to their lack of knowledge on CBT and business skills and negative attitudes toward cross-border traders as remaining barriers to facilitating cross-border trade, especially for women. This independent evaluation also found that

135 Update on the operationalization and launch of this platform was provided by ECA in feedback to this evaluation report. The actual operationalization and launch of the digital platform, titled “TechniAfrica”, to be hosted by the SADC Business Council, will be implemented under the UNDA13th tranche project “Innovative approaches for MSME competitiveness to promote trade and inclusive industrialization in Southern Africa in the Post-Covid context”.

57

some sectoral interventions on access to markets were gender-blind by design and they benefited women because a large proportion of workers in those sectors are female (e.g., cocoa and coffee in Peru). Informants referred to a lack of government interest in looking at gender issues amid a pandemic crisis, so implementing agencies, in those cases, focused on their priorities.

What innovative approaches or tools, if any, did the response use, and what were the outcomes and lessons learned from their application?

97. One of the key institutional innovations in the Surge project was the use of adaptive management through an increased consultative process, and a phased approach. The COVID-19 crisis was an opportunity for the DA programme to innovate on how it promotes sustainable development. Its orientation remains geared to the long run rather than to crisis responses, but the efforts to make the Surge project and the other COVID-19 response projects work created momentum to stimulate more joint projects, according to interview informants. They refer to the advantages of learning what other regions are doing and the possibilities to connect through digital technologies, reducing the costs of inviting someone from Asia to an event in Latin America, for instance. Informants also acknowledge the constraints to implementing such collaborations, such as distinct programmes of work in different entities, translation costs, time zone challenges, etc., but feel that the project promoted the interest and motivation for more synergies. The emerging benefits of, and new expertise, on joint projects through frequent consultation with DA focal points and relevant stakeholders (e.g., entities’ staff working on MSMEs) made this modality a good practice to be considered when planning regular DA projects that could benefit from joint implementation. The streamlining of guidelines136 to support the phased approach associated with the Surge project is also an innovative practice. The phased approach was qualified by several UN staff in the implementing entities as ‘unique’ in DA history and seen as one of the main innovative approaches at the institutional level.

98. The Surge Project developed innovative approaches and tools to respond to country needs, such as online delivery of trainings and knowledge-sharing events, the introduction of MSMEs into policy discussions in response to the pandemic crisis, and e-government services. Some of these innovative approaches and tools are being incorporated into the daily work of stakeholders. The most highlighted innovation by relevant stakeholders was the conversion of activities into the online format. Some advantages of this format were the massive extended access to training, inclusive during lockdowns (e.g., e-learning EPF TOT courses delivered, outcome 1.A, Empretec TOT courses, outcome 1.B, courses on role of technology and innovation and green technologies for MSMEs, outcome 4, course on competition policy for MSME resurgence, outcome 5), the opening of opportunities for exchanges between implementing entities (e.g., joint events and participation in each other’s events, all outcomes) and between policymakers (e.g., regional webinars, global conferences, all outcomes), and to bring together multiple stakeholders for wider discussions (e.g., webinars on the health economy in Africa, Asia and Latin America, outcome 1.A). At the content level, an innovative approach praised by different stakeholders was the focus on MSMEs in the context of competition policies (Latin America) and the inclusion of MSMEs in the existing circular economy and resources management tools (Europe) to answer to a global pandemic crisis. This approach had not been thought about and it was transformative to see it associated with the rapid digitalization of services, such as delivery services, and in policy guidelines on circular economy (outcomes 1.B and 5). This brought MSMEs to the mainstream debate on digitalization and competition policies and into ECE’s work on energy efficiency. Innovative online tools that generated rapid results were the e-government and e-registration tools (outcome 2), since they kept public services working during lockdowns and increased the interaction between MSMEs and public services in a digital one-stop shop that can be accessed by computer or mobile phone and saves on paper for accounting books. An additional gain is improving the digital literacy of users.

99. Respondents to the survey conducted by this independent evaluation listed other innovative or unique approaches and tools of the Surge project that they thought the UN should consider replicating or upscaling, with 57 contributions from 48 respondents. Here is a summary of these responses, categorized according to main themes:

136 The timeline for concept note and PRODOC development was shortened, and the concept note guidelines were

shorter than those for regular tranche projects (as the concept note was only subject to approval by the DA Steering

Committee, and not the General Assembly). The PRODOC template still included all the elements of the regular

tranche prodoc, and there was an additional Phase 3 budget proposal.

58

● Leaving no one behind (15 comments): reaching out to resource-poor areas in developing countries and seeking cooperation to do so and following up with them; business that clearly accommodates those with disabilities; supporting the local women and youth-led initiatives to increase accessibility of services by women and girls in hard-to-reach communities (e.g., rural areas); promoting and upscaling global entrepreneurship with e-commerce, especially for marginalized and vulnerable groups; considering the triple impact (economic, social and environmental) of women-led businesses; profiling women-led MSMEs was insightful and helped to develop legal amendments supportive to women employability.

“I live in a rural and poorest province in my country. I was impressed by the reach of the project because while my country (the capital city) is hosting many UN regional offices, UN programmes are unheard of in these parts, though I suspect they are visible elsewhere in the country and region.”

● Capacity-building for MSMEs, entrepreneurs and communities (12 comments): capacitating communities to engage in entrepreneurship; financial literacy training; entrepreneurial training (including on appropriate technology application and innovation); diversification of business opportunity innovations and mentoring of entrepreneurs.

● Scope of the project and mode of delivery (9 comments): online activities (i.e., courses, webinars and information dissemination); large-scale multifaceted project allowing for rapid intervention; the combination of independent study, YouTube videos, exercises, moderated online participation and tests facilitated greater engagement with the material, learning from the facilitator, and peer learning; room for project managers to try out new ideas, to innovate; undertaking a needs analysis and developing programs based on the needs of the MSMEs.

● Access to finance (5 comments): a simplified micro-lending business method, such as funding local small businesses with a day-to-day transaction of lending to collect daily loan offers with affordable interest rates; innovative digital finance solutions and regulatory frameworks to support both financial inclusion and growth of SMEs, especially those owned and led by women (new perspective for the private sector); increasing non-refundable investment support for SMEs in developing and underdeveloped countries; supporting MSEs to access grants; development of investment attraction mechanisms for the energy system.

● Institutional capacity-building and national capacities for policymaking (5 comments): opportunity or skills-sharing from neighbouring countries and building business relationships and Empretec relationships; face-to-face courses for policymakers to support SMEs; improving cooperation between countries; hybrid policy dialogues including the private sector; improving private sector governance to make it an engine of growth.

● Access to markets (4 comments): digitalization in a globalized world; reducing border barriers; the role of competition policy in the economic recovery of MSMEs (e.g., MSMEs continue to face challenges related to their interaction with bigger counterparts in the agricultural sector and current competition laws do not present effective solutions); the work dedicated to the analysis of structural barriers to trade helped to identify measures necessary for facilitation of exports in Africa.

● Guidance for crisis response (4 comments): recommendations to respond economically to a crisis; emphasis on sustainable and inclusive development; studies might contain innovative and/or unique conclusions and recommendations (should upscale and replicate); the whole approach was innovative, enabling one-stop shop for mechanisms and best practices towards post-COVID-19 resurgence of the MSME sector.

● Access to technology and innovation (2 comments): unique in tackling the novelty of COVID-19 for future resilience (upon lessons learned); the project's efforts to leverage technology and digital solutions for MSMEs are forward-thinking.

● Better policies and government strategies (1 comment): improving EE/RES (energy efficient/renewable energy) secondary legislation.

100. Challenges to the institutionalization of some of these innovations include the digital gap and the lack of technological infrastructure and literacy. Two examples illustrate these challenges. First, the pilot tests of the online version of the Empretec training indicated that, even if online training had worked well in training of

59

trainers’ courses, this is not the case for entrepreneurs. Empretec is a lengthy training programme that includes the observation of behaviour and the performance of team tasks, and the online format does not favour these characteristics of the program. Moreover, the course’s online games and activities would not be supported by the bandwidth and the computers the entrepreneurs have access to. Second, the implementation of the e- registration platform in El Salvador required many intermediary – presential and paper-based steps – to include female entrepreneurs of low literacy and technological skills. Implementers developed a paper version of the platform, where entrepreneurs could write down their financial information until they developed the financial and technological skills needed to use the platform. They needed a 6-month follow-up with visits by the implementers and, during training, they would require their help and the help of younger entrepreneurs to, for instance, create an e-mail account and a password and access it afterwards. Another challenge, at the implementing entities’ level, is access to digital platforms for large conferences and meetings. ECE, for instance, depended on UNDP to set up Zoom meetings with interpretation or had to hire interpreters, since the tools available to them would not offer this service.

5.4 Sustainability

What measures were adopted to ensure that outcomes of the response would continue after the project ended?

101. From the perspective of the implementing entities, the Surge project sowed the seeds for joint projects that are being planned or may emerge in the future. Lessons learned from the responses to the COVID-19 crisis seem to be part of a process of internal reflection by the DA-PMT to support better the design and evaluation of joint projects. The lessons learned from the 5 COVID-19 response projects will be reviewed and discussed at a meeting of the DA Network, as soon as all of these project evaluations are completed. Meanwhile, the management response to the final evaluation of the DA T10 Programme on Statistics and Data137, which report138 also took into consideration the impacts of the COVID-19 pandemic crisis, addresses actions to be taken in the course of 2023 and 2024 to improve and develop guidelines on planning, monitoring, and evaluation of joint projects, especially those with budgets above USD 1 million. In June 2023, DA-PMT expressed its intent through interviews to do more joint projects with a long-term view and responding to the evolving world. Since funds are non-earmarked, they can respond to the mandate of UN entities, including by operationalizing the HQ’s analytical work by cutting silos. Joint projects currently in the design phase (16th tranche) refer to the triple crisis – food, water, energy – and the food crisis response for Ukraine. The former has a global scope, covering urban resilience, energy security, financing etc., with a regional perspective (e.g., the climate crisis plays out differently in Europe and in Africa). DA-FPs interviewed acknowledge the uptake of learning from the COVID-19 response in the DA call for crisis-response projects. They refer to faster procedures, with an improved system of calls for proposals supporting the development of concept notes, project documents, and improved templates that are simpler and more concise. Reported improvements include clearer and more specific feedback of the DA-PMT to proposals, e.g., details on indicators, countries, logframe, including suggestions for aggregated activities. At the regional level, informants referred to how partnerships between Regional Commissions in implementing the Surge project have promoted learning about what others are doing and raised interest in seeking further partnerships. Some of the key factors that might limit or facilitate such joint initiatives, under regular project planning conditions, are the availability of resources, the Regional Commissions’ mandates, and their priorities in a moment in time. Future evaluations of joint projects that are planned under such regular conditions (i.e., not called for to a swift response to a major unpredictable global crisis) might benefit from considering evidence on these and other factors to demonstrate the added value of joint projects under more predictable global and regional contexts.

102. There is evidence of the sustainability of e-government tools, training activities and regional and national interventions, with measures related to expanding the functionalities of digital tools, training courses and the formalization of agreements between partner institutions. Across regions, e-registration platforms are being increasingly used by entrepreneurs, mostly youth, with their potential being acknowledged by MSMEs, governments and other relevant stakeholders, such as banks. The result is new requests from different countries to expand the functionalities of these digital platforms and the higher probability that they will remain active in the future. In Cameroon, for instance, the government is signing a decree to make e-registration

137 https://www.un.org/development/desa/da/wp- content/uploads/sites/52/2023/05/MR_to_Final_Evaluation_Report_1617A.pdf 138 https://www.un.org/development/desa/da/wp-content/uploads/sites/52/2023/01/DAT10-Programme-on- Statistics-and-Data-Final-Eval-Report.pdf

60

mandatory. Surge’s managers referred to the e-government platforms as a tool that can be available in crisis response to keep government offices operational. However, there is awareness that these platforms may not respond to all types of crisis.

103. In Southern Africa, the Surge project strengthened relationships between the ECA’s Sub-regional Office for Southern Africa and the SADC Business Council, with joint meetings on MSMEs being held to this day139. Recommendations from the side events held at the SADC Industrialisation Week 2021 and funded under the Surge project were incorporated into the key priority areas for stakeholders, including member states, in the Lilongwe Declaration, issued at the 5th Annual SADC Industrialisation Week (November 2021), under the section on SME and local development, gender, and youth.

104. The project raised the opportunity to pilot an UNCTAD/Empretec Farming as a Business training with small-holder vegetable farmers in Greater Accra, Ghana. However, despite the pilot’s success in empowering farmers who embraced the improved practices with measured increased yield and income, the sustainability of this pilot is not ensured, since its small scale may prove challenging in sustaining access to markets and pro-poor initiatives. In Nigeria, the Entrepreneurship Training Workshop has been recommended by former participants from the government, bringing in more government agencies and state governments to the program, thus enhancing its sustainability.

105. In Latin America, sustainability is rooted in strengthened links between government officials and their stakeholders, such as MSMEs, business associations, and training centers. Based on the courses of the Surge project, ECLAC has been supporting subregional and national initiatives for training MSMEs, using the tools generated to help and promote greater incorporation and formalization of MSMEs in the region, with special emphasis on those led by vulnerable groups such as women and youth. This follow-up work involves different divisions at ECLAC, such as international trade, social development and gender. Also, as a follow-up to the online generic courses on cross-border trade, ECLAC received sub-regional and country requests for tailor- made courses. This generated a customized course for Central America, organized in collaboration with the Regional Center for the Promotion of MSMEs (Cenpromype), on “the "Development of strategic capacities on public policies for electronic commerce" (2022140). This was followed by the planning of a Portuguese version of the course for Brazil.

106. In Thailand, after the end of the Surge project, the SME authority (OSMEP) and the Competition authority (OTCC) signed an agreement in August 2023 that incorporates SMEs into competition policies. This is evidence of the implementation of recommendations from the national case study discussed in the ‘Formal consultation between UNCTAD and Thai institutions: OTCC, NESDC, OSMEP, ISMED and SME Development Bank under the UN COVID19 Project: SME resurgence’ in which representatives of these two institutions were present.

107. The beneficiaries’ perspective on the potential impact and sustainability of results at the MSME level of the Surge project indicates potential long-lasting effects on resilience and competitiveness in post-COVID- 19 resurgence, through high-level achievements in each project outcome. MSMEs and government officials report on implementing activities learned during capacity-building, advantages of the new registration facilities, improved management practices, access to knowledge, empowerment of women and youth entrepreneurs, preparedness to respond to crises through innovation, access to finance and markets, and improved capacities of policymakers. Across the board, 73% of 106 survey141 respondents agree or strongly agree that the project contributed to improving the resilience and competitiveness of MSMEs in post- COVID- 19 resurgence (Figure 16). Given the small response rate, these findings cannot be generalisable to all project participants and represent the perceptions of those who responded to the question.

Figure 16: Contribution of the project to improve the resilience and competitiveness of MSMEs in post-. COVID-19 resurgence in a sustainable way (n=106 respondents). Source: Independent evaluation survey, 2023.

139 The latest one was a 3-day regional meeting, held in June 2023. ECA partners with SADC Business Council on technology and innovation for MSMEs in Southern Africa | United Nations Economic Commission for Africa (uneca.org) 140 136 participants mainly from Belize, Costa Rica, Dominican Republic, El Salvador, Guatemala, Honduras, and Panamá. 141 Survey conducted by this independent evaluation in July-August 2023, responded by responded by government, MSMEs, and other private sector-related stakeholders (e.g., business associations, commercial registry offices).

61

108. Most of these respondents also agree or strongly agree that the project impacted on better outcomes for women and youth entrepreneurs and other vulnerable and marginalized groups (64%) (Figure 17). Given the small response rate, these findings cannot be generalisable to all project participants and represent the perceptions of those who responded to the question.

Figure 17: Contribution of the project to better outcomes for women and youth entrepreneurs and other vulnerable groups in a sustainable way (n=105 respondents).

Source: Independent evaluation survey, 2023.

109. Survey results confirm the predominance of positive assessments of the potential impact of the project in each outcome. Because of the low number of responses in outcome 2 (between 2 and 3 respondents), outcome 3 (9 respondents), outcome 4 (11 respondents) and outcome 5 (between 8 and 9 respondents) to the questions on the potential impact of the Surge project, these results are considered as qualitative information that reflects the assessments of those who responded and do not represent a robust finding in quantitative terms.

110. Figure 18 presents the survey results for the impact of outcome 1.A, where 71% of respondents agree or strongly agree that the project contributed to the formulation of better policies for MSMEs.

Figure 18: Contribution of the project to formulate and implement enabling policies for MSME post-COVID- 19 resurgence in a sustainable way (n=41 respondents).

0% 10% 20% 30% 40% 50% 60% 70% 80%

The project contributed to improve the resilience and competitiveness of MSMEs in post COVID-19 resurgence

Do not know Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

0% 10% 20% 30% 40% 50% 60% 70%

The project contributed to better outcomes for women and youth entrepreneurs and other vulnerable and

marginalized groups

Do not know Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

62

Source: Independent evaluation survey, 2023.

111. Figure 19 shows moderate impact of the project on creation of new jobs, increases in sales, and creation of new or expansion of existing business ventures, with between 42% and 57% of the survey respondents agreeing or strongly agreeing with the survey statements (outcome 1.B).

Figure 19: Contribution of the project to improve the resilience and competitiveness of MSMEs in post-COVID- 19 resurgence in a sustainable way (n=33 respondents).

Source: Independent evaluation survey, 2023.

112. Qualitative survey responses regarding the potential impact and long-lasting effects of the project for beneficiaries were mostly about gains to MSMEs’ resilience. Out of 47 comments, 25 referred to impacts for MSMEs. They cover multiple ways in which the project supported MSME’s survival and resilience. The main service of the Surge project was capacity-building for entrepreneurs, such as in relation to registration, internal management practices, good guidelines towards circular economy, knowledge sharing about critical issues (e.g., market dynamics and export opportunities and barriers), empowerment of women (incl. women with disabilities) and youth to do business, access to support mechanisms, and contribution to community development (incl. on green technology), and preparedness to respond to crises through innovation and reaching out to consumers. Regarding MSMEs’ operation, comments refer to increased access to finance and support programs, economic improvement through income generation activities, methods to improve access to markets and supply chains through innovation, facilitation of business operation through e-commerce and digitalization, and contribution to increase the number of MSMEs. The 16 comments on the results of the project that were beneficial to governments covered policy advice on national MSME policies (e.g., e- commerce, law enforcement, sustainable development, gender-responsive support, MSME formalization,

The project contributed to formulate and implement enabling polices on green, resilient and inclusive

entrepreneurship for MSME promotion in post COVID-19 resurgence

0% 10% 20% 30% 40% 50% 60% 70% 80%

Do not know Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

0% 10% 20% 30% 40% 50% 60%

contributed to expanding my business venture or to start a new business

supported my business increasing sales

contributed to the creation of new jobs

The project...

Do not know Disagree/Strongly disagree Somewhat disagree Somewhat agree Agree/Strongly agree

63

new support schemes for MSMEs), e-registration, stimulus to public-private partnerships, awareness of the leveraging power of ICTs for MSMEs, awareness of the role of rural areas for development, and grounded work through project’s partnerships with local institutions.

113. Another source of evidence for sustained results was the survey responses on the ways in which beneficiaries use the knowledge or skills acquired through the project. Three main categories of knowledge/skills use emerged out of 71 comments:

● Use of knowledge or skills to improve MSME policy (22 comments): project beneficiaries reported that they had been raising MSMEs’ issues in new projects to respond to the COVID-19 crisis (e.g., introduction of entrepreneurial training, business counselling and access to credit), improving policies for youth employment and community development, incorporating MSMEs’ challenges in accessing markets and finance into work in competition law, mainstreaming sustainability principles in MSMEs’ development initiatives, supervising partnerships between MSMEs and larger companies, onboarding MSMEs to the e-commerce platform, easily assessing women’s problems in farms, establishing priority policies for women-owned, female-intensive businesses and social enterprises, using analytical tools (E-view, SPSS, GIS) to identify policy trends and priorities for MSMEs, sharing best practices within the organization and with other partners (e.g., the critical role of cooperatives in formalization). An unexpected impact was on increased exchanges between policymakers and the UN through presentations of the work they have done, and another was on changing cultural relationships with the environment:

The ocean economy is not very well explored in my area. The ethnic groups have always avoided water out of cultural and spiritual reverence. My development programmes never included the ocean economy until after this project. I have started an awareness initiative that presents the ocean as a possible and lucrative source of livelihood. I believe the initiative is making a breakthrough because locals (though still very few) are beginning to lodge applications for fishing licenses to the authorities.

● Use of knowledge or skills to strengthen MSME resilience (24 comments): project beneficiaries

reported that they have been applying their new knowledge in supporting and mentoring MSMEs in topics such as financial literacy, sustainability and ecological concerns, product exportation, and diversification of business opportunities. These consulting services have been assisting family enterprises too. MSMEs reported that they have been applying the new knowledge in their business growth plans and accounting systems, in designing and upscaling projects within the circular economy and the energy sector, creating an association to buy larger volumes from suppliers, applying entrepreneurship competencies to avoid fraud, diversifying production with facial masks for COVID-19, improving production by reducing pollution, implementing biosecurity protocols, and using digital technologies to reach out to prospective clients, do business marketing, expand collaboration with partners to participate in tenders and expand the business activities. Two unexpected impacts were a beneficiary who started an NGO to focus on climate change, and another who installed wi-fi internet in his village using solar panels to ensure that they could continue working remotely.

● Use of knowledge to take training forward (25 comments): beneficiaries reported that they are taking training forward either by taking further training (e.g., on green technologies, computer training) or by training others (e.g., teaching at vocational courses, designing new courses for MSMEs about internationalization and digitalization, and sustainable development, developing new training material for national counterparts, business development services’ offers). They had also been using what they learned to explain to work colleagues how to operate more effectively and efficiently, to do female entrepreneurship advocacy to other women entrepreneurs, and to protect their families from COVID- 19.

114. Challenges for beneficiaries to use their new knowledge or skills refer, mostly, to lack of financial resources, follow-up mentoring, and a favourable business environment. Fifty-three survey respondents indicated their difficulties in making use of the knowledge or skills acquired through the Surge project. The main difficulties were around unfavourable business environment for MSMEs in the country or weak business ecosystems, lack of local resources (policies, technologies, access to remote regions), financial constraints

64

(including barriers to access financial resources by entrepreneurs and governments), turnover of government officials, low political interest and commitment from senior management, limited technical skills and resources, lack of country data for effective policy analysis, lack of education for sustainable development, lack of entrepreneurial mindset in the community and among small business owners, lack of skilled workers, lack of post-training follow-up and mentoring, how to transfer knowledge to complex country situations, challenges to coordination and synergies between relevant agencies, and internet costs.

115. Beneficiaries’ suggestions for ways in which the UN could support MSMEs in their countries covered continuing to invest in capacity-building activities, facilitating access to finance, supporting inter-institutional coordination inside the country’s government bodies, and increasing interventions at the local level. Survey respondents suggested numerous ways in which the UN could support MSMEs in their countries (72 comments). They can be grouped into the following main categories: continue developing capacities of all actors (i.e., governments, MSMEs, other national partners, such as chambers of commerce and business development services) through training, mentoring and webinars, strengthen or contribute to facilitating access to finance for the MSMEs (including seed money, grants, access to credit, etc.), support improving the coordination across ministries of MSMEs-related policies, and increase interventions at local level (i.e., supporting NGOs and MSMEs outside of the main cities, supporting the organization of cooperatives, launching communication campaigns for information dissemination), conducting more regional and country-specific studies (e.g., policy papers, sectoral approaches), and provide tools to support internationalization of MSMES (e.g., network program). There is great emphasis, both in terms of number of comments as well as in relevance for inclusive and sustainable development, on the inclusion of women, youth, persons with disabilities and rural entrepreneurs across these suggestions.

5.5 Gender, Human Rights, and Leave No One Behind

116. This section explores the extent to which gender equality, human rights and leaving no one behind perspectives were considered in the design and implementation of the project.

To what extent were perspectives on gender equality, protection of human rights and reaching underserved groups integrated into design and implementation of the project? What results can be identified from these actions?

117. The Surge Project´s design was guided by two key UN documents that provided clear direction for the integration of gender, human rights, and LNOB: The UN Secretary-General’s Shared Responsibility, Global Solidarity report and the UN framework for the immediate socio-economic response to COVID-19. The Global Solidarity report 142 provided overall guidance by recognizing that the most vulnerable were the hardest hit by the pandemic, stating the UN’s commitment to help people and societies, especially the most vulnerable, and emphasizing the need for full respect of human rights, ensuring no one is left behind. Aiding in implementation of the Global Solidarity’s guidance, the Framework143 outlined detailed responses to the pandemic with specific deliverables and activities related to gender dimensions, human rights and Leave No One Behind (LNOB). Particularly, the relevant responses including the assessments were expected to be gender- responsive and involve a human rights-based approach to data collection with disaggregated data such as age, sex, migratory status, health status, socio-economic status, place of residence and other factors. The analysis of the human rights and gender impacts would then inform the design of policies that address these risks considering gender aspects and disability-inclusive responses. The Framework also provided a set of indicators to monitor and assess the human rights implications of the COVID-19 crisis, including socio- economic impacts and LNOB aspects. In addition, the Guidelines for the Preparation of Project Document for the Development Account144 required the applicants to outline how the project would contribute to gender equality and the enhancement of human rights, with particular emphasis on “leaving no one behind”, taking into consideration how the specific needs of groups such as youth, persons with disabilities, older people, refugees, migrants, the poor, and others would be addressed.

142 Shared Responsibility, Global Solidarity: Responding to the socio-economic impacts of COVID-19. March 2020. 143 A UN Framework for the immediate socio-economic response to COVID-19. April 2020. 144 https://www.un.org/development/desa/da/wp-content/uploads/sites/52/2022/07/T15-Project-Document- Guidelines-vf.docx

65

118. In line with the guiding UN documents, the Surge Project aimed at supporting the MSMEs that were considered the most vulnerable to COVID-19 within the private sector, highlighted the gender, HRBA and LNOB perspectives and fairly covered the gender aspects. Youth and other vulnerable groups were also covered, though to a lesser extent.145 Given the larger number of vulnerable groups working in the informal MSMEs- particularly women and youth - the project was developed to contribute to inclusive growth during the economic resurgence after the pandemic146. As highlighted in the project document, most informal MSMEs are led by the working poor, women, youth and other marginalized and vulnerable groups who need to be integrated into an inclusive economic resurgence after COVID-19 and who are at most risk of falling into poverty and being left further behind. The initial project document included a brief section on the assessment of vulnerable groups such as the women traders involved in cross-border trade, and initiatives benefiting the vulnerable groups, especially women and youth. The second phase proposed a coherent approach towards MSMEs, with the purpose of reaching out to the most affected target groups, including women and informal workers. A specific section on mainstreaming gender equality and human rights aspects including social protection was developed to outline the project approach, and intended to ensure a gender perspective, LNOB aspects and HRBA would be integrated into the activities of different clusters across preparedness, response, and recovery stages. The project document147 further provided a brief assessment of the gender issues and made reference to youth and vulnerable groups under each cluster and stated that the project interventions would address issues in an inclusive manner leaving no one behind. The 3rd phase project document also referred to the project approach of reaching out to the most affected target groups, including women and informal workers148. Specific activities integrating gender and youth were also covered including capacity- building workshops for national government officials and informal MSME entrepreneurs, especially youth and women, an entrepreneurship training workshop to assist entrepreneurs from vulnerable backgrounds and to deliver support to MSMEs in the region, placing its efforts on supporting the poorest, marginalized and women-led micro and SMEs. The result frameworks included specific outputs mainly benefiting women entrepreneurs such as training initiatives for cross-border traders (almost all female traders) and outcome- level indicators covering gender-disaggregated data as well as COVID-19 impact assessments. However, the project document covering the initial phase only mentions persons with disabilities in relation to the project’s links to the SDG targets 8.5 and does not specifically outline disability-inclusive responses. Furthermore, some of the outcome indicators include gender-disaggregated data but do not propose effective data collection methods to capture other vulnerable groups and vulnerabilities or gender dynamics beyond the limited quantitative data.

119. In addition to the key guidelines provided by the UN Secretariat, it was unclear to what extent internal procedures and programming guidance for each UN partner on gender mainstreaming, gender equality or LNOB played a role in the integration of all aspects in the design and implementation of the Surge project. Although the overall project was designed with a strong alignment to the UN guiding documents on HRBA and gender, the integration of these cross-cutting components in the implementation within each UN partner’s activities was less consistent. As an informant from one of the implementing UN entities stated, “gender mainstreaming and reducing inequality are part of their [the staff’s] DNA in all they do.” On the other hand, another claimed that due to many partners involved and the urgency of the situation, internal clearance processes regarding gender aspects for DA projects were not necessarily followed for the Surge project. Also, it was implied by yet another informant that with additional resources (i.e. human, financial, and time) then gender responsiveness, HRBA, and LNOB could have been better integrated into the implementation of the project components.

120. There was limited evidence available of systematic sharing of experiences, capacity, or lessons learned related to the integration of the cross-cutting themes, which might have enhanced leveraging synergies among the UN partner organizations or with other relevant agencies, such as UN Women. One example of a good practice was in the Republic of Moldova where UNECE worked closely with the UN Country team and UN Women contributed to the development of a gender-sensitive roadmap for the government based on a study on MSMEs conducted by the Surge project. One of the survey respondents further

145 UN. COVID-19 Response. July 2020. Global Initiative towards post-Covid-19 MSME sector: MSME Surge Project. Phase II Project Proposal. 146 UNCTAD. 2020. Concept note: Global Initiative towards post-Covid-19 MSME sector Phase I. 147 UN. COVID-19 Response. July 2020. Global Initiative towards post-Covid-19 MSME sector: MSME Surge Project. Phase II Project Proposal. 148 UN. COVID-19 Response. February 2021. Global Initiative towards post-Covid-19 MSME sector: MSME Surge Project. Phase III Project Proposal.

66

emphasized that the project was largely complementary to the work of a few agencies - such as UNDP, UNCTAD and UN Women - and helpful in the process of the CCA analysis and gender assessment. UN Women was also involved in supporting a regional conference on women in the Latin American region. Collaboration with Empretec enabled the development of a training programme for people with low literacy and the Farming-as-a-Business training, facilitating participation and empowerment of the vulnerable groups in the sector. Reflecting on the available documentation and evidence, the collaboration and sharing of experiences among project partners or other UN organizations regarding the integration of gender, HRBA and LNOB perspectives were limited to some cases as outlined above and not systematic across the project clusters.

121. The level of integration of these cross-cutting themes by the main counterparts depended on the counterpart’s understanding, willingness, and priorities. The UN partner organizations worked with the governments, including corresponding ministries and officials, as their main counterparts. One informant stated that as a DA initiative, their main counterparts were governments and they looked at how governments shifted their policies to support women in SMEs. Under cluster 5, one of the governments was not interested in women's studies, mentioning that they wanted everyone to benefit without discriminating against men or women-led enterprises. It was also challenging to receive inputs in places where there was no institutional structure dealing with gender issues or women’s empowerment such as in Cluster 5 or to ensure women’s participation, as it was reported that government counterparts were responsible for selecting participants for the Latin American and the Asia and the Pacific regions. The Bangladesh Start-up Ecosystem Assessment Report integrated gender aspects but the women participation at the follow-up panel was low. In Brazil, as reported by an informant, there was a presence of women in the webinars but no gender parity149. Even if there was a supportive government in Moldova where a roadmap for MSMEs covering gender aspects was developed, new priorities, the crisis in the region and limited funding were cited as reasons why there was very limited follow-up to the government roadmap or recommended actions in the project study supported by the Surge project.

122. In Gambia, however, the government considered youth and women as the main target beneficiaries, given that they made up the largest portion of the informal MSMEs sector in the country and looked at how to revitalize this sector and address their needs. As another example, in Malaysia, as a local counterpart, the National Association of Women Entrepreneurs of Malaysia (NAWEM) helped support women entrepreneurships and ensure women participation through engaging in the webinars. Furthermore, the Cross-border Trade Associations, together with the governments, were involved in selecting the participants, who were almost all women as the activities were targeting women as main stakeholders, for the training programs organized as part of the UNCTAD component on Women and Cross-border Trade, including in Malawi and Zambia. One informant suggested that a key lesson learned for the success of the overall cross- border trade and gender initiative was to collaborate closely with the government authorities and ensure their buy-in and support for the activities.

123. Several needs and impact assessments integrated gender and human rights impact of the pandemic among MSMEs to varying degrees. Under different clusters, several assessments of the impact of COVID-19 on MSMEs particularly focusing on gender aspects were conducted to inform design of project interventions including policy recommendations aiming at leaving no one behind. UN DESA conducted a study on the impact of COVID-19 among MSMEs in Kenya and their resilience mechanisms. The assessment included youth and women organizations, focused on the impact of the pandemic on women and youth-led entrepreneurs and provided specific recommendations for vulnerable groups like the elderly, widowed and orphans in Kenya. UN ESCAP Bangladesh Startup Ecosystem Assessment Report150 included a detailed gender analysis and highlighted the importance of integration of gender and inclusivity angles in the funding opportunities to support the ecosystem by bringing in women, people with disabilities, and people from underrepresented and marginalized groups. UN ECE rapid impact assessments on MSMEs were undertaken in the region with a view of informing gender-responsive trade policies. The assessment reports covered gender gaps, gender- disaggregated data, and strategic priorities to achieve women equality including gender-responsive trade policies. 151 UN ECA supported one region and 11 countries to assess the impacts of the recent crises on the MSMEs of its member states in Southern Africa. Almost all the reports cover gender and/or youth aspects

149 The participants list of the region does not have gender disaggregated data to confirm the informant´s statement. 150 Bangladesh startup ecosystem assessment report | ESCAP (unescap.org) 151 UNECE. The impact of COVID-19 on the trade and business development prospects of female-owned enterprises in the Republic in Armenia. 2021; and The impact of COVID-19 on the trade and business development prospects of female-owned enterprises in the Republic in Armenia. Available at: https://unece.org/trade/studies-regulatory-and- procedural-barriers-trade

67

while the study conducted in Eswatini only indicated that the gender details in terms of ownership were not provided by the state. Furthermore, ECLAC reviewed policies to mitigate the impact of the COVID-19 pandemic on SMEs in nine Latin American countries152 and identified government measures with special emphasis on employment opportunities for women, people with disabilities and youth. The global report of the UNCTAD on COVID-19 Pandemic Impact on Micro, Small and Medium Sized Enterprises: Market Access Challenges and Competition Policy identified gender inequalities, provided disaggregated data for gender, age, minorities, persons with disabilities, and recommendations for entrepreneurs and MSMEs from vulnerable backgrounds particularly affected by the digital divide. The national studies on analysis of the impact of COVID-19 on MSMEs in South Africa and Thailand covered gender issues and gender aspects and proposed recommendations for women, youth and vulnerable groups. The national study on competition and market access policies in the resurgence of MSMEs in post-pandemic in Brazil, however, has no reference to gender, human rights or LNOB aspects.

124. The majority of assessments were gender responsive and primarily focused on gender aspects and on the impact of the pandemic on women-led MSMEs. While several assessments included youth aspects, only a few studies and data analyses covered people with disabilities. When asked, two informants expressed an awareness that other vulnerable and marginalized groups were more rarely covered, though they did not explain why. Despite this limitation, the studies were well received and helped identify the needs of those vulnerable groups. When the survey respondents were asked if the project activities and the content reflected the perspectives of women, youth, people with disabilities or other vulnerable groups, the majority (68%) responded positively. The work dedicated to profiling women-led MSMEs was found insightful to develop legal amendments supportive of women employability, according to one survey respondent. Another respondent mentioned that the research on women’s entrepreneurship was considered helpful for policymakers and ministerial staff in Member States to improve their understanding of how support given to women in business pays off.

125. The degree of integration of gender, human rights and LNOB aspects and implementation of assessment recommendations varied across clusters. In a number of cases, documentation was lacking about how those assessments or policy documents were followed up and to what extent the issues pertaining to vulnerable groups were addressed. Even if the assessments were conducted to identify needs and actions were proposed to address those needs of the vulnerable, there were cases where the evidence showed implementation of the actions was lacking or limited. The Project readiness report on the development of a digital platform on technology and innovation by SMEs in Southern Africa required that users and partners consider women, youth and environmental issues. However, the output report of the digital platform did not cover engagement with women, persons with disabilities or other vulnerable groups or aspects. While the Bangladesh Startup Ecosystem Assessment Report153 included a detailed gender analysis, one of the informants stated that “there was a section on gender, that´s all.” There was no systematic follow-up on how the project-funded studies were utilized. Another informant stated that gender mainstreaming is often on paper and not in reality.

126. On the other hand, in Moldova, the recommendations based on the gender-responsive assessment were translated into a roadmap, in collaboration with the government and UNCT. The government stakeholders considered the study as a big input. However, as one informant described, only some of the recommendations were able to be followed up, such as cross-border transit physical capacity and e- commerce funded by USAID, due to the crisis in the region and limited funding opportunities, among other challenges. In Kenya, the rapid impact assessment findings on women and youth-led entrepreneurs informed the planning of another initiative integrating gender and youth aspects with funding from a project. The project team worked closely with the government and linked their funding streams to increase the overall impact. As informed, none of the proposed activities were stand-alone, but rather aligned with the results chain – cross- cutting vertical and horizontal. Entrepreneurship policy review informed the new MSMEs strategy in Uganda. For the first time, with the support from the Surge project and building on the existing partnership with UNCTAD, Uganda included migrants and refugees in the high-level strategic plan, in addition to women and youth as vulnerable groups, according to an informant. The strategy serves as a framework to mobilize resources to realize different pieces. Another informant described that recently the government launched a strategy to help refugee communities.

152 Argentina, Brazil, Chile, Colombia, Costa Rica, Ecuador, Mexico, Panama, and Uruguay 153 Bangladesh startup ecosystem assessment report | ESCAP (unescap.org)

68

127. Building on the “Informal Cross-border Trade for the Empowerment of Women, Economic Development, and Regional Integration in Southern and Eastern Africa” project of UNCTAD’s Trade, Gender and Development Programme, the capacity-building activities were designed based on the existing gender responsive programme which was then tailored to address emerging needs of women traders during the pandemic in Botswana, Kenya, Malawi, the United Republic of Tanzania, and Zambia. Given the urgency and time limitation, an informant said this component was adapted and implemented very quickly by including COVID-19-related considerations. As highlighted by another informant, it was relatively easy because of the existing relationship with the stakeholders. The training content was revised to include new trade regulations and address needs of different vulnerable groups to enhance knowledge of their rights and obligations, regional trade rules, and on the formalization and registration of business. In partnership with Empretec, training contents were tailored both for participants with low literacy and high literacy levels, enabling engagement with the most vulnerable communities. Gender considerations specific to women traders including harassment were mainstreamed into the activities. Beyond the training activities, the initiative brought together the border officials and authorities dealing with immigration and customs to raise the issue of harassment against women traders. In addition to raising the gender issues, as highlighted above (para 96), the project could further enhance its support to government capacities in cross-border trade by involving government officials, including border officials, in the training courses, as trainees, or having a dedicated training session for them. The majority of the participants who responded to the workshop surveys found the workshops useful and relevant for their businesses. The policymakers and other stakeholders who attended the regional policy dialogue on women and cross-border trade considered the workshops successful at providing a forum to discuss the challenges faced by women informal and small-scale cross-border traders, and formulate practical solutions to improve their situation and enhance the contribution of cross-border trade to economic development and regional cooperation (Box 2).

Box 2: UNCTAD Cross-border Trade and Gender Initiative.

128. Another project initiative primarily focusing on gender aspects was the ESCWA´s Women Empowerment for Technology and Entrepreneurship (AWETE)154 programme which was launched through the DEPAR platform to address the national challenges hindering women empowerment in technology and

154 https://www.unescwa.org/news/boosting-women%E2%80%99s-empowerment-technology-and-entrepreneurship https://depar.unescwa.org/regional/awete

This initiative is a good example of integrating gender responsiveness in the Surge project interventions in extremely vulnerable communities facing additional challenges due to the impacts of the pandemic. Building on the pre-existing programming, the Trade and Gender initiative was adapted quickly to respond to urgent and emerging needs of the cross-border traders, primarily women.

The cross-border trade and gender component of the Surge project replicated the training activities for informal and small-scale cross-border traders of the previous UNCTAD initiative “Informal Cross-border Trade for the Empowerment of Women, Economic Development, and Regional Integration in Southern and Eastern Africa¨ which was implemented from 2016 to 2019. The training workshops were based on the tailored Empretec programme by adapting an existing methodology to meet emerging needs of women traders during the pandemic. The expected results of this initiative included increased awareness of trade rules and procedures, improved knowledge of their rights and obligations, enhanced use of official border posts and trade through formal routes, ability to adapt the businesses and developed strategies, and awareness of COVID-19 measures. Under this initiative:

✓ Almost 300 cross-border traders participated in the workshops, almost all were women.

✓ 9 in-person workshops (6 days each) with two components organized: training on cross-border trade rules and development of entrepreneurial skills.

✓ Botswana, Kenya, Malawi, the United Republic of Tanzania, and Zambia covered.

✓ Project partners included relevant ministries, trade authorities and Cross-border Trade Associations of the respective countries.

✓ Over 90% of the participants found the workshops useful and relevant.

✓ 95% of the participants of the regional policy dialogue considered the workshops successful.

✓ Five traders´ guides ¨Women in informal cross-border trade: A small-scale trader's guide to trade rules and procedures¨ tailored for Botswana, Kenya, Malawi, Zambia and Tanzania.

69

entrepreneurship across the Arab region. AWETE provides a community space for female entrepreneurs, innovators, and professionals to share knowledge and build new partnerships. Under this initiative, the project developed women empowerment ecosystem maps for 22 Arab counties155 and organized a series of five roundtable discussions in the region, bringing key experts and stakeholders advocating for women´s rights.

129. The majority of the survey respondents had the perception that the project activities were accessible for women, youth, people with disabilities, or other vulnerable groups, although the document review identified limited data in project reporting and other relevant sources referring both to cases with varying degree of gender parity and others with no data on gender or other aspects. The ability to assess comprehensively the participation of vulnerable groups is limited to available documentation that mostly covers gender and no other vulnerabilities. In some cases, the data indicated that the events successfully facilitated the engagement of women, ensuring gender parity. The online Empretec TOTs organized in French, English and Spanish for low-literate people reached out to around 100 people, 49% women. Whereas the workshops/courses/webinars on circular economy reached out to 600 participants, only one-third of them women. In other cases, the information on gender or other aspects is not available to make a comprehensive assessment covering all project interventions. However, the majority of survey respondents (70%) believed that the project activities were accessible for women, youth, people with disabilities, or vulnerable groups (Figure 20). The respondents also mentioned that the project included or supported women, youth, people with disabilities or other vulnerable groups through empowering vulnerable groups especially women and supporting their access to funds.

Figure 20: Perceived accessibility of project activities for women, youth, people with disabilities or other vulnerable groups (n=126 respondents).

Source: Independent evaluation survey, 2023.

130. Adapting to COVID-19 restrictions enabled the engagement of the vulnerable groups to some degree, particularly women entrepreneurs in the project activities while, in some cases, their participation was hindered due to the lack of access to technology, selection process of beneficiaries, location of events, or the pandemic impacts and restrictions that made it difficult to bring people together. Going virtual equalized those who were able to participate in project activities and built bridges across countries and regions. However, participation was limited to only those with physical and financial access to the necessary technology, and ensuring participation was challenging as there were too many online initiatives from a range of development partners. Given the pandemic restrictions, most of the interventions particularly in the early stages of the pandemic including workshops and training programs were organized online, and online portals enabled free access to everyone with access. As reported by one of the informants, NGOs could follow some of Empretec´s work online, particularly vulnerable groups and women on the ground. Empretec´s TOT workshops were

155 https://depar.unescwa.org/regional/awete/ecosystemmaps

33%

37%

21%

3% 2% 0%

4%

Project activities were accessible for women, youth, people with disabilities, or other vulnerable groups

Strongly agree

Agree

Somewhat agree

Somewhat disagree

Disagree

Strongly disagree

Do not know

70

revised, and short videos were developed to better serve the vulnerable groups including rural entrepreneurs, farmers and low-literacy entrepreneurs156. One informant believed that having everything online helped people from diverse populations to link across groups and locations in the country. The final report of the project also highlighted that the skills development was facilitated thanks to the online workshops increasing delivery efficiency and accessibility with specific attention to vulnerable groups and people with low literacy levels157.  In addition, the online regional policy dialogues on the role of competition policy in supporting MSME’s economic recovery in the post-COVID-19 crisis were new, in the sense that they were online and open to all stakeholders from governments to MSMEs covering several countries at once. Although the online platforms provided unique opportunities and, in some cases, the only way to reach out to the vulnerable communities, they hindered the engagement of those marginalized groups with no financial and technological access.

131. Additionally, cases were also found of implementation modalities limiting participation of some vulnerable groups. For example, selections of cross-border trade workshop participants, almost all women traders, was the responsibility of the governments in collaboration with the Cross-border Trade Associations. In the Latin American region, it was reported that there were challenges to ensuring the participation of women or other vulnerable groups since it was the governments that conducted the selection of participants. In addition to the selection processes, it was reported that criteria in a few training programmes limited the participation of the most vulnerable. . When translation was provided in another aspect of the Surge project, it was indicated that the translation was difficult to understand, which limited the effectiveness of reaching out to the most vulnerable. In most cases, there was no reporting on participants’ self-identifying with a range of vulnerable groups, such as people with disabilities, making it difficult to assess the extent of engagement in the Surge project by various vulnerable groups. It was indicated by an informant that they had no way to measure how many and with which vulnerable group the participants identified. The locations of some of the events, such as holding them only in the capital city, were also considered as inhibiting participation from diverse groups and regions.

132. A few impact assessments point towards positive outcomes particularly for women. Overall, 64% of the survey respondents felt that the project contributed to better outcomes for women and youth entrepreneurs and other vulnerable and marginalized groups (Figure 21). The survey respondents revealed that the relevant areas on which the project has contributed to the resurgence of MSMEs, policy or economic improvements, or advancing sustainable development included improved inclusion of vulnerable groups with a special reference to women entrepreneurs, funding opportunities for women, addressing the needs and supporting women-led MSMEs. Capitalizing on the knowledge or skills acquired through the project, some survey respondents reported that they facilitated priority policies for women-owned and women-responsive businesses, development of female entrepreneurships, and access to credit for women entrepreneurs and women capacity-building.

Figure 21: Perceived contribution of the project to better outcomes for women and youth entrepreneurs and other vulnerable and marginalized groups (n=105 respondents).

156 Under the Surge project, Empretec offered in-person, hybrid as well as online training activities to respond to the different needs and COVID-19 restrictions of various contexts. 157 UNCTAD. September 2022. Final Report: Global Initiative towards post-Covid-19 resurgence of the MSME sector.

71

Source: Independent evaluation survey, 2023.

133. The UNCTAD component on Women and Cross-border Trade delivered solid results in terms of enhancing cross-border traders’ capacity, mostly women. The impact assessment on the cross-border trade in the pre- and post-pandemic environment covering 2019-2021158 indicated that the training workshops that were attended by almost all women helped women entrepreneurs become more effective cross-border traders. The impact assessment further outlines how the initiative helped enhance the knowledge and abilities of the cross-border traders to benefit from greater business resilience (Please see the effectiveness section of this report for assessment findings). The initiative also contributed to the formalization of the businesses of the cross-border traders. As reported by an informant, female traders were informal but now most of the training participants have registered their businesses.

134. The impact assessment on the Empretec workshops (ETWs) which assessed the economic impact of the ETWs on the participant entrepreneurs in selected project countries indicated positive results for women entrepreneurs, only for those areas where gender disaggregated data was provided159. Based on the findings, the number of women in business after the ETW increased by 15% in Benin, 7% in Ghana and 9% in Nigeria. 160 The F/M TEA ratio (Female and male, Total Early Stage) also demonstrated positive outcomes in Benin (15% higher161), Cameroon (13% higher), Gambia (57% higher) and Zimbabwe (20% higher).162

The project was also effective in facilitating women and youth-led MSME registration and formalization. UNCTAD helped facilitate e-registration and formalization of MSMEs in El Salvador, Benin, Cameroon and Mali through an e-registration portal. In Benin, the portal increased business registration by 91%, with 84% increase in women-led MSMEs and 181% increase in youth-led MSMEs. A similar trend was observed in Mali where business registration increased by 40% including 49% increase in women-led MSMEs, and 110% in youth-led MSMEs.

135. In contrast to survey respondents’ perceptions, the available data on inclusion of vulnerable groups and vulnerabilities is limited and hinders the evaluation’s ability to assess the degree to which gender, HR and LNOB aspects were integrated into the Surge project. Of the data that is available, it is mostly for the gender-

158 UNCTAD (2022). Cross-border Trade in the Pre- and Post-Pandemic Environment Evidence from Malawi, the United Republic of Tanzania and Zambia. Geneva. 159 The assessment provided limited gender disaggregated data and did not cover any other cross-cutting dimensions. 160 UNCTAD. MSME Surge Project. Empretec: Impact Assessment Survey Report. 2023. 161 In Benin, for instance, the ratio 0,8/1 referring to 8 MSME businesses opened by women to every 10 MSME businesses started by men – at Empretec the ratio is 0,92:1 (15% larger). 162 The F/M TEA (Total Early Stage) (Female / Male TEA ratio, applying only to early-stage businesses) is an indicator of how Entrepreneurship may be used as a tool for equity and women empowerment, especially in traditional male dominated societies as usually verifiable in Africa.

32%

32%

28%

2%

3% 0%

3%

The Surge project contributed to better outcomes for women and youth entrepreneurs and other vulnerable and marginalized groups

Strongly agree

Agree

Somewhat agree

Somewhat disagree

Disagree

Strongly disagree

Do not know

72

disaggregated information for the project beneficiaries with no or limited information on age, persons with disabilities or other characteristics. As reported by one informant, the initiative focused on female youth but there was no record on how many female youth entrepreneurs were reached. In some cases, the gender- disaggregated data was only available for those who responded to the event´s survey and not for all the participants, making it difficult to assess the gender parity163. The data collection tools were mostly not designed to collect information necessary to assess the cross-cutting dimensions. For example, one of the workshop evaluation forms used under cluster 1 did not have any questions to identify issues related to gender, age, disabilities, and other vulnerabilities of specific groups. This is also observed across clusters covering various project initiatives.

136. Furthermore, the project results framework mainly captured quantitative gender-disaggregated data, but did not capture the degree of the project´s contributions to different vulnerable groups and vulnerabilities. There was also limited follow-up to assess the impact of the training or other initiatives from gender and human rights perspectives. As one informant observed, follow-up monitoring should have been included to assess the impact of the training programs and nobody was doing it.

137. One of the exceptions was the Impact Assessment on the Empretec Workshops (ETWs) covering 6 African countries between 2020 and 2022, namely Benin, Cameroon, Ghana, Nigeria, The Gambia and Zimbabwe. The assessment on ETWs did not capture all the cross-cutting aspects comprehensively but provided gender-disaggregated data for the number of women in business after ETWs. Another assessment was conducted for the capacity-building activities organized by the UNCTAD Trade and Women initiative in Malawi, the United Republic of Tanzania and Zambia aimed at small-scale and informal cross-border traders, mainly women. The impact assessment covered the capacity-building activities carried out from 2019-2021 at selected border posts, including previous analytical work carried out by UNCTAD´s Trade, Gender and Development Programme since 2016 under a 10th tranche DA project164. The compilation of success stories of women entrepreneurs who received Empretec training - such as the story of how four women in Malaysia built success during COVID-19165- also helped provide anecdotal evidence on the project’s contributions to women empowerment and inclusive growth.

163 The number of participants of the UNCTAD Webinar Series on SDG Reporting by companies, including MSME—The adaption of “Guidance on core indicators for entity reporting on contribution towards implementation of the Sustainable Development Goals (GCI)” in China was 150 while the gender-disaggregated data is only available for 26 survey respondents. 164 The Development Account project “Informal Cross-border Trade for the Empowerment of Women, Economic Development, and Regional Integration in Southern and Eastern Africa”, implemented during 2016-2019. 165 https://unctad.org/news/how-four-women-malaysia-built-success-during-covid-19

73

6. Conclusions

138. Based on an analysis of the findings, the evaluation formulates the following conclusions.

Conclusions

Relevance

The project responded to the needs of Governments and MSMEs. Interventions contributed to and were informed by needs and impact assessments. Recipients of project interventions were often involved in or contributed to the design and delivery of project activities and outputs. On various occasions, project outputs were tailored to the needs of target users and beneficiaries, being countries or vulnerable groups. The relevance of the interventions was somewhat mitigated by the mandates and capacities of the implementing UN entities. The latter are equipped for supporting medium- and long- term policy changes rather than immediate crisis responses. Other challenges included the restrictive Covid-19 measures and the short time frame for implementation of the project. Nonetheless, on the whole, the implementing UN entities exercised adaptive management to ensure high relevance of their interventions. The phased approach, in particular, provided the opportunity to introduce new activities not foreseen at an earlier stage.

Coherence

The project was anchored in the mandates and comparative advantages of the implementing UN entities. Selected interventions followed previous work and specific requests from member States and were often coordinated with or complementary to existing activities. At conception, the target of the project was to address specific constraints faced by MSMEs. It was designed around the Entrepreneurship Policy Framework to show the complementarities across interventions that each implementing UN entity would deliver in response to regional priorities and capacities. Some collaborations between implementing UN entities were pursued and complementarities established, but on a limited level. A few synergies in the form of joint outputs were integrated in the 3rd phase of the project. By design, technical task forces created through the project were expected to bridge the implementing UN entities. However, this proved difficult to launch and sustain, owing to several constraints, including a highly ambitious objective given the global crisis and competing priorities within each agency. Nevertheless, the coordination and complementarities established by the project among the implementing entities contributed to expanded dissemination and outreach of outputs, avoided overlaps between them, and allowed significant knowledge exchanges aligned with and supportive of a “One UN” system. On the other hand, collaboration of the implementing UN entities with other UN organisations, including UNCTs, was limited. Despite this shortcoming, at national level, target recipients of project outputs found the interventions complementary to those of other UN agencies.

Efficiency

Project coordination was complex and demanding, involving seven UN entities and the target delivery of around 100 outputs, of which some were composed of many activities. Nevertheless, the coordination of the project was found to be efficient overall given the circumstances and resources available. The coordination of the design phase of the response was actively supported by the DA-PMT. Coordination of project implementation was led by UNCTAD in collaboration with a Steering Committee that met on a regular basis, most often bi-monthly. UNCTAD’s Budget and Project Finance Section (BPFS) produced the project’s financial monitoring table every month, which included the expenditure data of UNCTAD as well as the other implementing entities. A monitoring dashboard was created by UNCTAD to facilitate the tracking of UNCTAD’s project delivery. Data from some of the other implementing UN entities was added, when provided, at two reporting points during the lifetime of the project.

The project would have benefited from the allocation of more resources towards the coordination and monitoring of overall implementation. Across the implementing UN entities, the project was considered complex and hard to follow outside of the interventions under their direct control. Some staff perceived that there were too many participants in the coordination meetings, that the project lacked a Theory of Change, and that it was primarily a compilation of activities rather than a cohesive and mutually reinforcing bundle of interventions supporting in depth any given country. Some staff in the UN Regional Commissions questioned the efficiency cost of a global or inter-regional response versus regional responses. The project was implemented in three phases, which were supportive of adaptive management and perceived by staff as the most rational approach considering the circumstances. However, it was also pointed out that clearer visibility

74

from the onset on the funding available for future phases would have facilitated planning and the search for synergies.

Effectiveness

The Development Account programme, as a funding modality, was quick to respond and adapt to the COVID- 19 crisis. The prompt availability of resources and rapid orientation toward a global response contributed to the rapid launch of a wide range of interventions. However, questions remain as to its adequacy for addressing the immediate needs of a crisis in comparison with other UN agencies and mechanisms with a mandate for humanitarian response. The governance of this project, with DA-PMT’s oversight, the management structure with UNCTAD in the lead coordinating role, and each implementing entity leading the operational implementation at the regional level, ensured the global and regional scope of project delivery. At the operational level, implementing entities relied more on partnerships with local and regional institutions rather than coordination within the larger project management structure. The project governance and management mechanisms evolved in tandem with the COVID-19 crisis and the project’s emerging needs. The result was a high delivery rate but with limited inter-agency collaboration in the process. Yet, despite limited collaboration, this independent evaluation identified that the project was effective in producing identifiable results at the outcome level.

Overall, the project delivered 85% of the planned outputs. Phase 3, particularly, delivered the highest number of outputs but had the lowest outcome effectiveness rate, mostly because of budget cuts and some outputs still in progress at the time of this evaluation. There is evidence of the effective improvement of national capacities on formulating and implementing enabling policies on green, resilient and inclusive entrepreneurship and MSME promotion (outcome 1.A) [such as the adoption of a revised entrepreneurship strategy by South Africa, Uganda and Seychelles]; improved resilience and competitiveness of MSMEs (outcome 1.B) [for example with 64.5% of participants opening a new business after attending Empretec-based training]; facilitated MSME registration and formalization through e-platforms (outcome 2) in El Salvador, Benin, Cameroon and Mali; improved access to finance, including upskilling in financial literacy (outcome 3) [for example with 97% of the MSMEs in Latin America that attended trainings reporting improved financial literacy in accounting and reporting and also improved capacity to manage financial resources]; increased MSME access to innovation and technology (outcome 4) [including through capacity development on green technologies for SMEs in Southern Africa]; and increased access to local, regional and international markets through digitalization and non-tariff measures (outcome 5) [for instance with case studies and on competition in Thailand, South Africa and Brazil, followed by webinars and the creation of an online course on SMEs and competition policy, primarily for government officials].

In relation to the project indicators, the assessment of their effectiveness was based on limited information, proxies, and expert judgement since they often lacked SMART criteria. It was possible to infer that 85% of them showed some level of progress, either through objective measures (e.g., sales increase) or proxies (e.g., post-training satisfaction assessments and survey responses). The high effectiveness results can be partly attributed to the conceptual framework provided by the EPF, as well as the cluster-based approach with each cluster addressing specific regional constraints and responding to demand from member States. In addition to clustering about 100 outputs from seven implementing entities into 5+1 outcomes, the EPF provided a coherent thread to relate outputs that cut across outcomes. Further analysis would be needed, however, to clearly identify which of those outputs are the most suitable or adaptable for a crisis response.

The project improved the capacity of policymakers in designing and implementing policies supportive of MSMEs, especially in terms of contributing to a country’s government responses, country-specific studies, technical assistance, training courses, and the development of digital tools and regional interventions. This improved capacity is reflected in beneficiaries’ reports of how they have been considering the needs of MSMEs, including those led by women and youth, in their daily work. These results could become more transformative and sustainable with more institutional support to networks of policymakers and communities of practice for mutual and regional learning on supportive policies for MSMEs, as well as by mainstreaming gender and Leaving No One Behind (LNOB) approaches in government officials' capacity-building.

Sustainability

The project developed approaches, tools and capacities that are being transferred at multiple levels. At the policy level, the addition of more functionalities to e-government tools, expanding training courses to more countries, and the establishment of agreements to implement the project recommendations are all strong

75

evidence of the sustainability of the project's achievements. At the behavioural level, beneficiaries are applying new knowledge and skills to their daily work, be it by including MSMEs' issues in policy making or by improving MSMEs' management, performance, outputs and, by extension, resilience. Challenges to this transfer of knowledge relate to the lack of financial resources, need for follow-up mentoring, and lack of a favourable business environment.

Gender, human rights, and disability

The overall project design was well aligned with the two key UN documents that provide clear direction for the integration of a human rights-based approach, which include addressing the aspect of gender equality and Leaving No One Behind. Although these aspects were well-articulated in the project documents, strong follow-through and documentation of their implementation was limited to a few project components. For those components that did include these aspects, gender was more commonly integrated and youth to a lesser degree. There was very little evidence of the outreach, participation, or impact of project activities for people with disabilities or other vulnerable groups. The UNCTAD component on Women and Cross-Border Trade targeting five countries in Africa was a good practice in how to design and implement this type of intervention with a strong reflection of the needs of some of the most vulnerable women across border areas.

76

7. Recommendations 132. Based on the evaluation findings and conclusions, the following recommendations could be considered by the implementing UN entities and the DA-PMT.

139. Recommendation 1: (i) UNCTAD should further leverage the experience gained through the project to map out how components of the Entrepreneurship Policy Framework relate to and can be best positioned to support MSMEs in their recovery from different types of crises and (ii) DESA and the Regional Commissions should add their analysis of how their work can contribute towards the objective. UNCTAD, DESA and the Regional Commissions could build on the project to map their areas of intervention in support of the MSME sector, as well as capacities and knowledge on the implementation of the EPF components, including in a crisis context, and opportunities for broadening the uptake of EPF components at the regional level based on national needs. UNCTAD, DESA and the Regional Commissions could further consider identifying areas of joint interventions that would trigger complementarities and synergies between the agencies. This could involve collaborating on the development of capacities of all national and sub-national actors, i.e. Governments, MSMEs, other partners (e.g. Chamber of Commerce, Business Incubators, etc.) in line with United Nations Sustainable Development Cooperation Frameworks (SDCF) in respective countries; strengthening or contributing to facilitate access to finance for the MSMEs (including seed money, grants, access to credit, etc.); improving the coordination of MSMEs related policies across ministries; increasing interventions at the local level, such as by supporting NGOs or MSMEs outside of the main cities. The application of a human rights-based approach, gender responsiveness and inclusion of other vulnerable groups (LNOB) should be ensured.

140. Recommendation 2: UNCTAD should continue building on the momentum generated by the project to continue fostering knowledge exchanges and promoting the Entrepreneurship Policy Framework. UNCTAD, DESA and the Regional Commissions should identify means to more meaningfully continue to share good practices and lessons learned on the demand-driven support they provide to the MSME sector, including on areas such as green/circular economy; innovation policies and ecosystems; MSME formalization and strengthening MSME capacities for the empowerment and leadership of women, youth and groups in vulnerable situations. UNCTAD and the Regional Commissions could also propose organizing regional events and/or a global conference to promote entrepreneurship policy and further advance the goals and impact of the Surge project. Furthermore, learnings from the experiences with the Surge about external partnerships should encourage UNCTAD to promote the EPF and entrepreneurship development to other UN agencies also engaged in this area (e.g. ILO, FAO, UN Women, etc.), to development banks, or to related initiatives such as the recent Global Accelerator on Jobs and Social Protection. UNCTAD could also consider joining and supporting events that promote the achievement of SDG 8.3 as an avenue to further promote the EPF.

141. Recommendation 3: The DA-PMT should develop a clear framework for assessing the costs and benefits of implementing a global or inter-regional project versus regional projects. As a global crisis can affect regions and countries differently, global or joint projects should be developed only when there are clear benefits of joint implementation. Thus, a global or inter-regional vs. regional response would not be a priori decision, but a result of a clear assessment. A specific guideline or framework should be developed for this. Some of the assessment criteria could include the need or demand from member States for an integrated response; the range of common versus entity-specific activities and added value of complementary interventions (i.e. in terms of geographical coverage, reach of the target population, capacity, and/or coverage of multiple reinforcing technical areas, or networks and partnerships); capability to execute joint interventions (e.g. in terms of time, resources, logistics, and flexibility to pivot interventions without bureaucratic procedures); coordination costs (which increase with the number of participating UN entities); project inception modalities and governance and knowledge management requirements to facilitate collaboration and knowledge sharing; scaling and sustainability plans; expected results of spreading resources versus concentrating on fewer countries; and so forth. The time taken to conduct such assessments will increase project coherence and effectiveness and facilitate the identification of the financial and human resources needed for project implementation. The network of DA Focal Points is a key existing asset for this assessment.

77

142. Recommendation 4: Implementing UN entities should ensure that they have a comprehensive Results Framework for the entire project as well as an adequate monitoring plan, with indicators that are designed to support the ongoing monitoring. The results framework should have one objective and ideally have one outcome per cluster/workstream. The objective should state the intended goal of the project, describe the overall achievement targeted by the project, involving a process of change aimed at meeting the needs of identified beneficiaries, and reflect the overall funding available to the project. Each objective should include reference to the project’s beneficiaries and its substantive focus. The objective should not attempt to explain the ways in which the project intends to achieve the objective (i.e. it should not include the word ‘through’ or describe the internal work of the UN using verbs such as ‘support’, ‘facilitate’ or ‘contribute’). The outcomes (OCs) should describe the changes that are expected to occur as a result of the completion of outputs. The OCs should be achievable within the project’s timeframe and budget, and should be specific enough to be measured by the associated indicators of achievement. The indicators of achievement (IAs) should provide measures for monitoring progress towards achieving the OCs and reporting on them after completion of the project. Every indicator needs to provide clearly defined baselines, units of measurement and targets, detailing the quantity, quality and timing of expected results. The monitoring, evaluation, accountability, and learning (MEAL) system should also be designed to capture HRBA, gender and LNOB aspects. In phased interventions or during project execution, any changes in the logframe if/when pivoting activities should be clearly explained to the wider team.

143. Recommendation 5: Implementing UN entities should ensure that sufficient resources are allocated to project coordination, technical collaboration, and partnership building. The absorption capacity of implementing entities can be challenged by crisis response projects which add to the planned programme of work. This is further compounded by projects that come with an extensive UN partnership and a global scope. Sufficient resources should be dedicated to global coordination and to building global partnerships with strategic stakeholders (e.g., UN organisations engaged in supporting the MSME sector; development banks). Capacities should also be directed to supporting technical collaboration and the staff implementing interventions, including towards synergy or liaison with the UNCTs. When designing the project, implementing UN entities should consider featuring coordination and partnership-building in the Theory of Change or logframe of the project. Tools to support continuous connections and knowledge exchange, and to ensure institutional memory should be part of the response package, such as a project website, SharePoint space for all team members, and a Yammer network or Teams channel. The integration of cross-cutting aspects (HRBA, gender responsiveness, LNOB) also requires expertise with sufficient and dedicated time and resources. Guidance could be development to project managers on how to do this.

78

Annexes

Annex 1: Project’s Theory of Change .....................................................................................................................................

Annex 2: Evaluation matrix.......................................................................................................................................................

Annex 3: Data collection instruments ...................................................................................................................................

Annex 4: List of documents reviewed ...................................................................................................................................

Annex 5: List of individuals interviewed ...............................................................................................................................

Annex 6: Evaluation surveys ....................................................................................................................................................

Annex 7: Evaluation Terms of Reference ..............................................................................................................................

79

Annex 1: Project’s Theory of Change