ii
CONTENTS
Page
I INTRODUCTION ........................................................................................................... 1
1.1 Project Background ........................................................................................................... 1
1.2 Evaluation Purpose and Scope ....................................................................................... 2
1.3 Sampling strategy ......................................................................................................... 2
1.4 Evaluation Methodology .................................................................................................... 4
1.5 Evaluation questions ......................................................................................................... 6
1.6 Leaving No-One Behind ..................................................................................................... 8
1.7 Limitations ........................................................................................................................ 8
II. EVALUATION FINDINGS ........................................................................................... 9
2. RELEVANCE AND COHERENCE: WAS THE PROJECT DOING THE RIGHT
THING? .............................................................................................................................. 9
2.1 Appropriateness of project design for meeting the needs of beneficiary countries .............. 10
2.2 Integration of gender, human rights, and disability perspectives ........................................ 12
2.2.1 Gender, human rights, and disability perspectives: project results ..................................... 13
2.2.2 Potential including gender and human rights perspectives better in future project design
and implementation .................................................................................................................... 13
3. EFFECTIVENESS: WERE RESULTS ACHIEVED, AND HOW? .......................... 14
This section assesses the project results' achievement using the following sub-criteria: i)
overview of project objective achievement; ii) evolvement of the project's strategic vision; iii)
unintended effects; iv) factors affecting project performance, v) lessons learned, vi) changes in
the competences of innovation policymakers, vii) coordination with other UN and non-UN
stakeholders and viii) challenges and mitigation. ........................................................................ 14
3.1 Overview of achievement of project objectives ................................................................. 15
3.2 Strategic project vision and how it evolved ....................................................................... 20
3.3 Unexpected effects .......................................................................................................... 21
3.4 Factors affecting project performance .............................................................................. 22
iii
3.5 Lessons learned .............................................................................................................. 24
3.6 Changes in competencies of innovation policymakers ........................................................ 24
3.7 Coordination with other UN stakeholders ......................................................................... 25
3.8. Challenges and mitigation ............................................................................................... 26
4. EFFICIENCY: WERE RESOURCES USED APPROPRIATELY TO ACHIEVE
PROJECT RESULTS?..................................................................................................... 27
4.1 Adequacy of funding for project results ............................................................................ 27
4.2 Timeliness of results achievement and efficient organization ............................................. 28
4.3 Efficiency of resource use ................................................................................................ 28
5. SUSTAINABILITY: ARE RESULTS LASTING? ..................................................... 30
5.1 Measures to ensure sustainability of project results .......................................................... 30
5.2 Ownership of project results, institutionalization, and up-scaling ....................................... 32
5.3 Potential for replication ................................................................................................... 34
SECTION III: CONCLUSIONS AND RECOMMENDATIONS ................................... 36
ANNEX 1: TERMS OF REFERENCE OF THE EVALUATION ................................. 44
ANNEX 2: DOCUMENTS REVIEWED ....................................................................... 50
ANNEX 3: LISTS OF STAKEHOLDERS INTERVIEWED ........................................ 51
ANNEX 4: EVALUATION MATRIX ............................................................................. 52
ANNEX 5: COUNTRY CASE STUDIES: ARMENIA AND GEORGIA ...................... 56
iv
FIGURES
Figure 1: Infographic - Overview of the project and main evaluation results ................. xiii
Figure 2: Map of the South Caucasus with countries visited during the evaluation:
Armenia and Georgia .......................................................................................................... 3
Figure 3: Concept of theory-based evaluation .................................................................... 4
Figure 4: Overview of project stakeholders and interviews accomplished......................... 5
Figure 5: Stakeholder perception about the relevance of the project responding to their
needs ................................................................................................................................. 12
Figure 6: Perception about the achievement of project objectives ................................... 17
Figure 7: Donor Agreement work Streams delivery ......................................................... 18
Figure 8: Additional work streams ................................................................................... 20
Figure 9: Factors affecting project performance ............................................................... 23
Figure 10: Stakeholder perception of the project’s effectiveness ..................................... 26
Figure 11: Stakeholder perception about the project’s efficiency .................................... 29
Figure 12: Stakeholder perception about the project’s sustainability ............................... 34
Figure 13: Summary of key findings, conclusions, and recommendations ..................... 37
ACRONYMS AND ABBREVIATIONS
AICS Italian Agency for Development Cooperation
COVID-19 Corona Virus Disease
DAC Development Assistance Committee
EA Expected accomplishment
EESC Eastern Europe and the South Caucasus
EU European Union
I4SDR Innovation for Sustainable Development Review
IDP Internally Displaced Person
IEP Innovation Enhancing Procurement
IPI Innovation Policies Index
IPO Innovation Policy Outlook
IT Information Technology
MFAIC Italian Ministry of Foreign Affairs and International Cooperation
NGO Non-Governmental Organization
OECD Organisation for Economic Co-operation and Development
SDG Sustainable Development Goals
Sida Swedish International Development Agency
SME Small and medium-size enterprises
ToR Terms of Reference
UN United Nations
UNECE United Nations Economic Commission for Europe
US$ United States Dollar
USAID United States Agency for International Development
% Percentage
vi
EXECUTIVE SUMMARY
September 2023
This document constitutes the evaluation report of the UNECE project E317, "Promoting
innovation policy capacities in Eastern Europe and the Caucasus" (hereafter referred to as "the
project"). It was produced by Dr. Tania Tam, who is a senior international and independent
development project evaluator. She was selected by the Programme Management Unit, which is an
oversight body that ensures overall coordination of UNECE programme management (planning,
monitoring, reporting and evaluation) of all UNECE activities funded from regular and extra
budgetary resources.
Results summary
The UNECE project received excellent scores across its four evaluation categories. The project was
highly effective, with a stakeholder satisfaction score of 82%. It accomplished all seven expected
accomplishments, such as National Innovation for Sustainable Development Reviews (I4SDRs) for
Armenia, Georgia and the Republic of Moldova or developing the sub-regional Innovation Policy
Outlook (IPO). Stakeholder satisfaction reached 88% for the project’s relevance and coherence and
86% for its efficiency, given the excellent UNECE project management and implementation. The
sustainability of project results was also very high, with 83% stakeholder satisfaction due to high
results ownership. The most direct impact has occurred through the implementation of the I4SDR
and IPO recommendations.
Project background
The project was launched in November 2018 and is set to conclude in June 2024 after no-cost
extension. As per the project document, the project's main aim was to improve innovation
policymakers' competencies in the Eastern Europe and the South Caucasus (EESC) region. The
total budget for the project was the equivalent of US$ 2,215,026.
Evaluation purpose and scope
The evaluation assessed the project's relevance, effectiveness, efficiency, and sustainability in
enhancing innovation policy capacities in EESC countries – Armenia, Azerbaijan, Belarus,
Georgia, the Republic of Moldova, and Ukraine.
The primary intended audience of the evaluation is the Sida and UNECE staff involved in funding
and implementing the project. Secondary intended audiences include beneficiaries and country
representatives involved in the project. The Programme Management Unit plans to share the
evaluation with other UNECE divisions and accumulate the lessons learned through the evaluation
for improving other divisions’ work (especially in terms of the “Leave No One Behind” (LNOB)
approach). In line with UNECE evaluation policy, the objectives of evaluations are to: (1) Promote
organizational learning; (2) Improve programme performance; and (3) Ensure the accountability of
the UNECE to member States, senior UN system leadership, donors, and beneficiaries. The
evaluation had a duration from May 2023 to August 2023.
vii
Evaluation methodology
For this evaluation, the evaluator used a theory-based evaluation methodology to address the
timeline between the project activities, such as capacity building, data collection and analysis, and,
for example, changes in policy capacities.
The evaluation reached 74% of the project stakeholders identified by the project team, 49 in total,
through in-person interviews during a field visit to Armenia and Georgia (11), telephone and video
interviews (20) and an online survey (an additional 18). Thirty-one in-depth semi-structured
interviews were conducted, each consisting of 45-60 minutes. During these 31 interviews, all
respondents were also asked for survey responses, thus providing 31 quantitative survey responses
in addition to 18 respondents filling in the online survey.
UNECE’s Project Management Unit vetted project stakeholders interviewed during the evaluation.
The surveyed group consisted of policy specialists in innovation and expert stakeholders from
international organizations who understand innovation policy. Thus, by its very nature, the group
was small but highly relevant and knowledgeable about the evaluation subject. Thirty-one
interviews were highly in-depth and, combined with the high level of expertise on both innovation
and the project, yielded sufficient quality of input for the evaluation.
Regarding the gender of participants, 20 of the 49 respondents were female (41%), and 29 were
male (59%). The field visits to Yerevan and Tbilisi took place in the week beginning June 5, 2023.
The online survey launched in mid-July yielded a response rate of 60%, based on the 18 responses
received.
The evaluation reach is very satisfactory, given other UN Secretariat evaluation response rates
range between 15% and 30%.
Evaluation findings, conclusions, and recommendations
Relevance and coherence: The project was highly relevant to the six project countries despite
conflicts, ongoing political instability, and the pandemic since the project started in 2018.
The UNECE project gave crucial support to the region, with Sida and UNECE filling a critical gap
at the time when the project countries prepared national innovation strategies. The evaluation finds
that the project components were relevant, as all other analytical tools on science, technology and
innovation (STI) - such as the European Innovation Scoreboard - was less relevant for transition
economies, requiring a more sub-regional approach through an Innovation Policy Outlook (IPO)
and Innovation for Sustainable Development Reviews (I4SDRs).
The project addressed previously poor coordination among national stakeholders, exacerbated by
frequent staff changes in government agencies in all project countries.
As a Geneva-based body with no regional offices, UNECE provided high-quality analytical input
using international best practice recommendations to guide other organizations operating on the
ground. For example, in the I4SDR of Ukraine, currently at the research stage, a chapter will be
analyzing the current reconstruction strategies and suggesting improvements to these strategies.
Hence, the project efforts guide the larger international community in working within the EESC
region on the topic of science, technology, and innovation, as well as sustainable development more
broadly.
viii
The evaluation finds that the Leave No One Behind approach promoted by the United Nations
Secretariat was taken very seriously by the project team and showed a high level of coherence
during its implementation. Concerning gender equality, the project team considered the equal
participation of women and men when organizing training and conferences. This is a small and
subtle step but inviting diverse stakeholders to such events ensures that they receive the necessary
training and career capital that participating in such events grants and their perspectives are heard.
However, a broader Leave No One Behind prioritization was not reflected in governments' needs
and priorities, and results were limited. Hence, greater awareness and education is required among
stakeholders about the importance and benefits of integrating gender, human rights and disability
perspectives into project design and implementation. It should be noted that human rights and
disability were not included in the donor agreement. Thus, the donor did not demand mainstreaming
of these considerations. Instead, the donor agreement mentions environmental sustainability and
poverty reduction as overarching goals. Both were honoured in UNECE work and in beneficiary
countries. One way to promote the integration of gender, human rights and disability in future
programming is to establish indicators measuring the impact of these in the programs and relevant
targets in the relevant implementation sectors.
For example, regarding gender, monitoring (1) the gender balance among speakers, panellists, and
participants at innovation-related events and conferences and (2) equal access or use of mentorship,
networking, and professional development opportunities in the innovation space for women and
men can reflect inclusivity efforts. Further down the line after implementation of programs,
monitoring (3) gender pay gap within industries closely associated with innovation, such as
technology and STEM fields and/or (4) and gender distribution in leadership positions within
innovation-driven fields or within key roles such as researchers, inventors, entrepreneurs, investors,
and decision-makers within innovation-related organizations and institutions can demonstrate any
growth or change in LNOB implementation or prioritization in the future.
Regarding disabilities, indicators may include accessibility the degree to which innovation
initiatives and technologies consider and incorporate universal design principles, making them
accessible to individuals with various disabilities and monitoring the integration and enforcement
of accessibility standards and guidelines in innovation policies and practices, ensuring that
products, services, and technologies are usable by individuals with disabilities --– for example, the
upgrading of online procurement platforms for reading impaired persons.
Regarding human rights, indicators may include (1) the extent to which innovation policies
incorporate principles of data protection and privacy, safeguarding individuals’ personal
information from misuse or unauthorized access, (2) whether innovation policies encourage the
development and use of technologies that adhere to ethical guidelines, avoid harm, and prevent
discrimination or bias, (3) examination of how innovation policies address the impact of
technological changes on employment, job quality, and workers’ rights, ensuring fair treatment and
protection of workers’ rights, and (4) whether innovation policies mention informed consent and
individual autonomy when it comes to the collection, use, and sharing of personal data and
information.
One notable finding of the evaluation was the constant methodological improvements of UNECE’s
flagship analytical tools, notably, the I4SDRs and the IPO. Elective chapters were added, meeting
direct demand from the countries themselves. This work could now be scaled up to other regions
and/or countries with little cost. Operations are already running smoothly in the organization of
regular regional policy dialogue sessions and consultations, and in the dissemination of findings
and conclusions, making scaling the project up, continuing it, and adapting it to other regions cost-
ix
effective in terms of materials and operations, considering the expertise accumulated and the
significant momentum created over these past years.
Conclusions: The project addressed a significant sub-regional development cooperation gap
through its in-depth support to science, technology, and innovation (STI), in particular its focus on
STI policy and governance. The project provided visibility to Sida and UNECE, transferred large
numbers of relevant international good policy practices to the region (including lessons learned
from Sweden’s innovation development journey), and worked closely with all relevant stakeholder
groups and to the enhance the region’s STI ecosystem. However, not all deliverables reached all
countries evenly, and the Leave No One Behind approach was supply-driven. The adaptation of the
project for any future implementations will be highly cost-effective.
Recommendations for similar future projects:
Priority: medium, next 6-12 months for new project designs.
Effectiveness: The project accomplished two out of three objectives above expectations, with
the third one, the implementation of recommendations, showing results in some countries
while in others, it is still too early to assess.
The evaluation found the following results by project objective:
Project objective a) improved policy dialogue:
• The project successfully established a multi-stakeholder dialogue for developing the
Innovation Policy Outlooks (IPOs) leveraging UNECE's neutrality as a UN body, using
international and local expertise and comparing the six countries.
• The dissemination of lessons learned through UNECE’s IPO/I4SDR/CB/dialogue work at
its intergovernmental CICPPP and ToS-ICP sessions also enhanced policy dialogue, where
hundreds of member States and international organisations representatives listen in and
exchange knowledge about innovation.
R1: Given its high relevance, it is recommended to i) seek continuation funding to support
further Eastern Europe and the South Caucasus countries (especially for ensuring
sustainability and continuity); ii) seek new funding to replicate this project for other
subregions, such as the Western Balkans or Central Asia. At the same time, this can raise
donor visibility and complement investments into the European Union’s (EU's) Eastern
Partnership (in the case of Eastern Europe and the South Caucasus). UNECE’s governance-
focused approach using evaluation, accountability, and transparency also aims to reduce
corruption and informality in the EESC region, where this issue is of great relevance.
Priority: high, next six months for new project designs.
R 2: Increase awareness among national stakeholders in Member States about the
importance of the Leave No One Behind approach across policy-making, for example, by
adding relevant indicators in the project results framework, and supporting ongoing
national processes such as the upgrading of online procurement platforms for reading
impaired persons, with a focus on gender, disabilities, and human rights (keeping in mind
that the latter two were not included in the project document).
x
Project objective b): Improved understanding at the national level of policy options:
• The project successfully improved the understanding of policy options at the national level
based on research and the identification of policy challenges and objectives, as well as
UNECE trainings and capacity building seminars.
• The policy dialogue mentioned above at the national and international level also
contributed to an enhanced understanding of policy options.
• The synergies between UNECE’s intergovernmental work, technical assistance, and
capacity building were crucial for achieving this project objective. The secretariat studies
best international practices and success stories, implements them in less developed member
States, and disseminates lessons learned in the process to other member States, feeding the
foundation of knowledge UNECE have built over the years.
Project objective c) Enhanced national implementation of UNECE policy recommendations
• The project managed to accomplish many concrete policy change in the areas with potential
and demand for change by creating an evidence base and capacity building. Examples
include developing and adopting a new procurement law in Georgia emphasising
innovation-enhancing procurement (IEP), technology transfer in the Republic of Moldova,
and venture capital in Belarus.
• For other countries, the actual implementation of recommendations is too early to tell.
• The project made actionable, targeted, time-bound and prioritized recommendations, and
the dialogue with national stakeholders continued to facilitate the implementation of
recommendations.
Unexpected project results included the project’s ability to bring together diverse stakeholders from
countries’ innovation ecosystems due to good preparation of the project team and local intelligence
on the ground through the use of national experts.
The project performance was affected by positive and negative factors. Positive factors influencing
project performance included the quality of the project team, UNECE's convening power, the
acceleration of using virtual technology as a COVID-19 mitigation measure, and the project
duration of over four years. Concerning the project team, UNECE selected staff from diverse
academic backgrounds and established transparent and efficient internal processes and reporting,
contributing to efficient and accountable project management.
Negative factors influencing project performance comprised meeting and travel restrictions due to
COVID-19, the volatile political situation in the sub-region, which included several armed conflicts
inflicting project countries, the economic downturn and turf battles among government
stakeholders due to unclear or overlapping internal mandates concerning innovation.
Project-related improvements in policymakers' competencies included policymakers' broadened
understanding of innovation concepts transferred from UNECE’s expertise. The project mitigated
the frequent staff turnover in beneficiary countries through a networking approach, including the
technical level in governments. The evaluation finds it too early to assess stakeholder competencies
to support environmental sustainability, gender equality, good governance, and economic growth.
Finally, the project systematically involved other United Nations (UN) and non-UN stakeholders
in the implementation, including UN country representatives, EU delegations and Swedish
embassies, WIPO, OECD, and WEF, showing good coordination with UN stakeholders and other
international partners, leading to improved donor coordination and the avoidance of duplication.
xi
Conclusions: UNECE's neutrality, expertise and convening power helped engage stakeholders
during the project implementation, enhancing knowledge and awareness about innovation policies.
The project's positive results were attributed to a proactive and engaged professional team,
flexibility, and mitigation of unforeseeable factors. The team's systematic inclusion of UN and non-
UN stakeholders benefits Swedish embassies in the project countries by raising the innovation topic
on national agendas and sharpening Sweden’s profile on this topic in the region.
Recommendations for similar future projects:
R 3: In a context where donors increasingly focus on short-term results after 12 or 24
months, UNECE should encourage Sweden and other donors to continue investing in
projects with a systems change approach, leading to the implementation of research
recommendations as a means to ensure the sustainability of results. A project duration of 5
years is recommended for projects with such an approach.
Priority: medium, next 6-12 months for new project designs.
Efficiency: the project management was highly professional and an example of excellence for
many stakeholders.
The project has been well-executed, thanks to adequate funding for mobilizing consultants and a
professional project team within UNECE, showing an outstanding performance. The project team
used international experts and tapped into a network of well-connected national experts.
Implementing a focal point approach, which entails centralized coordination in each country,
streamlined communication, efficient organization, and better collaboration among stakeholders,
while catalysing the project implementation.
While COVID-19-related restrictions and armed conflicts in the region affected the project
implementation, requiring no-cost extension, the combination of timely instructions, guidelines,
and feedback from an organized project team set a solid foundation for success in the project.
Overall, the project duration for work on behavior change required 4 to 5 years, as it was complex
and involved significant efforts to modify people's attitudes, habits, and behaviors.
The evaluation finds high resource use efficiency, for example, during the COVID-19 pandemic.
By utilizing technology, such as video conferencing and online training tools, it became possible
to connect with a diverse range of individuals and groups remotely, reaching even more
stakeholders than during in-person visits. Also, the involvement of local experts was crucial in
energizing local stakeholders, particularly during challenging times like a pandemic. Rapid learning
in an unprecedented and stressful situation enabled the project team to do many more tasks virtually
and combine regular and frequent online meetings with missions, increasing efficiency further in
future projects.
Conclusions: The project was good value-for-money due to a professional and highly qualified
team, efficient implementation, local expert use, and centralized coordination through national
focal points during pandemic-related travel restrictions. UNECE even executed deliverables not in
the donor agreement using existing funds.
xii
R 4: Building on the good practices of this project, using local experts to gather intelligence
on the ground is recommended for similar future projects at the country level. At the same
time, national focal points should continue to be appointed for centralized project
coordination in project countries.
Priority: medium, next 6-12 months, for new project designs.
Sustainability: The evaluation finds that, as the project is still ongoing, results are mixed and
in different stages across project countries, thus making further continuous follow-up
necessary to solidify the results.
As the project streams ended, countries remained highly engaged to collaborate on the
implementation of policy recommendations and support national strategies and initiatives.
Ownership of results, institutionalization, and up-scaling are evident in Armenia, Azerbaijan,
Georgia, and the Republic of Moldova. However, the evaluation finds that governments still require
close support for national strategies and policies is required, including during the review and
updating of the latter, for example, in Armenia or the Republic of Moldova.
The evaluation finds that the project's methodology can be replicated in other sub-regions and
countries, with some amendments to the current approach and always considering sub-regional
contextualization. The upscaling could be undertaken with little cost, given the initial investment
in this project for improving the I4SDR review methodology. UNECE also learned much about the
operational side of running sub-regional projects, organizing dialogue sessions and consultations,
and disseminating findings and conclusions. This similarly makes scaling the project up or adapting
it to other regions cost-effective, considering the expertise accumulated within UNECE over the
past years.
Conclusions: Stakeholders demonstrate a strong buy-in and interest in the project’s
recommendations, institutionalizing them in many countries, but require continued external support
for innovation strategies and policies. The project approach is fit for purpose and ready to be
replicated in other sub-regions.
Recommendations for similar future projects:
R 5: As a follow-up to this project, UNECE should use its limited regular budget resources
to monitor the implementation of recommendations and keep engaging with the network
of focal points to share good practices for developing and implementing national
innovation-related strategies and policies.
Priority: very high, next 3-6 months for new project designs.
R 6: Senior management should use this evaluation report as a robust evidence base to
lobby for replicating the project approach in other sub-regions, for example, the Western
Balkans, with high relevance for donors like the EU and Sweden. This should be done, as
explained in R 1 by i) seeking continuation funding to support further EESC countries
(especially for ensuring sustainability); ii) seeking new funding to replicate for other
regions, such as the Western Balkans or Central Asia.
Priority: very high, next 3-6 months for new project designs.
xiii
Figure 1: Infographic - Overview of the project and main evaluation results
xiv
The evaluation found that the project delivered or “over-delivered” on all seven expected
accomplishments listed in the project proposal.
Expected Accomplishment Deliverable Status Additional deliverables not included in the initial
work plan
A1.1. Develop the methodology
for a pilot sub-regional
Innovation Policy Index (IPI)*
and apply it to six countries
(Belarus, Ukraine, Azerbaijan,
Armenia, Georgia, the Republic
of Moldova). The resulting data
and the analysis will be
published together with a range
of analytical chapters by ECE
and a range of contributors that
examine the results through a
wider analytical and regional
perspective. The exact topics
will be selected based on the
leading concerns that come
through in the process or
through discussions in UNECE
intergovernmental meetings.
*Please note that the Innovation
Policy Index (IPI) was renamed
to Innovation Policy Outlook
(IPO)
Completed – the IPO
was completed in early
2020 and launched at six
national high-level
meetings and a sub-
regional event for a total
of seven launches. All
the national-level
meetings gathered
attendance from Deputy
Ministers, representatives
of international
organizations, and the
Swedish Ambassadors to
the six countries.
Additional accomplishments: IIPO (2022), the
follow-up interim publication, was completed in
2022.
It was informed by policy dialogues with IPO focal
points that were organized in follow-up to the initial
publication and addressed the topics requested by
the IPO countries – innovation-enhancing
procurement and science-business linkages.
The IIPO was launched at the 16th session of the
Committee on Innovation, Competitiveness and
Public-Private Partnerships in front of 227
attendees from member states including Albania,
Armenia, Azerbaijan, France, Georgia, Germany,
Greece, Italy, Kazakhstan, Kyrgyzstan,
Montenegro, North Macedonia, Portugal, the
Republic of Moldova, Russian Federation,
Tajikistan, Turkey, Turkmenistan, Ukraine, and
Uzbekistan, as well as the European Union.
Representatives from non-ECE Member States such
as Brazil, Egypt, Libya, and several specialized
agencies in the United Nations system, including
UNICEF, UNCTAD, ESCAP, UNEP, UNDP,
UNIDO, and UNHCR, also attended the launch.
A1.2. Conduct National
Innovation for Sustainable
Development Review of
Georgia
Completed – the I4SDR
of Georgia was launched
in late 2020 at a high-
level event attended by
the Deputy Minister of
Foreign Affairs, Deputy
Chairperson of the State
Procurement Agency,
and the Deputy
Chairperson of
Sakpatenti. The review
inspired follow-up
capacity building.
Additional accomplishments: Handbook on
Innovation Enhancing Procurement for Georgia
In addition to the I4SDR and in follow-up to the
request from the Government of Georgia, UNECE
published a handbook on Innovation Enhancing
Procurement (IEP) for Georgia. This was followed-
up by a dedicated study tour on IEP with Georgian
officials from the SPA Georgia in Norway.
A1.3. Conduct National
Innovation for Sustainable
Development Review of the
Republic of Moldova
Completed Additional accomplishments: In 2022, UNECE
developed a technology transfer roadmap for the
Republic of Moldova.
The roadmap was requested by the Government of
the Republic of Moldova to assist in the
development of the new national programme on
research and innovation 2024 – 2027. The roadmap
was complemented with three capacity building
trainings with innovation stakeholders in the
country. Though not part of the project agreement,
the roadmap and the trainings were developed in
xv
response to the I4SDR review and supported the
implementation of the I4SDR recommendations.
A1.4. Conduct National
Innovation for Sustainable
Development Review of
Armenia
Completed Additional accomplishments: I4SDR follow-up
roadmap
UNECE is in the process of developing a roadmap
of future cooperation with Armenia
A2.1. Conduct twelve tailored
to the specific demands of the
countries advisory missions
(four per each country) on
specific policy reforms in the
innovation sector in Belarus,
Georgia, the Republic of
Moldova. Capacity-building
agreements including detailed
activity plans supporting
specific reform efforts to put
recommendations from national
reviews into practice will be
developed together with each
beneficiary country.
Completed and
overdelivered (13/12;
16/12 if Armenia and
Ukraine are included)*
• 4/4 Belarus
• 5/4 Republic of
Moldova
• 4/4 Georgia
*Several in-person
advisory missions were
suspended due to the
pandemic, so had to be
moved online.
Additional accomplishments: advisory missions to
Armenia (2022,2023) and a study tour for Ukraine
In addition to the three countries listed in the
project agreement, UNECE also conducted two
advisory missions to Armenia. The last advisory
mission to Armenia involved the presentation of the
I4SDR results and implications to the Minister of
Education, Science, Culture, and Sports, the
Minister of High-Tech Industry, the Deputy
Minister of Economy, and the Deputy Prime
Minister at high-level bilateral meetings with the
Executive Secretary of UNECE.
In addition to the three listed countries, UNECE
also organized a study tour for a Ukrainian
delegation in 2022 within the framework of ToS-
ICP to discuss post-war reconstruction (please see
below)
A3.1. Conducting six sub-
regional capacity building
workshops on supporting high
growth innovative enterprises
and related topics, most of
which will be held in the
beneficiary countries, with one
or two exceptionally in Geneva
in connection with CICPPP and
ToS-ICP sessions). For each
subregional meeting a
substantive background
document will be developed
that will feed into an English-
Russian language policy
handbook that will be available
for all ECE member States.
Completed and over-
delivered (7/6)
Please note: All meetings
were conducted online
due to COVID
Additional accomplishments: as of August 2023,
UNECE conducted seven IPO workshops.
As mentioned above, these workshops eventually
fed into another interim IPO publication requested
by member States.
A3.2. Prepare a policy
handbook on high-growth
innovative enterprises (in
English and Russian).
Completed Additional accomplishments: none
xvi
Though not part of the donor agreement, the activities below were delivered within the same budget
in response to member States’ demands.
Innovation for Sustainable
Development Review of Ukraine
UNECE is conducting an Innovation for Sustainable Development
Review of Ukraine with the explicit aim to support Ukraine’s
recovery and reconstruction efforts.
Innovation Matters Podcast To bolster its outreach efforts and create a public learning platform
on innovation, UNECE launched a podcast series titled
“Innovation Matters” in 2022. Currently, the series is at its 11th
episode.
More information: https://unece.org/eci/icp/innovation-matters-
podcast
Ukraine Study Tour In November 2022, the Economic Cooperation and Trade
Division organized a study tour for Ukrainian officials from the
Ministry of Education and Science, Ministry of Economy, and
National Academy of Sciences to Geneva. During the tour, the
participants discussed the role of innovation policies in supporting
a green and circular recovery, exchanged knowledge with experts,
and attended high-level discussions with international
organizations including UNCTAD, WIPO, and UNEP. The
participants also met with UNECE Executive Secretary, Olga
Algayerova, to explore UNECE's capacity building and policy
development support. The tour was part of capacity-building
efforts to support Ukraine’s reconstruction and allowed focal
points to partake in international dialogue.
Outreach missions to Brussels and
Rome to raise awareness of the
project and build partnerships
UNECE went on two outreach missions, one to Brussels and
another to Rome. During the Brussels mission, we met with
officials from the European Union’s Joint Research Centre,
Horizon Europe and DG Near to discuss the new UNECE project.
We discussed cooperation on future reviews and agreed to
exchange information and review each other’s publications to
avoid duplication of efforts. In Rome, UNECE met with
stakeholders from the Italian Ministry of Foreign Affairs and
International Cooperation (MFAIC) and the Italian Agency for
Development Cooperation (AICS). UNECE presented its work
streams and exchanged knowledge on innovation-related projects
in the UNECE region.
1
I Introduction
This document constitutes the evaluation report of the UNECE project E317, "Promoting
innovation policy capacities in Eastern Europe and the Caucasus" (hereafter referred to as "the
project").
1.1 Project Background
According to the project document, "the project will improve the competencies of innovation
policymakers in designing, running, reforming, and monitoring effective innovation policies and
institutions that make measurable contributions towards long-term economic development and
sustainable development."
The three expected accomplishments (EA) comprise the following:
• EA1.Enhanced capacity of national policymakers and stakeholders to analyse and
benchmark their innovation policies and institutions in line with UNECE good practices;
• EA2. Enhanced capacity of national policymakers and stakeholders in three countries to
design and carry out effective innovation policy and institutional reform;
• EA3. Improved subregional cooperation on harmonisation of innovation policies and
institutions
Main project activities comprised:
• Develop the methodology for a pilot sub-regional Innovation Policy Index (IPI) and
apply it to six countries
• Conduct National Innovation for Sustainable Development Review of Armenia, Georgia,
and the Republic of Moldova
• Conduct twelve tailored to the specific demands of the countries' advisory missions (four
per each country) on specific policy reforms in the innovation sector in Belarus, Georgia,
the Republic of Moldova
• Conducting six sub-regional capacity-building workshops on supporting high-growth
innovative enterprises and related topic
• Prepare a policy handbook on high-growth innovative enterprises (in English and
Russian).
The project aimed to address the following Sustainable Development Goals (SDGs): 5, 8, and 9.
2
1.2 Evaluation Purpose and Scope
The evaluation Terms of Reference (ToRs) outline the background of this evaluation as follows:
"to assess the extent to which the objectives of the UNECE project E317, "Promoting innovation
policy capacities in Eastern Europe and the Caucasus," were achieved. The evaluation will assess
the project's relevance, effectiveness, efficiency, and sustainability in enhancing innovation policy
capacities in Eastern Europe and the South Caucasus (EESC) countries – Armenia, Azerbaijan,
Belarus, Georgia, the Republic of Moldova, and Ukraine."1
UNECE's Innovative Policies Development Section implemented this project starting in November
2018 with an end date of May 2022 and a budget of US$ 2,215,026. The project benefitted from a
non-cost extension till June 2023, given implementation challenges during the COVID-19
pandemic.
The evaluators proposed using the internationally agreed Organisation for Economic Co-operation
and Development's (OECD) Development Assistance Committee (DAC) criteria for this
evaluation: relevance, effectiveness, efficiency, and sustainability. The impact criterion is not
considered in this evaluation, as the time lag is too short between the end of the project in the second
quarter of 2023 and the evaluation taking place in parallel. The coherence criterion is subsumed
under the relevance criterion.
Annex 1 contains the evaluation matrix for this evaluation, listing the specific evaluation questions
related to each evaluation criterion, which defines the evaluation scope further.
It was agreed with the donor that the evaluation would be concluded shortly before the end of the
project, and cover the period November 2018 (project beginning) to September 2023. The scope
includes all six project countries, with Armenia and Georgia selected for a field visit in June 2023.
This document constitutes the evaluation report of the UNECE project E317, "Promoting
innovation policy capacities in Eastern Europe and the Caucasus" (hereafter referred to as "the
project"). It was produced by Dr. Tania Tam, who is a senior international and independent
development project evaluator. She was selected by the Programme Management Unit, which is an
oversight body that ensures overall coordination of UNECE programme management (planning,
monitoring, reporting and evaluation) of all UNECE activities funded from regular and extra
budgetary resources.
1.3 Sampling strategy
The evaluator evaluated all activities under this project, covering the six project countries: Armenia,
Azerbaijan, Belarus, Georgia, the Republic of Moldova, and Ukraine.
Armenia and Georgia were selected for more detailed examination, as case studies, in this project
evaluation. The six project countries (Armenia, Azerbaijan, Belarus, Georgia, the Republic of
Moldova, and Ukraine) were ranked (1-5) according to their demonstrated interest in participating
in project activities. Armenia and Georgia were the top-scoring countries, scoring 5 points each,
repeatedly expressing their interest in meetings and making formal requests.
1 UNECE, 2023: Evaluation of UNECE project E317 “Promoting innovation policy capacities in Eastern Europe and
the Caucasus”. Terms of Reference, page 2.
3
The six countries were also ranked on whether they had recent similar innovation reviews. Armenia
and Georgia also ranked most highly on this ranking, scoring 5 points each. Neither Armenia nor
Georgia had conducted an innovation review of similar scope in the past five years.
Armenia and Georgia, see Figure 2, were thus chosen for closer examination in the project, and
therefore field visits to these countries were organized to produce a detailed review of the project
there.
Figure 2: Map of the South Caucasus with countries visited during the evaluation: Armenia and Georgia
Background on Armenia:
Armenia has made noteworthy progress towards achieving innovation-led, sustainable
development. Despite the challenges posed by regional and geopolitical instability and the COVID-
19 pandemic, Armenia retains a competitive information and communication technology (ICT)
sector and a thriving entrepreneurship scene. Well-developed tourism, mining, food processing and
agriculture sectors complement pockets of innovation excellence in ICT.
Despite these successes, Armenia still faces challenges in sustaining economic growth and social
development. Innovation, or systematic experimentation with new ideas, processes, and products,
can be the catalyst for solving these challenges and bolster Armenia's sustainable development.
Improvements in innovation policy through intergovernmental coordination, greater use of
evidence and evaluation in policymaking, diaspora involvement and revaluation of the current
innovation infrastructure mechanisms are necessary to foster innovation. This project is critical to
addressing these constraints and supporting Armenia on its journey of economic development.
Background on Georgia:
A lower-middle-income economy at the crossroads of Eastern Europe and Western Asia, Georgia
is an innovation achiever, gaining high scores in the Global Innovation Index. The Georgia
Innovation and Technology Agency was instituted in 2013 to coordinate all science, technology,
and innovation policy aspects. Critical challenges remain, however, especially in further reforms
of institutional project reviews and education.
Project activities aim to contribute to ongoing or planned reform efforts that address central
problems that hold innovation systems back in Georgia. They aim to do so by developing
assessment tools that help policymakers and other stakeholders understand and benchmark
innovation policies and institutions. This approach is coupled with a targeted capacity building that
Black Sea Georgia
Azerbaijan Turkmenistan
Armenia Caspian Sea
Turkey
Iran
Kazakhstan
4
feeds into specific reform efforts or addresses areas of joint concern. Georgia has made significant
strides in adopting legislative and policy reforms to foster gender equality, for example.
1.4 Evaluation Methodology
For this evaluation, the evaluator used a theory-based evaluation methodology to address the time
lag between the project activities, such as capacity building, data collection and analysis, and, for
example, changes in policy capacities.
The approach was successfully used in recent evaluations for international organizations, including
the UN Secretariat. A theory-based evaluation specifies the intervention logic, also called the
"theory of change," tested in the evaluation process. The theory of change is built on a set of
assumptions around how the project designers think a change will happen. Logically, it is linked to
the project's results framework contained in the project document.
Figure 3 outlines the theory-based evaluation approach using a concept developed by the University
of Wisconsin.
Figure 3: Concept of theory-based evaluation
5
The evaluation reached 74% of the project stakeholders identified by the project team, 49 in total,
through personal interviews during a field visit to Armenia and Georgia (11), telephone interviews
(20) and an online survey (18). Personal and telephone interviews also contained quantitative
survey questions, as in the online survey. Twenty of the 49 interviewees were female (41%), and
29 were male (59%). The field visits to Yerevan and Tbilisi took place in the week beginning June
5, 2023. An online survey launched in mid-July yielded a response rate of 64%, based on the 18
responses received.
The evaluation reach is very satisfactory, given other UN Secretariat evaluation response rates
range between 15% and 30%.
Figure 4: Overview of project stakeholders and interviews accomplished
Stakeholder mapping was conducted by identifying potential stakeholders based on their
involvement in the project. They were then categorized to ensure the identification of inequalities
(e.g., gender equality) to prioritize engagement strategies and time accordingly.
Through the Programme Management Unit, 46 direct project stakeholders were suggested from
project countries, partners, and independent experts. All were included in this evaluation. Ultimate
project beneficiaries, such as enterprises benefitting from innovation policies, were not included in
the stakeholder list, given that the project was still under implementation at the time of the
evaluation. For the online survey an additional 28 of stakeholders were suggested.
The box below lists the suggested tailored evaluation tools and processes for the project evaluation.
This mixed-methods approach aims to ensure rigorous triangulation of data. The full evaluation
matrix is presented in Annex 1.
Number of interviews accomplished/number of
stakeholders contacted
UNECE 17/17
Intergovernmental organizations 2/2
International experts 5/15
Armenia 11/11
Azerbaijan 2/3
Belarus 1/2
Georgia 8/8
Republic of Moldova 6/6
Ukraine 3/3
Others 7
Total 49/74
6
1.5 Evaluation questions
The evaluation matrix in Annex 1 specifies which data collection methods are used for the
specific evaluation questions and shows the triangulation approach for each question. The
evaluation questions were as follows
1. Relevance: Was the project responding to stakeholders' needs?2
1.1. To what extent was the project design appropriate or meeting the needs of beneficiary
countries?
1.2. To what extent was the project aligned with the SDGs?
1.3. What are the takeaways for ensuring the relevance of future UNECE projects?
1.4. How have gender, human rights, and disability perspectives been integrated into the
project? What results have there been in terms of gender, human rights, and disability?
1.5. How can gender and human rights perspectives be better included in future project
design and implementation?
2. Effectiveness: Were project results achieved, and how?
2.1. To what extent were the project objectives achieved?
• Improved policy dialogue on promoting sustained economic growth, innovative
development, and greater competitiveness in the UNECE region.
2 It is suggested to suppress the original evaluation question number 2 listed in the Terms of Reference “To
what extent did the project respond to the priorities and needs of the participating countries? How relevant
were they to the countries’ needs and priorities?”, as it is very similar to evaluation question 1.
o Desk review: the project document and evidence of results achieved such as
monitoring and other progress reports;
o Briefing calls with the project team and the UNECE Monitoring and
Evaluation Unit
o 45-60 minute virtual interview meetings mainly with relevant national
policy makers, practitioners, primary project partners such as the donor,
academic partners, and independent experts;
o Field visit to Armenia and Georgia
o Online-survey
o Presentation of emerging evaluation findings following data analysis to the
project team, UNECE Monitoring and Evaluation Unit, and the donor;
o Draft report for feedback to the Programme Management Unit (quality
assurance);
o Finalization of the evaluation report.
7
• Improved understanding at the national level of policy options to promote sustained
economic growth, innovative development, and greater competitiveness; and
• Enhanced national implementation of UNECE policy recommendations and
standards on promoting a policy, financial and regulatory environment conducive to
sustained economic growth, innovative development, and greater competitiveness.
2.1.1. Beyond these objectives, what has been your strategic vision for the project, and
how has it changed over the course of the project?
2.1.2. What (if anything) has prevented the project from achieving the desired results?
Which factors facilitated or hindered the achievement of the project results? What
approaches worked well and could be adapted to work in other sub-regions? What
are some relevant lessons learned?
2.1.3. Did the project deliver any unexpected/unplanned results?
2.2. How has the project improved the competencies of innovation policymakers in the
participating countries to design, develop, implement, reform, and evaluate national
innovation policies?
2.3. How effectively have the project activities been coordinated (e.g., peer review and
information exchange) and integrated (e.g., how the IPO complemented other sub-
regional reviews by filling in a gap) with those of other partners, particularly within the
context of other UN system entities
2.4. How did the challenges affect the project and impact the achievement of the expected
project objectives?
2.4.1. How successfully did the project overcome these?
2.5. How effectively has the project improved the competencies of innovation policymakers
to support environmental sustainability, gender equality, good governance, and
economic growth?
2.6. What are the lessons learned?
3. Efficiency: Were resources used appropriately to achieve results?
3.1. Were the resources adequate for achieving the results?
3.2. Were the results achieved on time, and were all activities organized efficiently?
3.3. Were the resources adequate for achieving results? Were they well used economically?
How could this be improved?
4. Sustainability: Are results lasting?
4.1. What ensures that project outcomes would continue after the project ends?
4.2. For example, to what extent do the partners and beneficiaries' own' the outcomes of the
work? How will stakeholders' engagement continue, be scaled up, replicated, or
institutionalized? How will risks be mitigated?
8
4.3. To what extent are the project's objectives (improve dialogue, improve
understanding of innovation governance, build capacity) still valid? How can the
project be replicated in other UNECE sub-regions (in particular, the Western Balkans)?
1.6 Leaving No-One Behind
This evaluation mainstreams cross-cutting issues such as gender equality, disability inclusion, the
environment and human rights as part of the Leaving No One Behind approach3. Gender equality,
disability inclusion, the environment and human rights are thus addressed in three out of the 18
evaluation questions, as presented in the evaluation matrix in Annex 1 of this Progress Report.
Those evaluation questions are as follows:
Evaluation question 1.4: To what extent were gender, human rights, and
disability perspectives integrated into the design and implementation of the
project?
Evaluation question 1.5: What results can be identified from these actions?
Evaluation question 1.6: How can gender and human rights perspectives be better
included in future project design and implementation?
Gender equality, disability inclusion, the environment and human rights were considered in the
document review. In addition, in the stakeholder mapping, particular attention was given to
identifying any disadvantaged groups (e.g., gender), existing inequalities and unjust power relations
that have influenced the project, in line with UNEG Evaluation Standards.
1.7 Limitations
There was an initial delay in contracting the evaluation. However, this was mitigated by the timely
support of the project team.
All stakeholders with substantial knowledge of the project were invited to participate in the
interviews and surveys. 49 out of 74 stakeholders participated in the end. Thus, not all of the
selected stakeholders were available for interviews. As noted previously, 34% of the chosen project
stakeholders did not reply to the evaluator's invitation for an interview.
Although this number of quantitative survey respondents (N=49) is still low for generalizability,
despite mitigating this shortcoming through an online survey launched towards the end of the
evaluation period, the qualitative semi-structured interviews were designed to dovetail with the
quantitative survey questions in this mixed approach to balance this limitation and to provide a
more robust description in painting a fuller picture of project results.
3 The project did not specifically target disability in its project design
9
II. EVALUATION FINDINGS
2. Relevance and coherence: Was the project doing the
right thing?
This section addresses the evaluation criteria of relevance and coherence. The sub-criteria used
comprise i) the appropriateness of project design and ii) the integration of gender, human rights and
disability perspectives. This section's principal sources of evidence are the document review, virtual
interviews, the online survey, and personal interviews during the field visit.
Key findings: The project was highly relevant to the six project countries, despite conflicts, ongoing
political instability, and the pandemic since the project started in 2018.
• The UNECE project promoted a regional perspective, filling a critical gap and serving to shape national
innovation strategies.
• Project components were relevant, as the European Innovation Scoreboard, the Global Innovation Index,
and other composite indices were based on output and input indicators, rather than the essential issue of
innovation governance and the particular features, challenges, and potential typically shared among
transition economies in general and among EESC countries in particular.
• Similarly, the sub-regional approach in most activities was highly relevant, given shared challenges and
legacies.
• The focus on innovation governance not only filled an important gap, but also addressed the much
neglected issue of monitoring and evaluation of and value-for-money from public spending and support
instruments – essential also because of fiscal constraints growing, especially in the wake of the pandemic
induced slump.
• The project addressed poor coordination between national stakeholders, exacerbated by frequent staff
changes in government agencies in all project countries.
• As a Geneva-based body with no regional offices, UNECE provided analytical input and recommendations
to guide on-going or planned donor-funded activities operating on the ground.
• The Leave No One Behind approach promoted by the United Nations Secretariat was taken very seriously
by the project designers and during its implementation.
• However, these cross-cutting issues were not as clearly reflected in governments’ needs and priorities and
results were limited.
• Greater awareness and education seem required among stakeholders about the importance and benefits of
integrating gender, human rights and disability perspectives into project design and implementation.
• The project provided visibility to Sida and UNECE, transferred large numbers of relevant international
good policy practices to the region (including lessons learned from Sweden’s innovation development
journey), and worked closely with all relevant stakeholder groups and to the enhance the region’s STI
ecosystem.
• The project successfully improved the understanding of policy options at the national level based on
research and the identification of policy challenges and objectives, as well as UNECE training, and
capacity-building seminars.
10
2.1 Appropriateness of project design for meeting the needs of
beneficiary countries
The evaluation finds that the project design was relevant for the six beneficiary countries, as the
European market is very competitive, requiring a focus on market niches, trade, and service
provision through innovation. Hence, the demand was high for the project from the transition
economies with historical pockets of excellence but a need for institutional strengthening to make
this happen. Governments needed to include the private sector to strengthen the ecosystem for
innovation.
The UNECE project gave visibility to the region, with Sida filling a critical gap at the time when
the project countries prepared national innovation strategies.
Also, the project provided the basis for peer learning and sharing technical experience despite
political tensions among some project countries. One example is the lively community of project
country focal points.
As a Geneva-based body with no regional offices, UNECE provided analytical input using best
practice recommendations to guide other organisations operating on the ground. For example, in
the I4SDR of Ukraine, currently at the drafting stage and to be published in 2024, a chapter will be
analyzing the current reconstruction strategies and suggesting improvements to these strategies.
Hence, the project efforts guide the larger international community in working within the EESC
region on the niche topic of innovation governance.
The project's needs assessment was one of the highly relevant project components. The project
managed to engage policy makers with designated focal points and other innovation stakeholders,
Non-Governmental Organizations (NGOs), the private sector, business cluster and incubator
representatives, and pertinent individuals, to evaluated their needs and experiences. The evaluation
finds that this approach was essential to bringing out the added value of the IPO and best practices
in the region. This multi-stakeholder approach often brought together stakeholders engaged in
innovation for the first time, for example, in Armenia.
At the same time, the region has experienced dramatic political changes since the start of the
project, which further enhanced the demand for innovation: the COVID-19 pandemic, supply chain
resilience, the trade conflict between the US and China, public unrest in Belarus and the war in
Ukraine, which also affects the project country Republic of Moldova.
The following paragraphs assess the relevance of specific project activities:
• Methodology for a pilot sub-regional Innovation Policy Outlook (IPO);
• Advisory missions (four per each country) on specific policy reforms in the
innovation sector in Belarus, Georgia, and the Republic of Moldova;
• Sub-regional capacity-building workshops on supporting high-growth innovative
enterprises and related topics;
The evaluation found that the above project components were relevant, as the European Innovation
Scoreboard was less relevant for the transition economies, requiring a more sub-regional approach
to IPO.
11
Project countries had to catch up to reach Western European levels of competitiveness, but also
required sub-regional benchmarks of similar countries with a similar recent history. Learning from
neighbours’ successes and pitfalls through capacity-building workshops was highly relevant. South
Caucasus countries were particularly interested in learning from Eastern European project
countries, the latter being more advanced in innovation.
• National Innovation for Sustainable Development Reviews of Armenia, Georgia, and the
Republic of Moldova;
The project addressed poor coordination between national stakeholders, exacerbated by frequent
staff changes in government agencies in all project countries. The identification of gaps between
policy recommendations and implementation was of particular value.
Similarly, the updated review methodology not only enabled the process to run smoothly and the
product to be coherent in analysis and recommendations, but provided substantial space for in-
depth scrutiny of country-specific areas of political and economic importance. These included
innovation-enhancing public procurement in Georgia and diaspora engagement in Armenia – both
areas of substantial underused potential.
Although typically innovation centres on capitals and well-developed regions, the project
recognized and underscored the importance of diffusion of ideas, especially for using digital
technology, to the rest of the country and the economy. This includes absorptive innovation to
boost agricultural productivity and compliance with EU requirements for the trade of agricultural
products.
• Policy handbook on high-growth innovative enterprises (in English and Russian).
Awareness about the handbook was uneven across the project countries, which seems particularly
limited in the South Caucasus. While policymakers quote the handbook in Belarus, where it is
considered a good benchmark, in Ukraine, the handbook and the trainings triggered momentum in
Ukraine to address the IHGE angle in a more targeted way, while an official definition of high-
growth innovative enterprises is yet to be developed.
The evaluation found common capacity challenges in project countries to work with research
results for innovation policy drafting and implementation.
Figure 5 summarizes project stakeholder perceptions concerning the pertinence of the project
meeting their needs. The quantitative results are positive, with 52,1% very high (very much so) and
26,5% high ratings (mostly).
12
Figure 5: Stakeholder perception about the relevance of the project responding to their needs
n=49
2.2 Integration of gender, human rights, and disability perspectives
The evaluation found that the Leave No One Behind approach promoted by the United Nations
Secretariat was taken very seriously by the project designers and during its implementation. This
prioritization was not reflected in governments' needs and priorities.
In the project design and implementation, gender, human rights, and disability perspectives were a
fundamental part of the project discussed at the methodology level and during policy analysis. In
the section on gender equality, for example, countries evaluated policy framework and
programming against it, resulting in a gap analysis to identify a need for policy reform in the I4SDR
reports.
The quotes below highlight some voices about the relevance of the gender, human rights, and
disability perspectives.
52,1
26,5
7,1
0,0 0,0
14,3
0,0
10,0
20,0
30,0
40,0
50,0
60,0
Very much so Mostly Somewhat A little Not al all No answer
Pe
rc
en
ta
ge
“The focus of this project was science and technology. Hence I did not put emphasis on
gender in the countries I worked for the project”.
“These were covered in the IPO research and report. However, these topics are generally
pushed by international organizations and less by in-country government”.
“Gender and disability are not well connected when addressing innovation policy, which is
targets every person anyway”.
“Rights to economic development are relevant, e.g. for IDPs [Internally Displaced Person]
and refugees outside the country”.
Sources: Project stakeholders across project countries
13
2.2.1 Gender, human rights, and disability perspectives: project
results
Overall, stakeholders had challenges finding results related to gender, human rights, and disability,
all being part of the UN's Leaving No One Behind agenda.
In Armenia, one recommendation on involving the diaspora in the country's innovation ecosystem
related to capacity building and making them welcome in case they want to return, considering the
needs of families with small children requiring preschool and elementary education.
In Georgia, the e-Procurement platform is not yet fully accessible for reading-impaired persons,
with work progressing. The importance of technology for people with disabilities is recognized, as
it can provide them with better job opportunities and intervention. The country benefits from an
innovation agenda specifically dedicated to disabled individuals, though the number of programs
in place is still limited.
2.2.2 Potential including gender and human rights perspectives
better in future project design and implementation
In order to better incorporate gender and human rights perspectives in future project design and
implementation, several steps can be taken:
o Research Questions: Both the IPO and I4SDRs had gender-specific indicators. However, it is
crucial to expand gender and human rights perspectives in the project's research questions
for the IPOs. The project can gather relevant data and information to inform its design and
implementation by explicitly addressing these issues. Subsequently, the use of statistical data
can help ensure that gender and human rights perspectives are included. By analyzing
relevant statistical data, such as gender-disaggregated data and human rights indicators,
project planners can better understand the specific challenges and needs faced by different
groups and incorporate them into the project design.
o Donor Presence: Donors are vital in promoting gender and human rights perspectives in
project implementation. Their involvement and funding can signal the importance of these
issues and encourage project stakeholders to prioritize them.
o Government Engagement: It is important to signal to the government the significance of
addressing gender and human rights perspectives in project design and implementation.
Incentives can be provided to encourage government officials to focus on these issues, such
as providing additional funding or recognition for projects that effectively integrate gender
and human rights considerations.
o Structural Challenges: It is crucial to acknowledge and address the structural challenges that
hinder progress in gender and human rights. For example, in some countries, there may still
be traditional and outdated views regarding the role and rights of women. By recognizing
these challenges, project planners can develop strategies to challenge and overcome such
barriers.
It is important to note that men mainly believe that prioritizing gender, disability, and human rights
perspectives might dilute the technical focus of an innovation project. Those perceptions highlight
the need for greater awareness and education among stakeholders about the importance and benefits
of integrating these perspectives into project design and implementation.
14
3. Effectiveness: Were results achieved, and how?
This section assesses the project results' achievement using the following sub-criteria: i) overview
of project objective achievement; ii) evolvement of the project's strategic vision; iii) unintended
effects; iv) factors affecting project performance, v) lessons learned, vi) changes in the competences
of innovation policymakers, vii) coordination with other UN and non-UN stakeholders and viii)
challenges and mitigation.
The principal data sources for this section are the document review, virtual interviews and
interviews during the field visits, and the online survey.
Key findings: The project accomplished two out of three objectives fully or exceedingly,
with the third one, the implementation of recommendations, while showing substantial
momentum in several cases, requires policies and institutional reforms that will take some
time and will benefit from further UNECE support.
• Project objective a) improved policy dialogue:
o The project successfully established a multi-stakeholder dialogue for developing the Innovation Policy
Outlooks (IPOs) leveraging UNECE's neutrality as a UN body, using international and local expertise
and comparing the six countries.
o The dissemination of lessons learned through UNECE’s IPO/I4SDR/CB/dialogue work at its
intergovernmental CICPPP and ToS-ICP sessions also enhanced policy dialogue, where hundreds of
member States and international organisations representatives listen in and exchange knowledge about
innovation.
• Project objective b): Improved understanding at the national level of policy options
o The project successfully improved the understanding of policy options at the national level based on
thorough research and the identification of policy challenges and objectives.
o The policy dialogue mentioned above at the national and international level also contributed to an
enhanced understanding of policy options.
o The synergies between UNECE’s intergovernmental work, analytical work, technical assistance, and
capacity building were crucial for achieving this project objective. The secretariat studies best
international practices and success stories, implements them in less developed member States, and
disseminates lessons learned in the process to other member States, feeding the foundation of
knowledge UNECE have built over the years.
• Project objective c) Enhanced national implementation of UNECE policy recommendations
o The project managed to accomplish concrete policy change in the areas with potential and demand for
change by creating an evidence base and capacity building. Examples include developing and adopting
a new procurement law in Georgia emphasising innovation-enhancing procurement (IEP), technology
transfer in the Republic of Moldova, and venture capital in Belarus.
o For other countries, the actual implementation of recommendations is too early to tell.
o The project made actionable, targeted, time-bound and prioritized recommendations, and the dialogue
with national stakeholders continued to facilitate the implementation of recommendations.
• Unexpected project results included the project’s ability to bring together diverse stakeholders from
countries’ innovation ecosystems due to good preparation of the project team and local intelligence on the
ground through the use of local experts.
• Positive factors influencing project performance included the quality of the project team, UNECE’s
convening power, the acceleration of using virtual technology as a COVID-19 mitigation measure, and the
project duration of over four years.
• Negative factors influencing project performance comprised meeting and travel restrictions due to COVID-
19, the volatile political situation in the sub-region, the economic downturn and turf battles among
government stakeholders due to unclear or overlapping mandates concerning innovation.
• Changes in policymakers’ competences: policymakers broadened their understanding of innovation
concepts. The project mitigated the frequent staff turnover the project mitigated this risk through a
networking approach, including at the technical level.
• Competences to support environmental sustainability, gender equality, good governance, and economic
growth: it is too early to assess changes at this level
• Coordination: the project systematically involved other United Nations (UN) and non-UN stakeholders in
the implementation, including UN country representatives, EU delegations and Swedish embassies, WIPO,
OECD, and WEF, showing good coordination with UN stakeholders and other international partners.
15
3.1 Overview of achievement of project objectives
This section analyses the project's achievement of its three objectives:
(a) Improved policy dialogue on promoting sustained economic growth, innovative
development, and greater competitiveness in the UNECE region.
(b) Improved understanding at the national level of policy options to promote sustained
economic growth, innovative development, and greater competitiveness; and
(c) Enhanced national implementation of UNECE policy recommendations and standards on
promoting a policy, financial and regulatory environment conducive to sustained
economic growth, innovative development, and greater competitiveness.
Project results:
(a) Improved policy dialogue on promoting sustained economic growth, innovative
development, and greater competitiveness in the UNECE region.
The evaluation finds that the project established a multi-stakeholder dialogue in the six project
countries for developing the IPOs and for subsequent, regular exchange and work on policy
progress. The dialogue was initiated through initial country missions. The variety of stakeholders
was particularly valuable for the policy dialogue, and the project increased the number of
opportunities for those stakeholders to get together. Stakeholders contributing to the policy
dialogue were relevant government agencies, academia, NGOs, business associations and some
businesses.
UNECE, being a neutral UN body, using international expertise and local know-how through
experts and comparing the six countries were leveraging factors for stakeholders to participate in
the policy dialogue. At the same time, those leveraging factors attracted media coverage of the
project.
While changes in government staff at the political level affected the dialogue, the involvement of
technical staff ensured the continuation of project activities related to dialogue in the project
countries.
The dissemination of lessons learned through UNECE’s IPO/I4SDR/CB/dialogue work at its
intergovernmental CICPPP and ToS-ICP sessions also enhanced policy dialogue, where hundreds
of member States and international organisations representatives listen in and exchange knowledge
about innovation.
This paragraph summarizes some insight into the policy dialogue at the country level. In Armenia,
the government organized policy dialogues every three months, inviting representatives from
various sectors and high-ranking experts. Those events provide a platform for the three ministries
involved in innovation to talk with each other and learn about their roles and actions.
Concerning Belarus, a valuable dialogue between stakeholders was interrupted by the reactions to
social unrest and many innovative private-sector companies leaving the country.
In the case of Ukraine, stakeholder groups like academia, NGOs, or government were previously
operating in silos, and the project contributed to more cross-fertilization through dialogue.
16
(b) Improved understanding at the national level of policy options to promote sustained
economic growth, innovative development, and greater competitiveness; and
The project successfully improved the understanding of policy options at the national level based
on thorough research and the identification of policy challenges and objectives. Interim IPOs, many
peer reviews on sub-regional and country chapters and panel discussions resulted in a good
understanding of options. The policy dialogue mentioned above at the national and international
level also contributed to an enhanced understanding of policy options. The synergies between
UNECE’s intergovernmental work, technical assistance, and capacity building were crucial for
achieving this project objective. The secretariat studies best international practices and success
stories, implements them in less developed member States, and disseminates lessons learned in the
process to other member States, feeding the foundation of knowledge UNECE have built over the
years.
Experts observed a broadened interpretation of innovation concepts in project countries, which
started with a narrow understanding of digitally oriented start-ups. This enhanced understanding
resulted in designing tax incentives and grants for industries as part of a comprehensive framework,
showing governments' capacities to formulate policy initiatives and develop action plans with
deadlines and budgets. The latter was observed, for example, in the Republic of Moldova. Despite
this enhanced understanding, stakeholders reported certain resistance to change at lower political
levels. In Ukraine, frequent turnover of staff and staff leaving for the military affected the
consistency in stakeholder engagement to enhance the understanding of policy options to promote
sustained economic growth, innovative development, and greater competitiveness.
Armenia: The project identified a significant divide in innovation policy among the three ministries
in charge of innovation: the Ministry of High Tech, the Ministry of Economy and Science and the
Ministry of Education, Culture, and Sports. The review provided a valuable opportunity to address
this issue, as there was limited communication among the ministries, resulting in overlaps and
unaddressed work. The evaluation found an improvement of understanding for policymakers, for
example, on the gaps that need to be addressed.
Azerbaijan: Policymakers enhanced their understanding of the legal framework guiding innovation
in the country and addressed any gaps identified by the project.
Belarus: The policy handbook on high-growth innovative enterprises prepared by the project was
partly used. The solid research served as a good picture of the country in 2019, before the social
unrest and the Ministry of Economy enhanced its understanding of policy options, using the results
of other countries as a benchmark.
Georgia: The visit to Norway significantly contributed to understanding the country's approach to
innovation and experiencing what innovation procurement is. This experience resulted in the
implementation of pilots ready to be scaled up. At the same time, the national procurement policy
has been adopted.
(c) Enhanced national implementation of UNECE policy recommendations and standards on
promoting a policy, financial and regulatory environment conducive to sustained economic
growth, innovative development, and greater competitiveness.
The evaluation found that the project managed to accomplish concrete policy change in the areas
with potential and demand for change by creating an evidence base and capacity building.
17
Examples include developing and adopting a new procurement law in Georgia emphasising
innovation-enhancing procurement (IEP), technology transfer in the Republic of Moldova, and
venture capital in Belarus. For other countries, the actual implementation of recommendations is
too early to tell.
The project made actionable, targeted, time-bound and prioritized recommendations, and the
dialogue with national stakeholders continued to facilitate the implementation of recommendations.
The comparison of countries in their reporting and progress made in implementing
recommendations incentivises governments to act. Also, the broad involvement of stakeholders
through the project's multi-stakeholder approach ensured the stimulation of the demand and supply
side for the implementation of UNECE policy recommendations.
In Armenia, the I4SDR Review was launched towards the end of this evaluation process, and the
level of recommendation implementation is too early to tell. Internal dynamics could affect the
implementation of recommendations in Azerbaijan, while in the Republic of Moldova and Ukraine,
the immediate priority of innovation issues for policymakers seems overshadowed by the war and
resulting political and economic tensions. Stakeholders in Ukraine were hopeful that the country's
implementation capacities and priorities might be in place after the end of the war. In Georgia, the
implementation of the national procurement policy is at its beginning. Progress in implementing
policy recommendations in Belarus was affected by the migration of many of the country's
innovative entrepreneurs who created innovation ecosystems in other countries, such as Georgia,
Poland, and the Baltic countries. In the public sector, no changes appear visible.
Overall, the 18 survey respondents provided positive feedback on the results achievement, as
presented in the figure below.
Figure 6: Perception about the achievement of project objectives
Project objective Achievement
level
Improved policy dialogue on promoting sustained economic growth, innovative
development, and greater competitiveness in the UNECE region
88,3%
Improved understanding at the national level of policy options to promote
sustained economic growth, innovative development, and greater
competitiveness
91,1%
Enhanced national implementation of UNECE policy recommendations and
standards on promoting a policy, financial and regulatory environment
conducive to sustained economic growth, innovative development, and greater
competitiveness
90,4%
n=18
18
This sub-section ends with a detailed list of expected accomplishments delivered. The evaluation
found that the project delivered all seven all seven expected accomplishments.
Figure 7: Donor Agreement work Streams delivery
Expected Accomplishment Deliverable Status Additional deliverables not included in the initial
work plan
A1.1. Develop the methodology
for a pilot sub-regional
Innovation Policy Index (IPI)*
and apply it to six countries
(Belarus, Ukraine, Azerbaijan,
Armenia, Georgia, Republic of
Moldova). The resulting data
and the analysis will be
published together with a range
of analytical chapters by ECE
and a range of contributors that
examine the results through a
wider analytical and regional
perspective. The exact topics
will be selected based on the
leading concerns that come
through in the process or
through discussions in our
intergovernmental meetings.
*Please note that the Innovation
Policy Index (IPI) was renamed
to Innovation Policy Outlook
(IPO)
Completed – the IPO was
completed in early 2020
and launched at six
national high-level
meetings and a sub-
regional event for a total of
seven launches. All the
national-level meetings
gathered attendance from
Deputy Ministers,
representatives of
international organisations,
and the Swedish
Ambassadors to the six
countries.
Additional accomplishments: IIPO (2022), the follow-up
interim publication, was completed in 2022.
It was informed by policy dialogues with IPO focal
points that were organised in follow-up to the initial
publication and addressed the topics requested by the
IPO countries – innovation-enhancing procurement and
science-business linkages.
The IIPO was launched at the 16th session of the
Committee on Innovation, Competitiveness and Public-
Private Partnerships in front of 227 attendees from
member states including Albania, Armenia, Azerbaijan,
Belarus, France, Georgia, Germany, Greece, Italy,
Kazakhstan, Kyrgyzstan, Montenegro, North Macedonia,
Portugal, the Republic of Moldova, Russian Federation,
Tajikistan, Türkiye, Turkmenistan, Ukraine, and
Uzbekistan, as well as the European Union.
Representatives from non-ECE Member States such as
Brazil, Egypt, Libya, and several specialized agencies in
the United Nations system, including UNICEF,
UNCTAD, ESCAP, UNEP, UNDP, UNIDO, and
UNHCR, also attended the launch.
A1.2. Conduct National
Innovation for Sustainable
Development Review of
Georgia
Completed – the I4SDR of
Georgia was launched in
late 2020 at a high-level
event attended by the
Deputy Minister of
Foreign Affairs, Deputy
Chairperson of the State
Procurement Agency, and
the Deputy Chairperson of
Sakpatenti. The review
inspired follow-up
capacity building.
Additional accomplishments: Handbook on Innovation
Enhancing Procurement for Georgia
In addition to the I4SDR and in follow-up to the request
from the Government of Georgia, UNECE published a
handbook on Innovation Enhancing Procurement (IEP)
for Georgia. This was followed-up by a dedicated study
tour on IEP with Georgian officials from the SPA
Georgia in Norway.
A1.3. Conduct National
Innovation for Sustainable
Development Review of
Republic of Moldova
Completed Additional accomplishments: In 2022, UNECE
developed a technology transfer roadmap for the
Republic of Moldova.
The roadmap was requested by the Government of the
Republic of Moldova to assist in the development of the
new national programme on research and innovation
2024 – 2027. The roadmap was complemented with
three capacity building trainings with innovation
stakeholders in the country. Though not part of the
project agreement, the roadmap and the trainings were
developed in response to the I4SDR review and
19
supported the implementation of the I4SDR
recommendations.
A1.4. Conduct National
Innovation for Sustainable
Development Review of
Armenia
Completed Additional accomplishments: I4SDR follow-up roadmap
UNECE is in the process of developing a roadmap of
future cooperation with Armenia
A2.1. Conduct twelve tailored
to the specific demands of the
countries advisory missions
(four per each country) on
specific policy reforms in the
innovation sector in Belarus,
Georgia, Republic of Moldova.
Capacity-building agreements
including detailed activity plans
supporting specific reform
efforts to put recommendations
from national reviews into
practice will be developed
together with each beneficiary
country.
Completed and
overdelivered (13/12;
16/12 if Armenia and
Ukraine are included)*
• 4/4 Belarus
• 5/4 Republic of
Moldova
• 4/4 Georgia
*Several in-person
advisory missions were
suspended due to the
pandemic, so had to be
moved online.
Additional accomplishments: advisory missions to
Armenia (2022,2023) and a study tour for Ukraine
In addition to the three countries listed in the project
agreement, UNECE also conducted two advisory
missions to Armenia. The last advisory mission to
Armenia involved the presentation of the I4SDR results
and implications to the Minister of Education, Science,
Culture, and Sports, the Minister of High Tech Industry,
the Deputy Minister of Economy, and the Deputy Prime
Minister at high-level bilateral meetings with the
Executive Secretary of UNECE.
In addition to the three listed countries, UNECE also
organised a study tour for a Ukrainian delegation in 2022
within the framework of ToS-ICP to discuss post-war
reconstruction (please see below)
A3.1. Conducting six sub-
regional capacity building
workshops on supporting high
growth innovative enterprises
and related topics, most of
which will be held in the
beneficiary countries, with one
or two exceptionally in Geneva
in connection with CICPPP and
ToS-ICP sessions). For each
subregional meeting a
substantive background
document will be developed
that will feed into an English-
Russian language policy
handbook that will be available
for all ECE member States.
Completed and over-
delivered (7/6)
Please note: All meetings
were conducted online due
to COVID
Additional accomplishments: as of August 2023,
UNECE conducted seven IPO workshops.
As mentioned above, these workshops eventually fed
into another interim IPO publication requested by
member States.
A3.2. Prepare a policy
handbook on high-growth
innovative enterprises (in
English and Russian).
Completed Additional accomplishments: none
Figure 8 summarizes the results of additional work streams that do not pertain to any of the
deliverables in the initial donor agreement. Though not part of the donor agreement, these projects
were delivered within the same budget in response to member States’ demands.
20
Figure 8: Additional work streams
Innovation for Sustainable
Development Review of Armenia
UNECE completed an I4SDR of Armenia and launched it in 2023 at a
high-level event attended by the UNECE Executive Secretary, Minister
of High-Tech Industry, Deputy Minister of Economy, Deputy Chairman
of the Science Committee, Ambassador of Sweden to Armenia and the
Interim United Nations Resident Coordinator.
Innovation Matters Podcast To bolster its outreach efforts and create a public learning platform on
innovation, UNECE launched a podcast series titled “Innovation
Matters” in 2022. Currently, the series is at its 11th episode.
More information: https://unece.org/eci/icp/innovation-matters-podcast
Ukraine Study Tour In November 2022, the Economic Cooperation and Trade Division
organized a study tour for Ukrainian officials from the Ministry of
Education and Science, Ministry of Economy, and National Academy of
Sciences to Geneva. During the tour, the participants discussed the role
of innovation policies in supporting a green and circular recovery,
exchanged knowledge with experts, and attended high-level discussions
with international organizations including UNCTAD, WIPO, and
UNEP. The participants also met with UNECE Executive Secretary,
Olga Algayerova, to explore UNECE's capacity building and policy
development support. The tour was part of our capacity-building efforts
to support Ukraine’s reconstruction and allowed our focal points to
partake in international dialogue.
Outreach missions to Brussels and
Rome to raise awareness of our work
and build partnerships
UNECE went on two outreach missions, one to Brussels and another to
Rome. During the Brussels mission, we met with officials from the
European Union’s Joint Research Centre, Horizon Europe and DG Near
to discuss the new UNECE project. We discussed cooperation on future
reviews and agreed to exchange information and review each other’s
publications to avoid duplication of efforts. In Rome, UNECE met with
stakeholders from the Italian Ministry of Foreign Affairs and
International Cooperation (MFAIC) and the Italian Agency for
Development Cooperation (AICS). UNECE presented our work streams
and exchanged knowledge on innovation-related projects in the UNECE
region.
3.2 Strategic project vision and how it evolved
The project's strategic vision evolved to different extents, showing differences between some
project countries.
In Armenia, the initial perception of the project evolved, as the project was not limited to just
support for start-ups but had a broader scope to promote sustainable development. It aimed to
identify gaps and provide recommendations, considering the interconnectedness of the innovation
system. The project focused on a systematic approach to address multiple gaps and policy reforms
rather than focusing solely on a single issue, such as creating Information Technology (IT) start-
ups. This allowed for a comprehensive systemic understanding of the innovation policy, including
financing and education aspects, and identifying the adverse effects of innovation. The review
system effectively listed the systematic problems that needed to be addressed. It took a systemic
perspective, which stakeholders appreciated as their understanding of the project evolved over time.
In Ukraine, although the overall context changed significantly after the beginning of the war in
February 2022, engagement with the project actually increased – resulting in an additional I4SD
21
review outside the commitments in the project document using left-over funds. Ukraine is
particularly interested in targeting IHGEs and experimenting with transparent and innovation-
enhancing procurement (especially important in view of the substantial increase in donor funding
and concessionary lending prospects once a certain degree of stability is secured).
3.3 Unexpected effects
The evaluation enquired about unexpected project results, and interviewees appreciated this
question, which required some level of reflection.
Those unexpected results are country-specific and specific to thematic components of project IPOs
and the I4SDR report.
Many note that they were positively surprised by the project's ability to bring together diverse
stakeholders from countries' innovation ecosystems despite institutional bottlenecks and, at times,
competing or overlapping mandates and outright turf wars. In this context, the excellent preparation
of the project team and local intelligence on the ground through the use of national experts proved
invaluable.
The participation of high-ranking politicians in some project meetings, even ministers as well as
senior staff from academia, was unexpected. The evaluation validated the finding that many young
professionals with Western Education were engaged in the project in countries like Armenia or
Georgia.
Access to data proved more difficult than expected in some project countries due to transparency
issues. Mainly, the war in Ukraine affected data availability due to security concerns and the
challenges of collecting data during a war. A spillover effect showed in Republic of Moldova,
where political priorities shifted after the beginning of the war in neighbouring Ukraine, which
resulted in political tensions in the country.
Another surprising result was that the project was able to adapt and use learnings from the
constraints to travel and physical meetings, becoming in a short time able to do virtual meetings
and even day-long trainings well, which will continue and complement physical meetings.
Finally, some stakeholders commented on the excellent cooperation between the UNECE project
and OECD activities on Small and medium-size enterprises (SME) policy in the region, with
exchange and cross-fertilization, for example, concerning the revision of the OECD's methodology.
While in Belarus, Ukraine, the Republic of Moldova, and Armenia, the export-driven IT sector was
advanced compared to other countries in the sub-region, the huge potential of innovation and
diffusion in the rest of the economy typically receive scant attention. Project activities aimed in
part at changing this perspective, and, as a result, some experts were surprised by policymakers '
interest in and private sector willingness to launch non-tech companies, not just high-tech ones.
“One of the unexpected findings was the lack of coordination in actions and low collaboration
among stakeholders. This posed significant challenges in addressing the issues related to
Innovation Policy. However, thanks to the review conducted as part of the project, we were
able to identify these gaps and take necessary actions to improve the situation.
Source: project stakeholder, Armenia
22
In Georgia, innovation-enhancing procurement was central to the review and to subsequent targeted
capacity building activities. Although a promising but nevertheless sensitive and difficult ambition
(even among the most developed EU member states), project activities have triggered substantial
momentum, including legislative modifications and pilot initiatives, in this area.
Similarly, project-funded research into the role of the diaspora in countries like the Republic of
Moldova, with a recent diaspora history, and Armenia, with a diaspora history starting over 100
years ago, revealed fascinating results4.
Migrants from lower-paid occupations, working in countries like Romania, Italy or Russia in the
case of Moldovan migrants, see opportunities to return and invest in their home countries, but with
often very limited capital. However, migrants with higher-paid occupations and more capital show
less engagement to return and invest in their home countries. The reason is that migrants with a
Western education and integrated into a new society at a middle-income level would likely
experience a reduced quality of life if they returned to their home countries. The latter can refer to
access to education, access to health or national and personal security.
In the case of Armenia, investments from the diaspora are less linked for persons with Armenian
origins to actually return to the country of their ancestors but more in the form of investments, given
that migration started around 1914 to 1923, when discord within the Ottoman Empire
caused genocide and mass migration.
3.4 Factors affecting project performance
Figure 9 summarizes the main positive and negative factors influencing the project performance,
as captured during the evaluation.
Among the positive factors were the quality of the project team, UNECE's convening power, the
acceleration of using virtual technology as a COVID-19 mitigation measure, and the project
duration.
The evaluation found in many testimonials from high-level stakeholders, for example, in the I4SDR
reports, praise for the highly efficient project team. The results of evaluation interviews endorsed
those views. Stakeholders interviewed referred to the excellent performance of a highly
professional and technically sound project team in UNECE.
4 See aslo http://agevorkyan.com/diaspora/
“The [UNECE] team's way of working was highly appreciated. They were disciplined with
their work plan. They maintained close communication and coordination with the ministries
throughout the project, which facilitated the achievement of the desired results. This approach
proved to be effective in ensuring progress and success”.
Source: Project stakeholder, Armenia
23
Besides, UNECE's strong convening power to bring together, at times competing stakeholders and
to successfully reach senior level, including ministers, was another critical success factor.
Figure 9: Factors affecting project performance
Stakeholders witnessed an unprecedented acceleration of virtual meeting technologies as the
project team's mitigation of COVID-19 travel restrictions. Those virtual tools allowed for a more
comprehensive reach of stakeholders during country consultations, particularly outside capital
cities.
Finally, the exceptional duration of over four years for a UNECE project was conducive to
implementing interventions linked to behaviour change, leading to the beginning of implementing
policy recommendations. A 5-year duration would probably have been preferable in the context of
primarily short-term UNECE projects.
Negative factors affecting the project performance included COVID-19 and resulting meeting and
travel restrictions, the volatile political situation in the sub-region, the economic downturn
following the first two limiting factors and turf battles among government stakeholders due to
unclear or overlapping mandates concerning innovation.
The pandemic hit project countries between 2020 and 2022, resulting in meeting and travel
restrictions. Given the project's use of multi-stakeholder dialogue, missions and meetings, those
restrictions significantly affected the project implementation and resulted in delays. Experts
interviewed were affected in undertaking their work, particularly the ones not having worked in
specific countries before.
24
Political instability negatively affected the project, too. The ongoing war in Ukraine, social unrest
in Belarus in 2020 and 2021, the armed conflict in South Caucasus in 2022, and government
changes, for example, in the Republic of Moldova, challenged the project implementation.
As a result of COVID-19 and the beforementioned political instability, the project countries
experienced an economic downturn. The unexpected and significant prioritization of public health
and military technology in government spending strongly competed with the government's
innovation agendas.
Overlapping ministerial mandates concerning innovation was another underlying factor that
hindered project implementation. The lack of clarity concerning mandates also contributed to turf
battles. The project's dialogue between relevant stakeholders of national innovation ecosystems
facilitated, to some extent, the clarification of the ministries’ role in the innovation context.
3.5 Lessons learned
Regarding lessons learned, stakeholder experience highlighted the importance of adaptability and
flexibility in project implementation, particularly in the face of unforeseen circumstances such as
the COVID-19 pandemic. It was crucial to have contingency plans and alternative strategies in
place to ensure progress even in challenging situations. Additionally, maximizing the involvement
of local stakeholders and experts enhanced the project's outcomes and sustainability significantly.
The exchange of experiences was very useful. More opportunities like this could have been made,
for example, with Eastern European countries or the Baltic States, which are particularly interesting
for the South Caucasus.
At the same time, it proved helpful for the ministries to come together and work together on the
national innovation agendas. From the UNECE project, national stakeholders learned about each
other's work and started with coordination efforts.
However, it would be good to have a longer-term view of such an innovation-focused project,
especially regarding implementation. For innovation to be achieved, results need a long period of
time to transpire.
3.6 Changes in competencies of innovation policymakers
Competences in designing, developing, implementing, reforming, and evaluating national
innovation policies
The evaluation found that policymakers broadened their understanding of innovation concepts.
While government change and turnover of government staff are a constant threat to
institutionalizing change, the project mitigated this risk through a networking approach, including
at the technical level in government agencies.
The I4SDR reports focused policymakers on specific issues, such as the role of the diaspora in
innovation in Armenia or Republic of Moldova. For the follow-up, a monitoring mechanism
would be required to trace the implementation of recommendations.
At the same time, the evaluation found that the lack of government support to entrepreneurs to
understand the compliance with national regulatory procedures for export certification
requirements, for example, to the EU, is one of the biggest hurdles for investment.
25
Specific changes in national policy-makers' competencies contributed to project results mentioned
earlier in this report, including the public procurement policy in Georgia, two innovation-related
laws in Azerbaijan5, discussions in parliament concerning the technology transfer law in Ukraine,
and in the Republic of Moldova, the preparation of the national road map for innovation and
technology transfer, and the ongoing review of law on innovation and technology parks.
Competencies to support environmental sustainability, gender equality, good governance, and
economic growth
Most stakeholders judged it too early to tell whether competencies for the above aspects have been
created. Overall, significant gaps seem to appear between the good understanding from
governments about requirements, for example, concerning export certification and the business
sector. The successful involvement of a United States Agency for International Development
(USDAID) supported IT association in the Republic of Moldova seemed a good practice to reach
the business sector better, but rather an exception. Stakeholders commented on the need for
policymakers to better include the pockets of excellence in the business sector in the national
innovation ecosystems.
In the case of Georgia, the project contributed to good governance through the new public
procurement law. The project has promoted good governance and economic growth in Armenia,
less in environmental sustainability and gender. The latter two topics appear to be pushed by
international organizations but lack internal demand.
3.7 Coordination with other UN stakeholders
The project systematically involved other UN and non-UN stakeholders in the project
implementation. UN country representatives, EU delegations and Swedish embassies were
consulted in all project countries.
Activities included round table talks, policy dialogue, bilateral meetings and the provision of data
and contacts.
Stakeholders also mentioned consultations with the OECD and World Bank local offices in project
countries for information exchange. Concerning multilateral development banks, the European
Bank for Development and Reconstruction and the Asian Development Bank figured among the
stakeholders consulted.
On the UN side, experts and beneficiaries referred to the engagement with agencies like UNDP,
UNCTAD, UNIDO, ILO, IOM, UNHCR or UNESCO, depending on the country context.
5
Two separate laws were drafted in Azerbaijan based on IPO results in 2022. The draft normative legal act comprising
support mechanisms for innovative projects and start ups and a draft law on innovation activities
26
3.8. Challenges and mitigation
The project encountered the following implementation challenges: i) COVID-19, ii) the war in
Ukraine and iii) social unrest in Belarus.
COVID-19 constituted a major challenge during the project implementation for most of 2020 to
2022. For thematic experts, travel to project countries was interrupted, missing observation on the
ground. Also, at least one study tour for project beneficiaries to Austria had to be cancelled. The
project team was agile in mitigating this shortcoming by engaging stakeholders remotely.
Interviews revealed that the online consultations had a much wider reach than personal visits,
reaching more persons in more diverse geographic settings of the project countries.
Armed conflict in the South Caucasus and especially, starting February 2022, in Ukraine further
challenged the reach of stakeholders in countries with an already complex operating environment.
For example, the project team extended the remote engagement with stakeholders for capacity
building. In spite of frequent staff changes at the political level, the project ensured continuity by
engaging with senior or mid-level officials.
Figure 10 summarizes stakeholder perceptions about the overall effectiveness of the project.
Ratings were very positive, with 40,8% very high ratings (very much so) and 33,7% high ratings
(mostly).
Figure 10: Stakeholder perception of the project’s effectiveness
n=49
40,8
33,7
6,1
4,1
1,0
14,3
0,0
5,0
10,0
15,0
20,0
25,0
30,0
35,0
40,0
45,0
Very much
so
Mostly Somewhat A little Not al all No answer
Pe
rc
en
ta
ge
27
4. Efficiency: were resources used appropriately to achieve
project results?
This section analyses the efficiency of the project. The following sub-criteria are used, as listed in
the evaluation matrix: i) adequacy of funding for project results; ii) timeliness of results
achievement and efficient organization; and iii) efficiency of resource use.
This section's primary data sources are the document review, virtual interviews, the online survey,
and interviews during the field visit.
4.1 Adequacy of funding for project results
The evaluation found that the project has been well-executed, thanks to adequate funding, the
involvement of experienced experts, and the outstanding performance of the project team.
The project team used international experts and tapped into a network of well-connected national
experts. Funding was adequate for mobilizing consultants and a professional project team in
UNECE.
The utilization of experienced consultants brought valuable expertise and insights to the project.
The national consultants proved invaluable during the COVID-19 pandemic as the project's
intelligence and to reach relevant national stakeholders.
Receiving the highest praise in interviews for excellent project management and proactive
communication is a significant achievement. Effective project management is essential for keeping
Key findings: Project management was highly professional and an example of excellence
for many stakeholders
• The project has been well-executed, thanks to adequate funding for mobilizing
consultants and a professional project team in UNECE showing an outstanding
performance. The project team used international experts and tapped into a network of
well-connected national experts.
• COVID-19-related restrictions affected the project implementation, requiring a non-
cost extension. However, the combination of timely instructions, guidelines, and
feedback, along with an organized project team, set a solid foundation for success in the
project.
• The project duration for work on systemic change required 4 to 5 years, as it was
complex and involved significant efforts to modify people's attitudes, habits, and
behaviours.
• The efficiency of resource use showed, for example, during the COVID-19 pandemic.
By utilizing technology, such as video conferencing tools, it became possible to connect
with a diverse range of individuals and groups remotely, reaching even more
stakeholders compared to personal visits.
• The involvement of local experts was crucial in energizing local stakeholders,
particularly during challenging times like a pandemic.
• Implementing a focal point approach, which entails centralized coordination in each
country streamlined communication, efficient organization, and better collaboration
among stakeholders catalysed the project implementation.
28
projects on track, managing resources efficiently, and achieving project objectives within the
defined timelines. Proactive communication ensures that stakeholders are well-informed, potential
issues are addressed promptly, and collaboration is facilitated among team members.
When project teams are recognized for their excellent project management and proactive
communication, it demonstrates their commitment, professionalism, and ability to deliver
successful outcomes. This recognition is a testament to their hard work, dedication, and positive
impact on the project.
4.2 Timeliness of results achievement and efficient organization
COVID-19-related restrictions affected the project implementation, requiring a no-cost extension.
However, the combination of timely instructions, guidelines, and feedback, along with an organized
project team that meets deadlines to the extent possible, set a solid foundation for success in the
project. The project was well-managed, which increased the likelihood of achieving the desired
outcomes within the designated timeline.
Effective communication and timely feedback were crucial for the success of any project. As
consultants received clear instructions and guidelines, it helped them understand their roles and
responsibilities, enabling them to perform their tasks efficiently.
The project duration for work on behaviour change required 4 to 5 years, as it was complex and
involved significant efforts to bring about behavioural changes. Behavioural change projects often
need a long-term approach to modify people's attitudes, habits, and behaviours effectively.
4.3 Efficiency of resource use
Resource use efficiency emerged, for example, during the COVID-19 pandemic. Virtual country
consultations during the pandemic have been an effective way to reach even more stakeholders than
personal visits. By utilizing technology, such as video conferencing tools, it became possible to
connect with a diverse range of individuals and groups remotely, including those who may be
unable to attend in-person meetings due to travel restrictions.
In the context of the Republic of Moldova, for example, virtual country consultations have played
a significant role in mitigating the impact of the pandemic. These consultations allow for broader
participation, involving stakeholders from different regions, backgrounds, and sectors. This
inclusive approach facilitates the exchange of ideas, expertise, and experiences, ultimately leading
to more comprehensive and informed decision-making processes.
The involvement of local experts was crucial in energizing local stakeholders, particularly during
challenging times like a pandemic. Local experts deeply understand their communities' specific
context, challenges, and opportunities. Their expertise and ability to communicate effectively with
local stakeholders help in encouraging participation and fostering ownership of proposed solutions.
Implementing a focal point approach, which entails centralized coordination in each country,
can enhance the effectiveness of virtual consultations. Having a designated focal point ensures
streamlined communication, efficient organization, and better stakeholder collaboration. This
29
approach facilitated the collection and dissemination of information, the alignment of efforts, and
the identification of synergies between different initiatives.
Figure 11 provides an overview of stakeholders’ perceptions about the project’s efficiency. While
30,6% of stakeholders were unable to respond to the question due to the lack of insights, 50% very
high ratings show (very much so) and 15,3% high ratings (mostly).
Figure 11: Stakeholder perception about the project’s efficiency
n=49
50,0
15,3
2,0 2,0
0,0
30,6
0,0
10,0
20,0
30,0
40,0
50,0
60,0
Very much
so
Mostly Somewhat A little Not al all No answer
Pe
rc
en
ta
ge
“I was pleasantly surprised by project team. They extended deadlines to allow for the review
of document. It was a very friendly and proactive team, while the UN can be very sterile”.
Source: project stakeholder.
30
5. Sustainability: are results lasting?
This section analyses the sustainability of project results using the following sub-criteria: i)
measures to ensure the sustainability of project results; ii) ownership of project results,
institutionalization, and up-scaling; and iii) potential for replication.
The main data sources used in this section are the document review, virtual interviews, the online
survey, and interviews during the field visit.
While according to stakeholder perceptions, the sustainability rating for project results reaches
94%, 33% of stakeholders stated that it was too early to see lasting results.
5.1 Measures to ensure sustainability of project results
The evaluation identified varying measures to ensure the sustainability of project results according
to country contexts and the maturity of national innovation ecosystems.
Overall, countries remained in touch to continue collaborating for more policy recommendations
and support national strategies and initiatives. Following the set-up of the sub-regional platforms,
countries are eager to perform well, and the evaluation found that the project incentivizes countries
to perform well on those indicators. Foundations for change have been created.
However, the evaluation finds that hand-holding for national strategies and policies is required,
including support during the review and updating of the latter.
Azerbaijan: The Ministry of Innovation has been established in Azerbaijan, indicating a
commitment to promoting innovation. Additionally, two draft laws related to innovation are
currently being considered. Unfortunately, no specific details were provided regarding the content
of these draft laws.
Key findings: The evaluation finds that the level of lasting results is mixed across project
countries.
• As the project streams ended, countries remained engaged to collaborate for more
policy recommendations and support national strategies and initiatives.
• However, the evaluation finds that handholding for national strategies and policies is
required, including support during review and updating of the latter, for example, in
Armenia or the Republic of Moldova.
• Ownership of results, institutionalization, and up-scaling are evident in Armenia,
Azerbaijan, Georgia, and the Republic of Moldova.
• The evaluation finds that the project's methodology and approach can be replicated in
other sub-regions and countries, with some amendments to the current approach and
always considering sub-regional contextualization.
31
In Armenia, ensuring the sustainability of project results, particularly in the context of national
innovation ecosystems, could be achieved through a combination of policy measures. Suggestions
to help the government implement recommendations and develop financial schemes to channel
private investments include:
1. Establish a National Innovation Strategy: The government should develop a
comprehensive strategy that outlines the vision, objectives, and action plan for fostering
innovation and sustainable development in Armenia. The innovation agenda requires
incorporation in policy making. While policymakers have competing priorities and tend to
focus on broader strategic tasks, support is required from task forces or other kinds of
dedicated teams in the ministries or by external actors such as the SDG Innovation Lab.
2. Strengthen the SDG Innovation Lab: The SDG Innovation Lab can serve as a platform
for collaboration between the government, private sector, academia, and civil society.
3. Enhance Access to Finance: To develop financial schemes and attract private
investments, the government should focus on improving access to finance for innovators
and entrepreneurs. This can be done by establishing dedicated funds, venture capital
networks, and innovation grants that provide financial support to promising projects.
Public-private partnerships can also be encouraged to leverage private sector resources for
innovative projects.
Belarus: Following political unrest in 2020-21, many private sector innovators left the country.
Despite this, IPO reviews continue. However, it seems that the overall innovation ecosystem in
Belarus has been negatively affected by the situation.
In Georgia, the evaluation revealed examples demonstrating the government's efforts in policy
change and the implementation of reform agendas. These measures collectively demonstrate
Georgia's commitment to modernizing its procurement system, enhancing transparency, and
promoting efficient resource allocation in the public sector. By implementing these reforms,
Georgia aims to create a favourable environment for businesses, attract investments, and ensure
public funds' effective and accountable use.
Examples of measures in Georgia are:
1. Public Procurement Policy and Law: Georgia approved a new public procurement policy
and law in March 2023. This development signifies a significant step towards ensuring
transparency and efficiency in the public procurement process. The policy and law were
formulated through a four-year process involving extensive consultations with stakeholders
and experts in the field. The objective is to improve accountability, prevent corruption, and
enhance competition in public procurement.
2. E-Procurement Platform: To address transparency issues in the public sector, Georgia
has implemented a functioning e-Procurement platform. This digital platform allows for
electronically submitting and managing procurement bids and contracts. By transitioning
to an online system, the government aims to reduce bureaucracy, enhance transparency,
and increase efficiency in the procurement process. The e-Procurement platform has the
potential to enable real-time monitoring, evaluation, and auditing of procurement activities,
thereby minimizing the potential for corruption.
3. Mandatory Innovative Procurement: Starting January 2025, Georgia will make
innovative procurement mandatory. This policy directive emphasizes the importance of
32
utilizing innovative and advanced technologies, approaches, and solutions in procurement.
Republic of Moldova: The country has developed a national road map for innovation and
technology transfer, which suggests a focus on advancing technological innovation. There is also
an ongoing review of the law on innovation and technology parks, indicating efforts to create a
favourable environment for innovation. However, the establishment of a National Innovation
Council is still pending.
Ukraine: Due to the ongoing war and military investments, high-level political prioritization for
innovation seems to be suffering in Ukraine. Similar challenges are faced by the Republic of
Moldova as well. It suggests that the Ukrainian government focuses more on military-related
matters than on promoting innovation and technological advancement.
5.2 Ownership of project results, institutionalization, and up-scaling
Ownership of results, institutionalization, and up-scaling are important aspects of any development
or innovation agenda. The evaluation found the following country insights:
1. In Armenia, there is a desire to enhance collaboration and cooperation between the scientific
community, business associations, and non-governmental organizations (NGOs). This aims to
foster innovation, boost economic development, and address the specific needs of industries such
as the IT, wine, and tourism sectors. The I4SDR publication was extremely well received, including
at a well-attended, very high-level launch event. The government and UNECE are already working
on a roadmap to implement recommendations.
2. Azerbaijan has an ambitious agenda and is interested in more frequent sub-regional Innovation
Policy Outlooks (IPOs). Institutionalization is also essential, as it involves embedding the processes
and structures related to IPOs into the country's existing institutional framework. This includes
establishing a dedicated regulatory body, creating legal and policy frameworks, and ensuring
transparency and accountability throughout the IPO process. In this context, the country has made
two laws related to innovation. Azerbaijan has officially requested an I4SDR to be carried out by
UNECE.
3. In Belarus, policy recommendations are being used as a backup to push for the adoption of
proposals in parliament. Hence, the project responded to the need for evidence-based decision-
making and a recognition of the importance of research and analysis in shaping policies. The current
situation, however, seems less conducive to advancing innovation in the country, with many
private-sector actors having left the country.
4. Georgia: The government designed and adopted a law on public procurement, which
mainstreams innovation enhancing procurement and is considered one of the biggest
successes of the whole project. Learning from this success, the institutionalization, and
upscaling of initiatives, it is crucial to foster a sense of ownership among relevant government
agencies, embed the changes within existing systems and structures, build capacities for internal
training, and involve employees at all levels to ensure continuity. The latter is critical due to the
frequent staff changes in government agencies' management. Each new management team tends to
bring its own agenda and may not fully appreciate the previous efforts. This underscores the
importance for UNECE of reaching mid-level and lower-level technical staff, too. Stakeholders
33
highlighted the importance of a study tour to Norway to learn from good practices in Western
European procurement policies and to establish contacts with Norwegian counterparts.
5.the the Republic of Moldova, subsequently to the I4SDR publication, a joint Government-
UNECE task force was established to develop an innovation roadmap, leading to already significant
reforms including legal framework adjustments, strategy development, and an enhanced SME-
research collaboration framework. Whether national or donor funding is available to advance the
innovation agenda is questioned. Ownership of results in this context means that the Republic of
Moldova takes responsibility for driving the innovation agenda and ensuring its successful
implementation. This requires a commitment from the government to allocate resources, both
financial and human, to support innovation activities. Institutionalization involves creating
structures and processes within the government to coordinate and oversee the implementation of
the innovation agenda. Several steps in this direction have been taken already.
6.Ukraine: Following the beginning of the war, Ukraine is undertaking comprehensive
reconstruction and modernization plans that will require an innovation policy. In this case,
ownership of results means that Ukraine should take ownership of the outcomes and benefits of
these reconstruction and modernization efforts, which are currently hindered by frequent staff
turnover and a strong focus on the country's military efforts. UNECE is supporting Ukraine within
this project along three high-impact avenues:
I. Supporting the drafting of national and international reconstruction strategies to ensure
they are coherent and do not overlap + ensure innovation plays a part in revitalizing the nation's
infrastructure, economy, and social fabric post-conflict.
II. Contributing to the upcoming National Technology Development Strategy and Roadmap,
based on innovation policy practices from other countries that have undergone armed
conflicts, like Western Balkans, Armenia. UNECE will leverage its unique role of facilitating
the exchange of specific practices between member States.
III. Enhancing policy evaluation and monitoring to make sure incoming international funds are
spent correctly. Evaluation and monitoring practices in Ukraine were weak and there is an
imperative to enhance capacity to fight the misuse of funds.
7. At the regional level, the Innovation Policy Outlook was launched in 2020. The process gave
birth to a unique policy exchange forum on innovation policy in Eastern Europe and the South
Caucasus, which has grown organically and now meets regularly and semi-autonomously with
limited resource needs.
Figure 12 analyses stakeholders’ perceptions about the project’s efficiency. While 30,6% of
stakeholders were unable to respond to the question, as they felt it was too early to tell whether
project results would last, 38,8% very high ratings were given (very much so) and 19,4% high
ratings (mostly). Medium ratings achieved 6,1%(somewhat) and low ratings 5,1% (little).
34
Figure 12: Stakeholder perception about the project’s sustainability
n=49
5.3 Potential for replication
Based on stakeholder feedback, the evaluation finds that the project's methodology and approach
can be replicated in other sub-regions and countries.
Stakeholders clarified that for the replication of the UNECE project, the EC's Policy Support
Facility and other similar support by EC would require consideration in Eastern Partnership
countries.
National stakeholders would require more support in disseminating research results and conducting
information campaigns for a broader outreach to wider stakeholder groups. The importance of peer-
to-peer learning across a sub-region, e.g. based on specific chapters of the I4SDR, would be
desirable.
For replication, stakeholders would find it beneficial to publish the research methodology for
transparency reasons and trust creation. Stakeholders would benefit from knowing the research
questions in advance and clarifying the research purpose.
However, the evaluation also identified limitations of replicating project results. The box below
exemplifies limitations concerning the replicability of Georgia's procurement platform.
38,8
19,4
6,1 5,1
0,0
30,6
0,0
5,0
10,0
15,0
20,0
25,0
30,0
35,0
40,0
45,0
50,0
Very much
so
Mostly Somewhat A little Not al all No answer
Pe
rc
en
ta
ge
35
Learning from the implementation of the project, the following possible scenario emerges for
replicating the project in other UNECE sub-regions:
• Situation analysis: readiness assessment for developing innovative ecosystems, also
considering differences between maturity in urban and rural settings, as well as the current
policy environment
• Activation of the innovation ecosystem and local connectedness to create a sense of
community for network development and dialogue: government, private sector, academia
• Research: IPO, sub-regional benchmarks,
• Study tours: showcasing examples from other parts of Europe, e.g. Baltic countries and
Central/Western European countries
• Capacity building, focusing on i) government to reach policymakers who are in charge of
driving innovation policy development, e.g. through a training of trainers approach; and ii)
academia, to create or adapt curricula in universities and training institutes
• Complementing this approach through a private sector angle:
o Identification of crucial stone businesses
o Analysis of the potential for amplification of investments in innovation
o Linkages of keystone businesses to innovation ecosystems, including academia
o Identification of possible geographic or sector innovation clusters
“I strongly believe that the evaluation of procurement practices cannot be universally applicable due to
the significant differences among countries. Each country has its own unique economic level, policies,
public administration model, governance model, and traditions. For example, Armenia has a small
department for public procurement policy, while Azerbaijan and Ukraine have different models. The
Republic of Moldova, on the other hand, has no secondary legislation in place. It is challenging to
evaluate and replicate practices across such diverse contexts. In my country, we have a strong focus on
secondary legislation and a well-established governance system with electronic procurement, e-payment
systems, and efficient processes.
The project we are currently working on is only possible because of the new law that includes provisions
for innovative procurement. Without these tools and provisions, the project would be meaningless. Our
aim is to implement these provisions effectively and ensure that they are part of the legal framework. We
are already conducting training modules, workshops, and seminars to promote this knowledge. By 2025,
it will be mandatory in my country to procure innovative goods and services, and we are already
promoting the use of electronic procurement procedures and tools”.
Source: project stakeholder, Georgia
36
Section III: Conclusions and Recommendations
The following conclusions and recommendations emerge based on the main findings
summarized at the beginning of the findings' sections for each evaluation criterion. Figure 13
presents the logical flow from key findings to conclusions and recommendations. As the
project is about to end, all recommendations are for similar future projects.
37
The following conclusions and recommendations emerge based on the main findings summarized at the beginning of the findings' sections for each
evaluation criterion. Figure 13 presents the logical flow from key findings to conclusions and recommendations. As the project has ended, all
recommendations are for similar future projects.
Figure 13: Summary of key findings, conclusions, and recommendations
Key findings Conclusions Recommendations for similar future
projects
R
el
ev
an
ce
a
nd
c
oh
er
en
ce
The UNECE project promoted a regional perspective, filling a critical gap and
serving to shape national innovation strategies.
The project addressed a significant
sub-regional development
cooperation gap through its in-depth
support to science, technology, and
innovation (STI), in particular its
focus on STI policy and governance.
The project provided visibility to
Sida and UNECE, and transferred
large numbers of relevant
international good policy practices
to the region (including lessons
learned from Sweden’s innovation
development journey).
R1: UNECE: Given its high relevance, it is
recommended to i) seek continuation funding to
support further Eastern Europe and the South
Caucasus countries (especially for ensuring
sustainability and continuity); ii) seek new
funding to replicate this project for other
subregions, such as the Western Balkans or
Central Asia. At the same time, this can raise
donor visibility and complement investments
into the European Union’s (EU's) Eastern
Partnership (in the case of Eastern Europe and
the South Caucasus). UNECE’s governance-
focused approach using evaluation,
accountability, and transparency also aims to
reduce corruption and informality in the EESC
region, where this issue is of great relevance.
Priority: high, next six months for new project
designs
Project components were relevant, as the European Innovation Scoreboard, the
Global Innovation Index, and other composite indices were based on output and
input indicators, rather than the essential issue of innovation governance and the
particular features, challenges, and potential typically shared among transition
economies in general and among EESC countries in particular.
Similarly, the sub-regional approach in most activities was highly relevant, given
shared challenges and legacies.
The focus on innovation governance not only filled an important gap, but also
addressed the much neglected issue of monitoring and evaluation of and value-for-
money from public spending and support instruments – essential also because of
fiscal constraints growing, especially in the wake of the pandemic induced slump.
The constant methodological improvements of UNECE’s flagship analytical tools,
notably, the I4SDRs and the IPO. Elective chapters were added, meeting direct
demand from the countries themselves. This work could now be scaled up to other
regions and/or countries with little cost.
The project provided visibility to Sida and UNECE, transferred large numbers of
relevant international good policy practices to the region (including lessons learned
from Sweden’s innovation development journey), and worked closely with all
relevant stakeholder groups and to the enhance the region’s STI ecosystem.
The project provided visibility to Sida and UNECE, transferred large numbers of
relevant international good policy practices to the region (including lessons learned
from Sweden’s innovation development journey), and worked closely with all
relevant stakeholder groups and to the enhance the region’s STI ecosystem.
38
E
ff
ec
tiv
en
es
s
The project established a multi-stakeholder dialogue for developing the IPOs
leveraging UNECE's neutrality as a UN body, using international and local expertise
and comparing the six countries. The dissemination of lessons learned through
UNECE’s IPO/I4SDR/CB/dialogue work at its intergovernmental CICPPP and ToS-
ICP sessions also enhanced policy dialogue, where hundreds of member States and
international organisations representatives listen in and exchange knowledge about
innovation.
UNECE's neutrality and convening
power helped engage stakeholders
during the project implementation,
enhancing knowledge and
awareness about innovation policies.
See R 1.
The project successfully improved the understanding of policy options at the national
level based on thorough research and the identification of policy challenges and
objectives.
The policy dialogue mentioned above at the national and international level also
contributed to an enhanced understanding of policy options.
The project's positive results were
attributed to a proactive and
engaged professional team,
flexibility, and mitigation of
unforeseeable factors.
R3: UNECE: In a context where donors
increasingly focus on short-term results after 12
or 24 months, UNECE should encourage
Sweden and other donors to continue investing
in projects with a systems change approach,
The project successfully improved the understanding of policy options at the
national level based on research and the identification of policy challenges and
objectives, as well as UNECE training, and capacity-building seminars.
The project addressed previously poor coordination between national stakeholders,
exacerbated by frequent staff changes in government agencies in all project
countries.
The project worked closely with all
relevant stakeholder groups and to
the enhance the region’s STI
ecosystem. However, not all
deliverables reached all countries
evenly, and the Leave No One
Behind approach was supply-driven.
The adaptation of the project for any
future implementations will be
highly cost-effective
As a Geneva-based body with no regional offices, UNECE provided analytical input
and recommendations to guide on-going or planned donor-funded activities
operating on the ground.
The Leave No One Behind approach promoted by the United Nations Secretariat
was taken very seriously by the project designers and during its implementation.
However, these cross-cutting issues were not as clearly reflected in governments’
needs and priorities and results were limited.
R2: UNECE: Increase awareness among
national stakeholders in Member States about
the importance of the Leave No One Behind
approach across policy-making, for example,
by adding relevant indicators in the project
results framework, and supporting ongoing
national processes such as the upgrading of
online procurement platforms for reading
impaired persons, with a focus on gender,
disabilities, and human rights (keeping in mind
that the latter two were not included in the
project document).
Priority: medium, next 6-12 months for new
project designs
Greater awareness and education seem required among stakeholders about the
importance and benefits of integrating gender, human rights and disability
perspectives into project design and implementation.
39
The synergies between UNECE’s intergovernmental work, analytical work,
technical assistance, and capacity building were crucial for achieving this project
objective. The secretariat studies best international practices and success stories,
implements them in less developed member States, and disseminates lessons learned
in the process to other member States, feeding the foundation of knowledge UNECE
have built over the years.
leading to the implementation of research
recommendations as a means to ensure the
sustainability of results. A project duration of 5
years is recommended for projects with such an
approach.
Priority: medium, next 6-12 months for new
project designs The project managed to accomplish concrete policy change in the areas with
potential and demand for change by creating an evidence base and capacity building.
Examples include developing and adopting a new procurement law in Georgia
emphasising innovation-enhancing procurement (IEP), technology transfer in the
Republic of Moldova, and venture capital in Belarus.
For other countries, the actual implementation of recommendations is too early to
tell.
The project made actionable, targeted, time-bound and prioritized recommendations,
and the dialogue with national stakeholders continued to facilitate the
implementation of recommendations.
Unexpected project results included the project's ability to bring together diverse
stakeholders from countries' innovation ecosystems due to good preparation of the
project team and local intelligence on the ground through the use of local experts.
Unexpected project results were
positive and underscored the sound
project implementation approach by
a highly professional project team.
Sida's flexibility to allow for a
project duration over four years
contributed to the project's success.
Positive factors influencing project performance included the quality of the project
team, UNECE's convening power, the acceleration of using virtual technology as a
COVID-19 mitigation measure, and the project duration of over four years.
Negative factors influencing project performance comprised meeting and travel
restrictions due to COVID-19, the volatile political situation in the sub-region, the
economic downturn and turf battles among government stakeholders due to unclear
or overlapping mandates concerning innovation.
Negative factors influencing project
performance were largely
unforeseeable and beyond the
project team's control but were
successfully mitigated where
possible.
No recommendation.
Changes in policymakers' competencies: Policymakers broadened their
understanding of innovation concepts. The project mitigated the frequent staff
turnover the project mitigated this risk through a networking approach, including at
the technical level.
It is too early to assess changes in competencies to support environmental
sustainability, gender equality, good governance, and economic growth.
Coordination: the project systematically involved other United Nations (UN) and
non-UN stakeholders in the implementation, including UN country representatives,
EU delegations and Swedish embassies, WIPO, OECD, and WEF, showing good
coordination with UN stakeholders and other international partners.
The project team's systematic
inclusion of UN and non-UN
stakeholders is a good practice,
No recommendation.
40
benefitting Swedish embassies
directly in the project countries.
E
ff
ic
ie
nc
y
The project has been well-executed, thanks to adequate funding for mobilizing
consultants and a professional project team in UNECE showing an outstanding
performance. The project team used international experts and tapped into a network
of well-connected national experts.
The project was good value-for-
money for due to a professional
team, efficient implementation,
local expert use, and centralized
coordination through national focal
points during pandemic-related
travel restrictions. UNECE even
executed deliverables not in the
donor agreement using existing
funds.
See R 3 on the recommended duration of
projects.
COVID-19-related restrictions affected the project implementation, requiring a no-
cost extension. However, the combination of timely instructions, guidelines, and
feedback, along with an organized project team, set a solid foundation for success in
the project.
The project duration for work on systemic change required 4 to 5 years, as it was
complex and involved significant efforts to modify people's attitudes, habits, and
behaviours.
The efficiency of resource use showed, for example, during the COVID-19
pandemic. By utilizing technology, such as video conferencing tools, it became
possible to connect with a diverse range of individuals and groups remotely, reaching
even more stakeholders compared to personal visits.
The involvement of local experts was crucial in energizing local stakeholders,
particularly during challenging times like a pandemic.
The efficiency of the project
implementation showed in the
invaluable use of local experts,
especially during pandemic-related
travel restrictions in 2020 and 2021,
as well as the centralized project
coordination in each country
through national focal points.
R 4: UNECE: Building on the good practices of
this project, using local experts to gather
intelligence on the ground is recommended for
similar future projects at the country level. At
the same time, national focal points should be
appointed for centralized project coordination in
project countries.
Priority: medium, next 6-12 months, for new
project designs
Implementing a focal point approach, which entails centralized coordination in each
country, streamlined communication, efficient organization, and better collaboration
among stakeholders, catalysed the project implementation.
Su
st
ai
na
bi
lit
y
As the project streams ended, countries remained engaged to collaborate for more
policy recommendations and support national strategies and initiatives.
Stakeholders demonstrate a strong
interest in project
recommendations, institutionalizing
them in many countries, but require
continued external support for
innovation strategies and policies.
R 5: UNECE: As a follow-up to this project,
UNECE should use its limited regular budget
resources to monitor the implementation of
recommendations and keep engaging with the
network of focal points to share good practices
for developing and implementing national
innovation-related strategies and policies.
Priority: very high, next 3-6 months for new
project designs
Handholding for national strategies and policies is required, including support
during the review and updating of the latter, for example, in Armenia or the Republic
of Moldova.
Ownership of results, institutionalization, and up-scaling are evident in Armenia,
Azerbaijan, Georgia, and the Republic of Moldova.
41
Overall, the project's methodology and approach can be replicated in other sub-
regions and countries, with some amendments to the current approach and always
considering sub-regional contextualization.
The project approach is fit for
purpose and ready to be replicated in
other sub-regions.
R 6: UNECE: for replicating the project
approach in other sub-regions, for example, the
Western Balkans, with high relevance for
donors like the EU and Sweden. This should be
done, as explained in R 1 by i) seeking
continuation funding to support further EESC
countries (especially for ensuring
sustainability); ii) seeking new funding to
replicate for other regions, such as the Western
Balkans or Central Asia.
Priority: very high, next 3-6 months for new
project designs
42
Given the logical flow of the evaluation analysis presented in Figure 13 the following conclusions
and recommendations emerge.
Relevance and coherence
Conclusions: The project addressed a significant sub-regional development cooperation gap
through its in-depth support to science, technology, and innovation (STI), in particular its focus on
STI policy and governance. The project provided visibility to Sida and UNECE, transferred large
numbers of relevant international good policy practices to the region (including lessons learned
from Sweden’s innovation development journey), and worked closely with all relevant stakeholder
groups and to the enhance the region’s STI ecosystem. However, not all deliverables reached all
countries evenly, and the Leave No One Behind approach was supply-driven. The adaptation of the
project for any future implementations will be highly cost-effective.
Recommendations for similar future projects addressed to UNECE:
Priority: medium, next 6-12 months for new project designs.
Effectiveness
Conclusions: UNECE's neutrality and convening power helped engage stakeholders during the
project implementation, enhancing knowledge and awareness about innovation policies. The
project's positive results were attributed to a proactive and engaged professional team, flexibility,
and mitigation of unforeseeable factors. The team's systematic inclusion of UN and non-UN
stakeholders benefits Swedish embassies in the project countries by raising the innovation topic on
national agendas and sharpening Sweden’s profile on this topic in the region.
Recommendations for similar future projects:
R 3: In a context where donors increasingly focus on short-term results after 12 or 24
months, UNECE should encourage Sweden and other donors to continue investing in
projects with a systems change approach, leading to the implementation of research
R1: Given its high relevance, it is recommended to i) seek continuation funding to support
further Eastern Europe and the South Caucasus countries (especially for ensuring
sustainability and continuity); ii) seek new funding to replicate this project for other
subregions, such as the Western Balkans or Central Asia. At the same time, this can raise
donor visibility and complement investments into the European Union’s (EU's) Eastern
Partnership (in the case of Eastern Europe and the South Caucasus). UNECE’s governance-
focused approach using evaluation, accountability, and transparency also aims to reduce
corruption and informality in the EESC region, where this issue is of great relevance.
Priority: high, next six months for new project designs.
R 2: Increase awareness among national stakeholders in Member States about the
importance of the Leave No One Behind approach across policy-making, for example, by
adding relevant indicators in the project results framework, and supporting ongoing
national processes such as the upgrading of online procurement platforms for reading
impaired persons, with a focus on gender, disabilities, and human rights (keeping in mind
that the latter two were not included in the project document).
43
recommendations as a means to ensure the sustainability of results. A project duration of 5
years is recommended for projects with such an approach.
Priority: medium, next 6-12 months for new project designs.
Efficiency
Conclusions: The project was good value-for-money due to a professional team, efficient
implementation, local expert use, and centralized coordination through national focal points during
pandemic-related travel restrictions. UNECE even executed deliverables not in the donor
agreement using existing funds.
Recommendations for similar future projects:
R 4: Building on the good practices of this project, using local experts to gather intelligence
on the ground is recommended for similar future projects at the country level. At the same
time, national focal points should continue to be appointed for centralized project
coordination in project countries.
Priority: medium, next 6-12 months, for new project designs.
Sustainability
Conclusions: Stakeholders demonstrate a strong interest in project recommendations,
institutionalizing them in many countries, but require continued external support for innovation
strategies and policies. The project approach is fit for purpose and ready to be replicated in other
sub-regions.
Recommendations for similar future projects:
R 5: As a follow-up to this project, UNECE should use its limited regular budget resources
to monitor the implementation of recommendations and keep engaging with the network
of focal points to share good practices for developing and implementing national
innovation-related strategies and policies.
Priority: very high, next 3-6 months for new project designs.
R 6: Senior management should use this evaluation report as a robust evidence base to
lobby for replicating the project approach in other sub-regions, for example, the Western
Balkans, with high relevance for donors like the EU and Sweden. This should be done, as
explained in R 1 by i) seeking continuation funding to support further EESC countries
(especially for ensuring sustainability); ii) seeking new funding to replicate for other
regions, such as the Western Balkans or Central Asia.
Priority: very high, next 3-6 months for new project designs.
44
Annex 1: Terms of Reference of the Evaluation
TERMS OF REFERENCE
E317: Promoting innovation policy capacities in Eastern Europe and the Caucasus
I. Purpose
The purpose of this evaluation is to assess the extent to which the objectives of the UNECE project
E317 “Promoting innovation policy capacities in Eastern Europe and the Caucasus” were achieved.
The evaluation will assess the relevance, effectiveness, efficiency and sustainability of the project
in enhancing innovation policy capacities in Eastern Europe and the South Caucasus (EESC)
countries.
The evaluation will also assess any impacts the project may have had on progressing human rights,
gender equality, disability inclusion, climate change and disaster risk reduction in the context of
this engagement. The evaluation will finally look at the activities repurposed to address the impact
of the COVID-19 crisis, and assess, to the extent possible, UNECE’s COVID-19 early response
through this project.
II. Background
Launched in November 2018, the project aimed to improve the competencies of policymakers in
designing, running, reforming, and monitoring effective innovation policies and institutions that
make measurable contributions towards long-term economic sustainable development.
The project looked at the six Eastern Europe and the South Caucasus (EESC) member States
because of their shared features including a common historic legacy (predominance of central
planning, a strong role for public research institutions; history of strong, at times frontier research),
economic structure (strong manufacturing tradition; a legacy of state ownership of the economy;
focus on heavy industries), geography (proximity to the EU and CIS), and factor conditions (high
levels of education in the workforce; high labour participation rates among women). These shared
traits enabled the Innovation Policy Outlook (IPO) and other project work streams to build a strong
basis for policy learning and enable benchmarking.
The project was implemented by UNECE Subprogramme 4 Economic Cooperation and
Integration/Economic Cooperation and Trade Division and reflected UNECE’s mandates in the
sphere of innovation, competitiveness and public-private partnerships. In particular, the objectives
of the project were:
(a) Improved policy dialogue on promoting sustained economic growth, innovative
development, and greater competitiveness in the UNECE region;
(b) Improved understanding at the national level of policy options to promote sustained
economic growth, innovative development, and greater competitiveness; and
(c) Enhanced national implementation of UNECE policy recommendations and standards on
promoting a policy, financial and regulatory environment conducive to sustained economic
growth, innovative development and greater competitiveness.
To achieve the project objective, UNECE engaged in several core activities which were agreed-
upon in consultation with national focal points:
1. Sub-regional Innovation Policy Outlook
UNECE developed the methodology, conducted primary research, and published a pilot Sub-
regional Innovation Policy Outlook (IPO) study benchmarking innovation governance performance
across all six EESC member States. Later, UNECE also conducted an Interim IPO (IIPO) on topics
of interest to the countries.
2. Innovation for Sustainable Development Reviews
45
Second, UNECE carried out national-level Innovation for Sustainable Development Reviews
(I4SDR) of Georgia (2021), the Republic of Moldova (2022), Armenia (ongoing), and Ukraine
(ongoing – publication date is to be determined). Complementing the sub-regional level assessment,
the I4SDR examines national innovation systems in greater detail and includes in-depth analysis
elective topics.
3. Capacity Building
The third element of the project was conducting capacity-building activities supporting specific
reform efforts to put recommendations from national reviews into practice. UNECE also engaged
in sub-regional capacity building in follow-up to the IPO. For example, UNECE conducted a
capacity-building program focused on innovation-enhancing procurement (IEP) for Georgia. As a
follow-up to the I4SDR of the Republic of Moldova, UNECE also developed a roadmap on
Innovation and Technology Transfer and implemented two trainings for Moldova.
4. IPO Policy Dialogue Sessions
To provide a platform to share policy recommendations and findings from the studies, UNECE
organized several IPO policy dialogue sessions.
The project involved the following focal point institutions: Armenia: Ministry of High-Tech
Industry; Ministry of Economy; Ministry of Education, Science, Culture and Sport. Azerbaijan:
Ministry of Transport, Communications and High Technologies. Belarus: Belarusian Institute of
System Analysis and Information Support of S&T Sphere (BELISA), under the State Committee
for Science and Technology of the Republic of Belarus. Georgia: Georgia’s Innovation and
Technology Agency (GITA), under Ministry of Economy. Moldova: Ministry of Education,
Culture and Research; National Institute for Economic Research (NIER), under Ministry of
Economy. Ukraine: National Academy of Sciences; Ministry of Economy; Ministry of Education
and Science.
III. Evaluation objectives, scope and questions
The evaluation will be guided by the objectives, indicators of achievement and means of
verification established in the logical framework of the project document. The evaluation will be
conducted in Q1-Q2 of 2023 at the request of the donor. It will cover close to the full
implementation of the project, from November 2018 to March 2023 in six countries (Armenia,
Azerbaijan, Belarus, Georgia, the Republic of Moldova, and Ukraine). The project is currently
planned to be completed by August 2023, with a possible extension till February 2024 currently
under discussion with the donor. The majority of activities will have been completed by March
2023.
The final evaluation of the project has the following specific objectives:
• Determine as systematically and objectively as possible the relevance, effectiveness,
efficiency and sustainability of the project results in light of its goals and objectives;
• Assess how the project activities contributed to gender equality and women’ s
empowerment, as well as the realization of human rights, with an emphasis on ‘leaving no
one behind’ and, if needed, it will make recommendations on how these considerations can
be better addressed in future activities of the subprogramme.
• Identify good practices and lessons learned from the project and formulate action-oriented,
forward-looking recommendations addressed to the subprogramme for improving future
interventions.
The evaluation criteria are relevance, effectiveness, efficiency and sustainability.
46
Relevance
1. To what extent was the project design appropriate for meeting the needs of beneficiary
countries?
2. To what extent did the project respond to the priorities and needs of the participating
countries? How relevant were they to the countries’ needs and priorities?
3. To what extent was the project aligned with the SDGs?
4. What takeaways are there for ensuring relevance of future UNECE projects?
5. To what extent were gender, human rights and disability perspectives integrated into the
design and implementation of the project? What results can be identified from these
actions? How can gender and human rights perspectives be better included in future the
projects design and implementation?
Effectiveness
6. To what extent were the project objectives and expected accomplishments achieved?
7. To what extent did the project improve the competencies of innovation policy makers in
the participating countries to design, develop, implement, reform, and evaluate national
innovation policies?
8. To what extent are the project activities coherent and harmonized with those of other
partners operating within the same context, particularly those of other UN system entities?
9. What were the challenges/obstacles (including COVID-19 and sub-regional instability) to
achieving the expected results? How successfully did the project overcome these?
10. What (if anything) has prevented the project from achieving the desired results?
11. How effectively has the project tackled its underlying objective of improving the
competencies of innovation policy makers to support environmental sustainability, gender
equality, good governance, and economic growth in the participating countries?
Efficiency
12. Were the resources adequate for achieving the results?
13. Were the results achieved on time and were all activities organized efficiently?
14. To what extent were the resources used economically and how could the use of resources
be improved?
Sustainability
15. What measures were adopted to ensure that project outcomes would continue after the
project ended and to what extent have these measures addressed the existing risks for
sustainability?
16. To what extent do the partners and beneficiaries ‘own’ the outcomes of the work? How is
the stakeholders’ engagement likely to continue, be scaled up, replicated, or
institutionalized?
17. To what extent are the objectives of the project still valid? How can the project be
replicated in other UNECE sub-regions, in particular the Western Balkans?
47
IV. Evaluation approach and methodology
The evaluation will be conducted in accordance with: the ECE Evaluation Policy 6 ; the
Administrative instruction guiding Evaluation in the UN Secretariat7; and the United Nations
Evaluation Group (UNEG) Norms and Standards for Evaluation 8 . Human rights and gender
equality considerations will be integrated at all stages of the evaluation9: (i) in the evaluation scope
and questions; (ii) in the methods, tools and data analysis techniques; (iii) in the findings,
conclusions and recommendations of the final report. The evaluator will explicitly explain how
human rights, gender, disability, SDGs, and environmental considerations will be taken into
account during the evaluation.
The evaluator is required to use a mixed-method approach, including qualitative as well as
quantitative data gathering and analysis as the basis for a triangulation exercise of all available data
to draw conclusions and findings. The evaluator shall conduct online surveys and interview a wide
range of diverse stakeholders from both the private and the public sector, academia, international
organisations, and, where applicable, the diaspora and civil society. To ensure representativeness,
the evaluator shall speak to a large sample of stakeholders including high-level government
interlocutors whom UNECE has worked with. UNECE also strongly suggests that the evaluator
organises in-person visits to the countries under review to conduct interviews and gather data.
The evaluation should be conducted based on the following mixed methods to triangulate
information:
1. A desk review of all relevant documents, including the project document and information on
project activities (monitoring data); materials developed in support of the activities (agendas,
plans, participant lists, background documents, donor reports and publications); Proposed
programme budgets covering the evaluation period; project reports to the donor.
2. Online survey of key stakeholders and beneficiaries: the survey will be developed by the
consultant on her/his preferred platform.
3. Interviews (in-person and by telephone/video): the evaluator shall interview a wide range of
diverse stakeholders and beneficiaries from both the private and the public sector, academia,
international organisations, and, where applicable, the diaspora and civil society. To ensure
representativeness, the evaluator shall speak to a large sample of stakeholders including high-
level government interlocutors whom UNECE has worked with. UNECE also strongly suggests
that the evaluator organises in-person visits to one or more countries of project implementation
to conduct interviews and gather data.
4. Case Study/ies, which will include a detailed examination of the project intervention in one or
more of the six countries of project implementation (the selection criteria will be included in
the inception report).
The evaluator will further elaborate on the evaluation methodology in the Inception Report that will
among others include the survey questions, travel plans and whether any of the six countries will
be selected for an in-depth assessment. The evaluation report will be written in English, will consist
of approximately 30 pages and will include an executive summary (max. 2 pages) describing the
evaluation methodology, key findings, conclusions and recommendations. The evaluator will also
produce an evaluation brief summarizing key evaluation findings, lessons learned and
recommendations, including through images and infographics.
6 UNECE Evaluation policy
7 ST/AI/2021/3
8 UNEG 2016 Norms and Standards for Evaluation
9 In line with UNEG Guidance contained in Integrating Human Rights and Gender Equality in Evaluations
48
V. Evaluation schedule10
January 2023 ToR finalized
March 2023 Evaluator selected
April 2023 Contract signed. Evaluator starts the desk review
End-April 2023 Evaluator submits inception report including survey design
May2023 Launch of data gathering, including survey and interviews
Early June 2023 Evaluator submits draft evaluation report and evaluation brief
End-June 2023 Evaluator submits final evaluation report and evaluation brief
VI. Resources and Management of the evaluation
An independent consultant will be engaged to conduct the evaluation under the management of the
PMU. The assignment will involve travel to selected countries among the six beneficiary countries
to conduct an in-depth assessment. Payment will be made upon satisfactory delivery of work.
The Programme Management Unit (PMU) will manage the evaluation and will be involved in the
following steps: Selection of the evaluator; Preparation and clearance of the Terms of Reference;
Provision of guidance to the Project Manager and evaluator as needed on the evaluation design and
methodology; Clearance of the final report after quality assurance of the draft report.
The Project Manager, in consultation with the Division Director, will be involved in the following
steps: Provide all documentation needed for desk review, contact details, support and guidance to
the evaluation consultant as needed throughout the timeline of the evaluation; Advise the evaluator
on the recipients for the questionnaire and for follow-up interviews; Process and manage the
consultancy contract of the evaluator, along the key milestones agreed with PMU.
VII. Intended use / Next steps
The results of the evaluation will be used in the planning and implementation of future activities of
the UNECE Economic Cooperation and Integration Subprogramme. Findings of this evaluation
will be used when possible to:
• improve direct project’s follow up actions, implementation of products by project
beneficiaries and dissemination of the knowledge created through the project;
• assess the gaps and further needs of countries in the area of this project;
• formulate tailored capacity building projects to strengthen the national capacity in
enhancing innovation.
The results of the evaluation will be reported to the inter-governmental Team of Specialists on
Innovation and Competitiveness Policies and the Committee on Innovation, Competitiveness and
Public-Private Partnerships.
Following the issuance of the final report, the Project Manager will develop a Management
Response for addressing the recommendations made by the evaluator. The final evaluation report,
the management response and the progress on implementation of recommendations will be publicly
available on the UNECE website.
VIII. Criteria for evaluators
The evaluator should have:
1. An advanced university degree or equivalent background in relevant disciplines.
10 Final timetable to be agreed following engagement of the evaluator
49
2. Specialized training in areas such as evaluation, project management, social statistics,
advanced statistical research and analysis.
3. Knowledge of and experience in working with intergovernmental processes, innovation,
sustainable development.
4. Relevant professional experience in design and management of evaluation processes with
multiple stakeholders, survey design and implementation, project planning, monitoring and
management, gender mainstreaming and human-rights due diligence.
5. Demonstrated methodological knowledge of evaluations, including quantitative and
qualitative data collection and analysis for end-of-cycle project evaluations. Demonstrated
experience in conducting questionnaires and interviews is an asset.
6. Fluency in written and spoken English. Knowledge of Russian will be an advantage.
Evaluators should declare any conflict of interest to UNECE before embarking on an evaluation
project, and at any point where such conflict occurs.
50
Annex 2: Documents reviewed
Ministry of High-Tech Industry of the Republic of Armenia, 2023: Support letter for Promoting
innovation policy capacities in Eastern Europe and the Caucasus project.
Ministry of Education and Research of the Republic of Moldova, 2023: Support letter for Promoting
innovation policy capacities in Eastern Europe and the Caucasus project.
Ministry of Education and Science of Ukraine, 2022: Support letter for Promoting innovation policy
capacities in Eastern Europe and the Caucasus project.
UNECE, 2023: Innovation for Sustainable Development. Review of Armenia.
UNECE, 2022: Handbook on innovation-enhancing procurement for Georgia.
UNECE, 2022: E317 Promoting innovation policy capacities in Eastern Europe and the Caucasus.
Narrative report 1 January 2021 – 31 December 2021.
UNECE, 2022: Interim Sub-regional Innovation Policy Outlook 2022: Eastern Europe and the South
Caucasus.
UNECE, 2022: Study visit of Georgian officials to Norway. UNECE capacity-building programme on
innovation-enhancing public procurement. 14-16 September 2022
UNECE, 2021: Innovation for Sustainable Development. Review of Republic of Moldova.
UNECE, 2021: Sub-regional Innovation Policy Outlook 2020: Eastern Europe and the South Caucasus.
UNECE, 2021: 2020 evaluation of sub-programme 4. Innovation for Sustainable Development Reviews
and sub-regional Innovation Policy Outlook. Evaluation response.
UNECE, 2021: Sub-regional Innovation Policy Outlook 2020: Eastern Europe and the South Caucasus.
Summary and insights.
UNECE, 2021: E317 Promoting innovation policy capacities in Eastern Europe and the Caucasus.
Narrative report 1 January 2020 – 31 December 2020.
UNECE, 2020: Innovation for Sustainable Development Reviews and Sub-regional Innovation Policy
Outlook.
UNECE, 2020: Innovation for Sustainable Development. Review of Georgia.
UNECE, 2020: E317 Promoting innovation policy capacities in Eastern Europe and the Caucasus.
Narrative report 1 January 2019 – 31 December 2019.
UNECE, 2018: Promoting innovation policy capacities in Eastern Europe and the Caucasus. Project
document.
51
Annex 3: Lists of stakeholders interviewed
The evaluator collected data from 49 stakeholders. The list of 31 stakeholders interviewed in person
as well as by telephone and video call is available but will not be published with this report to
safeguard the participants' anonymity. The 18 stakeholders responding to the online survey did so
anonymously. Many respondents requested not to be named, but further information on those who
did not request this can be provided at the request of Sida.
52
Annex 4: Evaluation matrix
Evaluation questions/issues Indicators Proposed evaluation tools Data source
1.
R
el
ev
an
ce
1.1 To what extent was the project design appropriate for
meeting the needs of beneficiary countries?
Evidence that project design responds to
specific country and partner institution
needs priorities.
Document review;
Interviews with UNECE staff, country
representatives, and independent experts
Project
documentation;
Project
stakeholders.
1.2 To what extent was the project aligned with the
SDGs?
Evidence that the project is likely to
contribute to:
SDG 5.5, 5a, 5b, 8.1, 8.2, 8.3, 8.5, 8.6,
9.2, 9.3, 9.5, 9b, 9c
Document review Project
documentation
1.3 What takeaways are there for ensuring the relevance
of future UNECE projects?
--
1.4 To what extent were gender, human rights, and
disability perspectives integrated into the design and
implementation of the project?
1.5 What results can be identified from these actions?
1.6 How can gender and human rights perspectives be
better included in future project design and
implementation?
Evidence of the consideration of gender
issues during the design and
implementation, e.g., the existence of a
gender analysis that identified the gender
dimensions that the activities could
address
Evidence of gender aspects in the
activities
Proof of results regarding gender issues
addressed by the project
Document review;
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
53
Evaluation questions/issues Indicators Proposed evaluation tools Data source
2.
E
ff
ec
tiv
en
es
s
2.1 To what extent were the project objectives and
expected accomplishments achieved?
Evidence of project contribution to
expected accomplishments
Document review
Interviews with UNECE staff, country
representatives, and independent experts
Project
documentation;
Project
stakeholders.
2.2 To what extent did the project improve the
competencies of innovation policymakers in the
participating countries to design, develop, implement,
reform, and evaluate national innovation policies?
Evidence from crucial staff of improved
ability of policymakers to design,
develop, implement, reform, and
evaluate innovation policies.
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
2.3 To what extent are the project activities coherent and
harmonized with those of other partners operating within
the same context, particularly those of other UN system
entities?
Evidence of consideration of other
partners operating in a similar context,
e.g., UN entities
Evidence that project activities integrate
with those of these other partners
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
2.4 What were the challenges/obstacles (including
COVID-19 and sub-regional instability) to achieving the
expected results? How successfully did the project
overcome these?
Evidence of awareness and addressing of
challenges to achieving results.
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
54
3.
E
ff
ic
ie
nc
y
3.1 Were the resources adequate for achieving the
results?
Evidence of resource utilization in
comparison with indicator results
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
3.2 Were the results achieved on time, and were all
activities organized efficiently?
Time and budget extensions and reasons
thereof
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
3.3 To what extent were the resources used
economically, and how could the use of resources be
improved?
Financial utilisation across years and in
aggregate
Extent to which the management of the
resources of the partnership was based
on results, including the existence of an
RBM policy
Extent to which the project management
structures facilitated the implementation,
including evidence of actions taken to
improve implementation
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
Evaluation questions/issues Indicators Proposed evaluation tools Data source
2.5 What (if anything) has prevented the project from
achieving the desired results?
Examination of the theory of change and
the linkages between project and results
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
2.6 How effectively has the project tackled its
underlying objective of improving the competencies of
innovation policymakers to support environmental
sustainability, gender equality, good governance, and
economic growth in the participating countries?
Evidence from key staff of improved
ability of policymakers to support
environmental sustainability, gender
equality, good governance, and
economic growth.
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
55
4.
S
us
ta
in
ab
ili
ty
4.1 What measures were adopted to ensure that project
outcomes would continue after the project ended, and to
what extent have these measures addressed the existing
risks for sustainability?
Evidence from national policymakers
and practitioners that they have
initiated/taken actions such as allocating
dedicated staff and resources towards
further activity and/or knowledge
management in terms of UNECE good
practices and policy recommendations.
Evidence of consideration and mitigation
of sustainability risks
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
4.2 To what extent do the partners and beneficiaries'
own' the outcomes of the work? How is the stakeholders'
engagement likely to continue, be scaled up, replicated,
or institutionalized?
Evidence of the continuation of
stakeholder engagement, scaling,
replication, and/or institutionalization,
for example, through linkages with
activities and/or a strategy for knowledge
management
Document review
Interviews with UNECE staff, country
representatives and independent experts
Project
documentation;
Project
stakeholders.
4.3 To what extent are the objectives of the project still
valid? How can the project be replicated in other UNECE
sub-regions, particularly the Western Balkans?
Evidence of the validity of current
project objectives.
Evidence of replicability of the project in
other UNECE sub-regions.
Document review
Interviews with UNECE staff
Project
documentation;
Project staff
56
Annex 5: Country case studies: Armenia and Georgia
Armenia and Georgia were selected for more detailed examination, as case studies, in this project evaluation. The six project
countries (Armenia, Azerbaijan, Belarus, Georgia, Republic of Moldova, and Ukraine) were rated (1-5) according to their
demonstrated interest in taking part in project activities. Armenia and Georgia were the top-scoring countries, scoring 5
points each, both repeatedly expressing their interest in meetings as well as making formal requests.
The six countries were also rated on whether they had recent similar innovation reviews, and Armenia and Georgia also
rated most highly on this ranking, with a score of 5 points each. Neither Armenia nor Georgia had conducted an innovation
review of similar scope in the past five years.
Armenia and Georgia were thus chosen for closer examination in the project, and therefore field visits to these countries
were organized to produce a detailed examination of the project there.
Armenia
Armenia has made noteworthy progress towards achieving innovation-led, sustainable development. Despite the challenges
posed by regional and geopolitical instability and the COVID-19 pandemic, Armenia retains a competitive information and
communication technology (ICT) sector and a thriving entrepreneurship scene. Pockets of innovation excellence in ICT are
complemented by well-developed tourism, mining, food processing and agriculture sectors.
Despite these successes, Armenia still faces challenges to sustain economic growth and social development. Innovation, or
systematic experimentation with new ideas, processes, and products, can be the catalyst in solving these challenges and
bolster Armenia’s sustainable development. Improvements in innovation policy through intergovernmental coordination,
greater use of evidence and evaluation in policymaking, involvement of the diaspora and revaluation of the current
innovation infrastructure mechanisms are necessary to foster innovation. This project is critical to addressing these
constraints and support Armenia on its journey of economic development.
Relevance: To what extent was the project design appropriate for meeting the needs of beneficiary countries?
57
The project design was appropriate and directly applicable. There is a strong call for assistance on innovation from Armenia.
High-level Ministers, including H.E. Mr. Robert Khachatryan, Minister of High-Tech Industry, H.E. Mr. Vahan Kerobyan,
Minister of Economy, and H.E. Mr. Sargis Hayotsyan, Chairman of the Armenia Science Committee, are fully supportive
of this project, and the government is very involved in ownership of the project. The private sector and academia have also
inputted comments on the I4SDR (Innovation for Sustainable Development Review), which has just been launched, with
peer reviews and input from many stakeholders.
“The topic of Innovation is very important here. It is on the priority list of the government.”
The project design was helpful overall in terms of mainstreaming innovation in governance and coordinating projects and
activities with the government and partners. It aimed to address the major challenges and capitalize on the strengths of
beneficiary countries.
However, there may not be a good understanding of the outputs of the I4SDR yet. In much of the Armenian Government,
when it comes to innovation, there is a narrow focus on high tech outputs – e.g., funding tech startups, accelerators, and
developing potential unicorns. The project helped government staff and stakeholders see innovation more broadly – beyond
seeing innovation as just ICT, it introduced new perspectives (e.g., infrastructure, innovation ecosystem, regulatory reform)
that stakeholders found very helpful.
The project aimed to identify gaps and provide recommendations, considering the interconnectedness of the innovation
system. The project focused on a systematic approach to address multiple gaps and policy reforms, rather than focusing
solely on single issues such as the support of IT startups. This allowed for a comprehensive systemic understanding of the
innovation policy, including financing and education aspects, and the identification of negative effects of innovation. The
reviews effectively listed the systematic problems that needed to be addressed. It took a systemic perspective that
stakeholders found important.
The I4SDR has just been launched, and the dispersal of this knowledge will be key to development of innovation in
Armenia. However, respondents report that there is a lack of promotion of this product, and that this now needs to be
amplified by the government partners. For the future, a focus on promoting and implementing suggestions from the
I4SDR is advised. The project also provided a comparison (benchmark) of different countries, which stakeholders found
useful, stating that competition between countries is a strong driver of change. It is an excellent sign that UNECE and the
government immediately engaging in drafting an actionable roadmap, based on the recommendations.
58
In addition, the project has helped to promote the encouragement of the Armenian diaspora to invest in innovative sectors.
Sector-specific initiatives should also further development to promote innovation, including agriculture, health care, and
environment. This involves implementing targeted policies, providing financial support, and creating specialized programs
to foster innovation and entrepreneurship within these sectors.
Effectiveness: To what extent were the project objectives achieved?
(a) Improved policy dialogue on promoting sustained economic growth, innovative development, and greater
competitiveness in the UNECE region
The policy dialogues were fully achieved and beneficial to Armenia. They provide a platform for the three different
ministries involved in innovation to actually talk with each other and figure out who does what on innovation, which did
not occur previously. The government now organizes policy dialogues every three months, inviting representatives from
various sectors and experts. Over time, respondents have noticed that these dialogues have improved significantly, and it is
essential for these to continue to improve communication.
(b) Improved understanding at the national level of policy options to promote sustained economic growth,
innovative development, and greater competitiveness.
Understanding can be improved. There is a significant divide in innovation policy among the three ministries: Ministry of
High Tech, Economy, and Science; Ministry of Education, Culture, and Sports. The review provided a valuable opportunity
to address this issue, as there was limited communication among the ministries, resulting in overlaps and unaddressed work.
The roadmap will be important in mitigating these issues.
(c) Enhanced national implementation of UNECE policy recommendations and standards on promoting a policy,
financial, and regulatory environment conducive to sustained economic growth, innovative development, and
greater competitiveness.
The I4SDR has just been launched on July 7, 2023; thus policy recommendations have not yet been implemented. The IPO
presented guidelines which were considered, and there is awareness of them, although implementation has not occur yet.
For example, as a direct benefit of this project, there are untapped aspects of innovation that have been opened. Venture
capital from the Armenian diaspora and tech transfer for example will be explored for the first time in policy. Again, the
roadmap is an important step into a direction of implementation and impact.
59
Beyond these objectives, what has been your strategic vision for the project, and how has it changed over the course of the
project?
Innovation has not always perceived as a means of development; however it is highly important. Initially, there was a lack
of dialogue among policymakers regarding the importance of innovation and what innovation actually means. The review
played a crucial role in introducing this new perspective, as it was a novel concept for them. The UNECE work has been
particularly influential in this regard.
Which factors facilitated or hindered the achievement of the project results? What approaches worked well and could be
adapted to work in other sub-regions? What are some relevant lessons learned?
The team's way of working was highly appreciated. They were disciplined with their work plan. They maintained close
communication and coordination with the ministries throughout the project, which facilitated the achievement of the desired
results. This approach proved to be effective in ensuring progress and success.
“The project team is great… it was not just their formal obligations and duties, but their attitude. They were really engaged
in the project, this specific project team… It was a learning experience to work with them, for example, with their work
ethics.”
However, there were certain factors that posed challenges and hindered the project's outcomes. Stakeholders found that
distance from the well-organized project staff hindered the project’s development. During COVID, for example,
consultations had to be conducted virtually. This limited the amount of time the project team and experts spent in the country
and reduced the level of engagement with national experts. To enhance future projects, stakeholders recommended
involving more local experts from Armenia to keep everyone engaged. It’s important to continue with local capacity
building. One recommendation from Armenian respondents was to engage one or two local project managers in
country. It was positive that UNECE engaged comprehensively local partners (local consultants, UN country teams,
Swedish embassy, et al.).
The project also recognized the need to address the silo-based, top-down nature of the public sector. It emphasized the
importance of bottom-up approaches, allowing public servants in general, not just supervisors, to have a say in decision-
making processes. Building capacity through a consistent set of activities was considered crucial. The project sought to
establish in-country teams to ensure continuous engagement and effective collaboration with the government.
60
From a project design perspective, the methodology adopted worked well and can be adapted for use in other sub-regions.
This approach contributed to the project's success and allowed for comparisons and benchmarking with others in the region,
which remains a useful incentive.
Recommendation: Now that the I4SDR has been launched in Armenia, it would be useful to work on incorporating
its recommendations into policy design for policymakers, through workshops and training events with relevant
stakeholders. An additional benefit may be to instill a local project management team to coordinate and build
capacity. Benchmarking with other sub-regional counterparts should ideally also continue to allow for comparisons
and instill a healthy sense of competition – and regular reviews of this.
How effectively has the project improved the competencies of innovation policymakers to support environmental
sustainability, gender equality, good governance, and economic growth?
The project has promoted good governance and economic growth, less in environmental sustainability and gender.
There is talk about gender and environment, but it is mainly pushed by international organizations and not by the
government or stakeholders in country. There is no real demand for this in country. It is the presence of UN organizations
that encourages focus on gender and environment.
How have gender, human rights, and disability perspectives been integrated into the project? What results have there been
in terms of gender, human rights, and disability?
Gender, human rights, and disability have not been viewed as a priority, or even a problem, by Armenian stakeholders.
Statistics and data are lacking in these areas. Armenia has some statistics on gender, for example, gender inequality in
companies, but less on disability and even less on human rights.
Indicators would be useful for keeping track of gender, human rights, and disability perspectives, which adds
visibility to these issues and pressure from international organizations.
Conclusion: The project has made great strides in Armenia to engage and coordinate local stakeholders on innovation, and
to set up a platform on which implementation of innovation policy can now be launched through the IPO and I4DR, which
has taken a welcomed country-driven perspective.
61
As an external entity, the project was restricted in terms of the duration of missions, as experts and UNECE team could
only fly in for a week or two, and managed the project from outside the country. Increasing project management in country,
or project co-management, would increase understanding and build capacity and trust with the beneficiaries, and is called
for by country representatives. Cooperation with local partners (local experts, UN country teams, Swedish embassy, et al.)
went a long way in mitigating this issue. While current capacity development efforts were effective, there is a need to ensure
practical comprehensive capacity building in country.
“Participating in this project has contributed to understanding – it is on-the-job training for policymakers.”
Policy targets, such as in the Innovation Policy Outlook, allows benchmarking as a country, and provide
reinforcement government and regional comparisons. Stakeholders believe this will drive the implementation of
recommendations.
As there are distinct borders between Ministries (Ministry of High Tech, Ministry of Economy, Minister of Culture, Science,
Sport, and Education which encompasses the Science Committee), and it is often unclear who is pushing what forward, a
local project management team would be useful to organize the regular meetings and push an innovation agenda
forward.
“We have a good basis for starting evidence driven dialogue now, and we need ownership and regular data collection. It
can’t be just this one-time only data collection. The results of this will be important for structuring discussion amongst the
stakeholders to address key gaps.”
Recommendation: The next iteration of the project should now focus on implementation of the I4SDR Review and
local capacity. Multiple stakeholders recommended taking the project forward with a local project management
team and local experts if possible, to organize, or co-organize, workshops and meetings on innovation.
Indicators and a regular IPO or benchmarking can be conducted, so that performance can be seen over time.
Generating policy is a changing, adaptive process, and the government is keen on collecting indicators. Data and
innovation indicators are currently lacking.
The Armenian government is the largest buyer of goods and services – more than $1 billion. But procurement is rigid, risk
averse, and specific. Stakeholders have not heard much for example about implementing innovation procurement yet. This
may be an agenda item for the future, and the experts and/or colleagues from Georgia’s successful experience could be
introduced.
62
Georgia
A lower-middle-income economy at the crossroads of Eastern Europe and Western Asia, Georgia is an innovation achiever,
gaining high scores in the Global Innovation Index. The Georgia Innovation and Technology Agency was instituted in 2013
to co-ordinate all aspects of science, technology, and innovation policy. Important challenges remain however, especially
in further reforms of institutional project reviews and education.
Project activities aimed to contribute to on-going or planned reform efforts that address central problems that hold
innovation systems in back in Georgia. They aim to do so by developing assessment tools that help policy makers and other
stakeholders understand and benchmark innovation policies and institutions, coupled with targeted capacity building that
feed into specific reform efforts or address areas of joint concern. Georgia has made significant strides in adopting
legislative and policy reforms to foster gender equality, for example.
There is a new innovative procurement law just adopted in Georgia as a direct result of this project. The new law aligns
with EU directives and regulations and will come into effect in 2025, and it includes provisions for innovative procurement.
This was a great example of coordination on the project, as GITA (Georgia’s Innovation & Technology Agency) initially
came up with the idea of involving the State Procurement Agency. It was fortunate timing, as the Georgian government
was revising its procurement law, and included innovative procurement as part of this project with expertise from
independent advisors from the project with whom they are still in contact. They have together actively developed a
groundbreaking law and multiple pieces of secondary legislation, including guidelines and training materials. In addition,
a delegation of high-level public officials from our agency and GITA visited Norway on a well-received Study Tour to
learn from their experiences and gain practical knowledge in the field of innovation.
This will have many positive downstream effects in the future for innovative products and services. Trainings and awareness
raising among businesses and procurement authorities are already underway regarding this new procurement method. This
is a big and lasting achievement for the project.
To what extent was the project design appropriate for meeting the needs of beneficiary countries?
63
The project design was appropriate. Stakeholders were gathered, including key government representatives. Experts were
hired to identified challenges in areas such as creating venture capital funds, improving government procurement
procedures, and commercializing research and development.
To address these challenges, trainings were conducted and additional support was successfully sought from the United
Nations Sustainable Development Cooperation and the Ministry of Foreign Affairs. The Innovation for Sustainable
Development Review compared different countries' innovation ecosystems and provided recommendations for policy
improvements.
The Study Tours were greatly beneficial. Delegates from the State Procurement Agency and others visited Norway to learn
from their experiences in innovation and sustainable development first-hand. This Study Tour garnered very positive
feedback (rated 9.7 out of 10).
To what extent were the project objectives achieved?
Yes, all objectives have been achieved. The project improved policy dialogue among key stakeholders, enhanced
policymakers' understanding of innovation, and implemented policy recommendations and standards. Now, more focus on
implementation is vital for success. Research, selection, and adoption of new laws, like the procurement law for example,
have been positive developments.
Further raising awareness of the project in Georgia amongst other donors would be useful. Resources could be shared and
other donors may want to contribute to or partner to work on aspects of the project.
“When I had meetings with, say, UNDP, UN Women, World Bank, and the EU, they didn’t seem to know about all the
details of the project.”
Project activities were effectively coordinated. There have been peer reviews and information exchange among countries
in the region, creating a good networking opportunity that was highly appreciated by stakeholders.
“We have conducted so much research on all different ways to innovation in the region. So many new topics, ideas, and
analysis/results came up! … This was capacity building for me. This was new to me before… All the experts from different
countries did peer review of each other’s work, and I’d never seen that before. This is a good strategy.”
64
Challenges were addressed appropriately by the project. Three challenging areas stakeholders identified in Georgia were:
(1) VC fund creation, (2) R&D commercialization, and (3) Procurement. To address these challenges, GITA worked closely
with the Ministry of Economy and other government entities, consulted with experts from the Ministry of Foreign Affairs
to ensure our actions align with the country's overall goals, and conducted ICT trainings funded by the European Union to
develop a competitive workforce and attract investments in the ICT sector. This was an additional positive collaboration of
the project with country stakeholders.
One significant factor that facilitated success was the work of the dedicated UNECE project team as well as selection of
local partners who invested their time and effort into the project. This sense of co-ownership remains crucial in moving
forward.
How have gender, human rights, and disability perspectives been integrated into the project? What results have there been
in terms of gender, human rights, and disability?
The new procurement law includes provisions for reserve contracts to promote companies owned by people with disabilities
or belonging to ethnic or gender minorities. These provisions aim to support local community-based organizations and
women-owned businesses. The State Procurement Agency is currently working with the Chamber of Commerce and
Industry to define the criteria for women-driven businesses.
During the project, the focus was on procurement and innovation, rather than gender or disability issues. However, outside
of the project, the State Procurement Agency has conducted training programs for disabled individuals to encourage their
participation in public tenders.
In general, gender, environmental, and social issues have been given specific attention and integrated into the project design.
These aspects are integrated in the reports. This is something however that is promoted mostly by international donors.
Now, the key lies in the practical implementation and whether these aspects will be effectively translated into actions in
country.
Providing specific guidance and support mechanisms to assist stakeholders in utilizing the project's policies and
recommendations would be beneficial.
What ensures that project outcomes would continue after the project ends?
65
Multiple stakeholders in both Armenia and Georgia have requested country comparisons, as it stimulates achievement
amongst countries. Both Armenian and Georgian stakeholders have said that it would be highly beneficial for them to see
how better-developed countries in Europe have developed their innovation policy and benchmarked themselves against the
same indicators, such as the Global Innovation Index. Stakeholders also want to work on implementing the
recommendations of the I4SDRs.
“It is important going forward to have diagnostic studies, as it is important to understand where we stand.”
The engagement of government stakeholders and the establishment of collaborative networks between countries in the
region contribute to a higher likelihood of lasting results. It would be good to raise awareness about the project's goals and
achievements among the wider audience, including other UN bodies, donors, and relevant organizations. This could help
foster collaboration and ensure a more coordinated approach towards economic development, sustainability, and innovation
in Georgia.
“The project outcomes have a good chance of continuing after the project ends. Many of the recommendations have been
embraced by state entities (like the procurement law), and they are already implementing them. Policymakers have shown
a willingness to change strategies and reform agendas based on the recommendations. This demonstrates the project's
sustainability, as its impact extends beyond its duration.”
- I Introduction
- 1.1 Project Background
- 1.2 Evaluation Purpose and Scope
- 1.3 Sampling strategy
- 1.4 Evaluation Methodology
- 1.5 Evaluation questions
- 1.6 Leaving No-One Behind
- 1.7 Limitations
- II. EVALUATION FINDINGS
- 2. Relevance and coherence: Was the project doing the right thing?
- 2.1 Appropriateness of project design for meeting the needs of beneficiary countries
- 2.2 Integration of gender, human rights, and disability perspectives
- 2.2.1 Gender, human rights, and disability perspectives: project results
- 2.2.2 Potential including gender and human rights perspectives better in future project design and implementation
- 3. Effectiveness: Were results achieved, and how?
- This section assesses the project results' achievement using the following sub-criteria: i) overview of project objective achievement; ii) evolvement of the project's strategic vision; iii) unintended effects; iv) factors affecting project performance...
- 3.1 Overview of achievement of project objectives
- 3.2 Strategic project vision and how it evolved
- 3.3 Unexpected effects
- 3.4 Factors affecting project performance
- 3.5 Lessons learned
- 3.6 Changes in competencies of innovation policymakers
- 3.7 Coordination with other UN stakeholders
- 3.8. Challenges and mitigation
- 4. Efficiency: were resources used appropriately to achieve project results?
- 4.1 Adequacy of funding for project results
- 4.2 Timeliness of results achievement and efficient organization
- 4.3 Efficiency of resource use
- 5. Sustainability: are results lasting?
- 5.1 Measures to ensure sustainability of project results
- 5.2 Ownership of project results, institutionalization, and up-scaling
- 5.3 Potential for replication
- Section III: Conclusions and Recommendations
- Annex 1: Terms of Reference of the Evaluation
- I. Purpose
- II. Background
- III. Evaluation objectives, scope and questions
- Relevance
- Effectiveness
- Efficiency
- Sustainability
- IV. Evaluation approach and methodology
- V. Evaluation schedule9F
- VI. Resources and Management of the evaluation
- VII. Intended use / Next steps
- VIII. Criteria for evaluators
- Annex 2: Documents reviewed
- Annex 3: Lists of stakeholders interviewed
- Annex 4: Evaluation matrix
- Annex 5: Country case studies: Armenia and Georgia
