1
Emergency multisectoral humanitarian assistance to conflict-affected people in Ukraine
Final Performance Evaluation
Inception Report
2
Table of Content
Acronyms 3
1. Introduction 4
2. Context and Project background 4
2.1. Intervention logic 6
3. Assessment methodology 7
3.1. Overall approach, objectives and scope 7
3.2. Data collection 10
3.3. Explanation of Sampling and Proposed Sample size 11
3.4. Methods for analysis 13
4. Protecting Data and Personal Identifying Information 14
5. Risks and possible limitations 14
6. Resources and logistic support 15
Annex I: Logframe of the project Emergency multisectoral humanitarian assistance to conflict-affected
people in Ukraine 16
Annex 2. Deliverables and Milestones, Schedule 17
Annex 3. Selection of Stakeholders for in-depth interviews 18
Annex 4. Evaluation Matrix 19
Annex 5. Proposed Data Collection Tools / Protocols 20
Annex 5. Final Performance Evaluation. Terms of Reference 21
3
Acronyms
CATI Computer assisted telephone interview
CC Community centre
FGD Focus group discussion
GCA Government controlled areas
II In-depth interview
KII Key informant interview
MEL Monitoring, evaluation and learning
NFI Non-food item
NGCA Non-government controlled areas
PAH Polish Humanitarian Action
PIT Performance indicator table
PSS Psychosocial support
WASH Water, sanitation and hygiene
4
1. Introduction
By the USAID/BHA evaluation policy, the Polish Humanitarian Action Mission in Ukraine ordered the summative
performance evaluation of its ‘Emergency multisectoral humanitarian assistance to conflict-affected people in
Ukraine’ Project. The project’s implementation period is August 20, 2021 – August 19, 2023. Vox Populi Agency
has been selected to realize the evaluation. The purpose of this inception report is to present a detailed approach
to this task. The report includes an overview of the project and its intervention logic, evaluation methodology,
evaluation questions judgment criteria, sampling approach and size, and possible research limitations and risks.
This report also details the strategies to collect and analyze data. It presents data collection tools for the target
audiences, an evaluation work plan, and a tentative list of key informants to select.
A separate section of the report explains how personally identifiable information will be protected and secured
at all stages, from collection to reporting. The report references relevant Ukrainian norms regulating research
involving human subjects and specifies how the measures will be applied in this assignment to comply
accordingly.
2. Context and Project Background
The protracted crisis in eastern Ukraine escalated to a full-scale military conflict with at least 17.6 million people
in need, according to the UN OCHA Humanitarian Needs Overview 2023. This includes 6.3 million internally
displaced people, 4.4 million returnees and 6.9 million people who have remained at their homes throughout the
war. The daily hostilities result in a sharp increase of civilian casualties (23,015 civilian casualties, as of April 17,
2023), damaging critical infrastructure and severely impacting the daily lives of the people residing in the areas
and IDPs, and generating multiple humanitarian needs. Besides direct damage from shelling and forced internal
displacement of up to 6.3 million people, an affected population with psychosocial problems, destroyed
livelihoods, no access to proper WASH facilities, no capacity to cover basic needs, and often with no means to
practice their rights appeared as a result of the armed conflict. The consequences of the conflict are more severe
for those already vulnerable. Thus, many older people living along the contact line in Donetska and Luhanska
GCAs and in NGCA have their well-being significantly degraded because of stress from continual shelling, lack
of support from displaced relatives and friends, mobility restrictions, chronic diseases getting worse, poor access
to quality water, and limited resources to cover their basic food needs. Recognizing the complexity of the
humanitarian and protection needs of the conflict-affected people in Ukraine, the evaluated project applies a
multi-sectoral approach.
Initially, Polish Humanitarian Action (PAH) was implementing the project titled Protection and humanitarian
assistance to highly vulnerable conflict-affected older people living in Donetska and Luhanska Oblasts funded
under USAID/BHA grant (Award No. 720BHA21GR00400). The project implementation period was from 20
August 2021 until 19 August 2022. The total amount of the project’s budget is USD 1150,000,00. The program
Goal was to reduce the vulnerabilities of the conflict-affected older persons, residing in Donetska and Luhanska
Obasts through provision of emergency assistance in terms of protection, cash assistance, WASH and food
security. The project is being implemented in locations in area along the “contact line”, the most conflict-affected
areas in Donetska and Luhanska GCAs, and in the locations throughout Donetska NGCA, where PAH is
operating through its local partner.
The project before revision and extension consisted of three main components: 1) Protection, 2) WASH
(Sanitation and WASH NFIs), 3) Food Security. PAH’s approach to protection included individual PSS, individual
psychological counselling, home-based care, and legal assistance provided by the employees of Community
Centers, as well as other project specialists remotely. In Donetska NGCA, protection was focused on individual
PSS sessions based on a simplified program, home-based care. It was planned that hobby clubs would include
5
elements of PSS program to provide group PSS to their participants in the Community Centers (based in the
local Territorial Centers). In GCAs, older people with mobility restrictions were provided with assistive devices
that enabled them to be more independent in their daily tasks. In Donetska NGCA, older people were provided
with food kits to cover their needs during the winter season. In Donetska NGCA, protection activities and
distribution of food kits were done by PAH Partner “Charitable Foundation “Donbass Development Center”
(Further – CF DDC). In order to implement programming in Donetska NGCA, each organization should receive
accreditation of the project activities to be implemented from de facto authorities. In 2020, PAH’s Partner has
received accreditation from de facto authorities to implement such project activities as distribution of food and
WASH NFIs kits (including for people with special needs). Protection activities started being implemented along
with the distribution of the kits. In January 2021, CF DDC received rejection to continue further implementation
of projects activities (distribution of assistive devices as well). In the view of the received rejection, protection
activities were implemented only partially. PAH through UN OCHA strived for negotiation with the de facto to
receive accreditation; the process is ongoing.
To improve access to safe water for conflict-affected population PAH planned to additionally conduct
rehabilitation works in selected water supply sites, but did not manage to do it before invasion.
Based on the changes that took place since the early morning of February 24, 2022, PAH put in place an updated
set of contingency plans. Then implemented activities were suspended due to the ongoing conflict and were
replaced with multisectoral emergency response. The remaining amount of $ 789 105 USD together with and
additionally requested $ 3 210 895 USD are redirected to the activities within the following components 1)
Protection, 2) Food Security, 3) Multipurpose Cash Assistance (MPCA), 4) WASH, and 5) Shelter & Settlements.
After revision and approval of the changes by BHA the project title is Emergency multisectoral humanitarian
assistance to conflict affected people in Ukraine. It aims at reducing the suffering and improving well-being of
conflict affected people in Donetska, Luhanska, Kyivska, Zaporizka, Dnipropietrovska and Sumska oblasts,
namely through the following activities:
1) Providing vulnerable men and women with home-based care and individual PSS for the most vulnerable older
people, psychological first aid (PFA), and psychological consultations for conflict-affected men, women, and
children to improvetheir psychosocial well-being;
2) Providing most vulnerable conflict-affected people with MPCA (in the areas feasible for CBA) to enhance their
ability to cover basic needs;
3) Rehabilitating critical WASH infrastructure and distributing WASH NFI kits in collective centers for IDPs/social
institutions and establishing WASH facilities (toilets, hand washing stations) near the border crossings to improve
access to dignified hygiene conditions;
4) Providing highly vulnerable conflict-affected men and women with food kits containing essential food items to
enhance their ability to meet basic food needs;
5) Providing NFI kits at both institutional (collective centers) and household levels to improve affected
population’s shelter solutions.
Being present in Ukraine since August 2014, PAH established a full-fledged structure, with its country office in
Kyiv and field offices – in Kramatorsk and Bakhmut (Donetska GCAs). PAH activities are funded by a broad
range of donors: USAID/BHA, UNICEF, UHF and Polish Ministry of Foreign Affairs with additional funds
originating from Polish society. PAH has experience operating in both GCA and NGCA and a strong expertize
in implementing projects with focus on psychosocial support and developing capacities of community members,
6
while taking into consideration long term impact and sustainability. For more than six years of presence in
Ukraine, PAH has implemented following activities: 1) Protection, 2) MPCA, 3) WASH, 4) Food assistance and
livelihoods. While implementing these activities, PAH has built strong working relationships with local authorities
(GCA), sub-awardees (both GCA and NGCA) and local communities. Moreover, since establishing first
Community Centres for older people in 2018 and by the beginning of 2022, PAH has trained more than 100
community workers in provision of PSS in more than 20 locations in the areas within 0-10 km along that time
contact line. However, due to the invasion a number of CCs had to terminate their work and the staff of the
community workers decreased.
PAH is an active member of the Protection and Health Clusters, WASH Cluster, Food Security and Livelihood
Cluster, Cash Working Group, MHPSS TWG, and Age and Disability Working Group. Regular updates on PAH
activities, covering their nature, targeted beneficiaries, covered locations and sources of funding are shared with
the Clusters/WGs on a regular basis in order to ensure proper coordination of humanitarian activities and
referrals of beneficiaries. To minimize overlaps and gaps, PAH regularly shares information about its activities
and needs of beneficiaries with other NGOs and UN agencies, including People in Need, HelpAge International,
UNHCR, Proliska, Caritas, NRC, ASB and others.
2.1. Intervention logic
The project’s hypothesis is the following:
If the conflict-affected people are provided with the emergency life-saving assistance in terms of
protection, WASH, food, NFI, shelter and MPCA, then they will reduce their vulnerabilities and increase
their resilience towards the consequences of the conflict.
The project’s principal objective is: To reduce suffering and improve well-being of people affected by the conflict
in Ukraine through emergency time-critical assistance and ensuring access to basic essential services.
Recognizing the complexity of humanitarian and protection needs of the conflict-affected people in Ukraine, the
programme is applying a multi-sectoral approach. Specifically, the programme aims at achieving the following
objectives:
- Objective 1: To improve psychosocial well-being of conflict-affected men and women, boys and girls;
- Objective 2: To improve ability of highly vulnerable conflict-affected population to meet their basic needs;
- Objective 3: To improve access to dignified hygiene conditions for conflict-affected population;
- Objective 4: To enhance ability of conflict-affected older population to meet their basic food needs;
- Objective 5: To increase access of conflict-affected men and women to NFIs.
The chain of project activities from objectives to outputs and results that details what PAH aimed to achieve and
how it was measured, is shown in Annex I: Logframe of the project Emergency multisectoral humanitarian
assistance to conflict-affected people in Ukraine.
For the project’s hypothesis the underlying assumptions and external factors that may influence program
performance are as follows:
- The large-scale military offensive does not escalate in proposed project sites and does not result in
interruption or suspension of project activities.
- Political situation enables to conduct project activities and reach the beneficiaries in selected project
locations.
7
- There is a sufficient number of recruited, trained and qualified staff to provide high quality assistance
relevant to the needs of the affected people
- Suppliers and/or local workers are reliable and transparent in the areas of project implementation,
including compliance with policies, law and procedures.
- Beneficiaries are not exposed to harm, abuse and/or violence during participation in project activities as
unintended negative effect of the action.
- Safety of beneficiaries.
- Prices and currency exchange rates will remain stable as foreseen during the period of project
implementation.
- Stability of the market and access of beneficiaries to market to meet their basic needs.
- The most vulnerable people are targeted by the project.
- Contractors/vendors/ suppliers responsible for provision of services are able to provide them to
beneficiaries according to the requirements.
3. Assessment methodology
3.1. Overall approach, objectives and scope
The main aim of this evaluation is ensuring accountability to beneficiaries and improving PAH’s program
effectiveness. The results of the evaluation will be described and grouped into three main sections: findings,
conclusions, and recommendations. The recommendations will be incorporated into PAH’s future programming
in Ukraine as well as shared with PAH HQ to be potentially incorporated into programming at other PAH missions.
The evaluation will follow the BHA Guidance for M&E1
, the American Evaluation Association (AEA)
recommendations and Quality Standards for Development Evaluation of Organization for Economic Cooperation
and Development (OECD)2
. The evaluation team will use such OECD DAC evaluation criteria as
relevance, effectiveness of the intervention for analyzing and assessing how the Project achieved its results to
date. Efficiency, impact, sustainability and coherence criteria are not in the focus of this evaluation.
The evaluation will be inclusive in terms of collecting perspectives of a broad range of stakeholders.
To ensure the accuracy and validity of the study findings the team will ensure that:
● More than one data source is used for the key findings of the research (triangulation);
● Respondents are aware that there are no right or wrong answers to the evaluators’ questions, and can
speak freely;
● The team members demonstrate neutral and independent behavior with non-biased views;
● Conclusions of this evaluation are clearly based on findings, and recommendations are clearly based on
conclusions;
● All analytical deliverables are practical, easy to read and usable for PAH, USAID/BHA and other
1
Bureau for Humanitarian Assistance Technical Guidance for Monitoring, Evaluation, and Reporting for Emergency Activities.
https://www.usaid.gov/sites/default/files/2022-05/BHA_Emergency_ME_Guidance_February_2022.pdf
2
https://www.oecd-ilibrary.org/docserver/9789264083905-
en.pdf?expires=1599754118&id=id&accname=guest&checksum=8365C8007DC39C9C12CBD7F3B78FF6DB
8
stakeholders.
Assessment objectives
The objectives for this summative performance evaluation are as follows:
1. To evaluate the effectiveness and relevance of the project activities in relation to the activity’s purposes and
results;
2. To identify best practices and lessons-learnt in the activity design and implementation for achieving the
activity’s expected results.
Evaluation questions and evaluation framework
To assess the relevance and effectiveness of the selected project activities, this evaluation will answer 3
evaluation questions, presented below:
1) Relevance: To which extent have the activity objectives, design and results addressed beneficiaries needs
and priorities, including across different population groups?
2) Effectiveness: To what extent were the outcomes and objectives achieved / are likely to be achieved?
3) Effectiveness: What were the major factors influencing the achievement or non-achievement of the objectives?
In order to achieve the objective 2 the study will answer additional questions regarding best practices and
lessons-learnt:
1) Which practices (cases) related to design and implementation are the most effective for achieving the activity’s
expected results and might be incorporated into PAH’s future programming in Ukraine as well as shared with
PAH HQ to be potentially incorporated into programming at other PAH missions?
2) Which practices used by the Project, on the contrary, are less effective in Ukrainian conditions?
The primary output and outcome-level indicators of interest this evaluation will use to assess the overall
performance of the project are the following:
BHA Indicator
No. Indicator Title
C1
Percentage of people reporting improvement of psychosocial well-being, by sex
and age (CUSTOM)
C2
Percent of beneficiaries reporting that humanitarian assistance is delivered in a
safe, accessible, accountable, and participatory manner (CUSTOM)
M2
Percentage of households who report being able to meet their basic needs of
their households (all/most/some/none), according to their priorities
M3
Percentage of beneficiaries reporting that humanitarian assistance is delivered
in a safe, accessible, accountable and participatory manner
M6
Percent of beneficiary households reporting adequate access to household nonfood
items
FS1
Percent of (beneficiary) households by Food Consumption Score (FCS) phase
(Poor, Borderline, and Acceptable)
M8
Percentage of households who have reduced essential WASH related basic
needs expenditures
K01 Total USD value of cash transferred to beneficiaries
9
W19
Percent of latrines/defecation sites in the target population with handwashing
facilities that are functional and in use
W26
Percent of households reporting satisfaction with the contents of the WASH
NFIs received through direct distribution (i.e., kits) or vouchers
W28
Percent of households reporting satisfaction with the quality of WASH NFIs
received through direct distribution (i.e., kits), vouchers, or cash
W13
Number of individuals directly utilizing improved sanitation services provided
with BHA funding
W20 Average number of users per functioning toilet
FS1
Percent of households with poor, borderline, and acceptable Food Consumption
Score (FCS) (BHA).
C3
Percent of households reporting satisfaction with the contents of the food kits
received through direct distribution (CUSTOM)
C4
Percent of households reporting satisfaction with the quality of food kits
received through direct distribution (CUSTOM)
FS2 Mean and median Reduced Coping Strategy Index (rCSI) score (BHA)
F02
Percent of households where women/men reported participating in decisions on
the use of food assistance
C5
Percent of beneficiaries reporting that humanitarian assistance is delivered in a
safe, accessible, accountable, and participatory manner (CUSTOM)
S05
Number and percent of beneficiaries reporting satisfaction with the quality of the
NFIs received
P6 Number of individuals participating in psychosocial support services (BHA)
M1
Total number of individuals (beneficiaries) assisted through multipurpose cash
activities
W25
Total number of individuals receiving WASH NFIs assistance through all
modalities (without double-counting)
C6
Number of WASH NFIs kits provided to collective centres for IDPs/social
institutions
F1 Number of beneficiaries receiving food assistance
S7
Number and percent of beneficiary households receiving NFIs in identified
settlement(s) through use of in-kind NFIs
C7 Number of NFI kits provided to the collective centres (CUSTOM)
3.2. Data collection
Due to the security situation, degraded transportation availability and in order to be able to reach beneficiaries,
data collection will be realized online and via phone.
The following data collection methods will be used in this evaluation:
● desk review of the project’s documentation and reports,
● key informants’ interviews (KII) with the implementing partners (PAH country office and regional offices
representatives), CCs representatives, partner organizations, volunteers;
● in-depth interviews (II) with local stakeholders (local authorities in programme locations, management of
the assisted social and shelter facilities);
● focus group discussions (FGD) with beneficiaries of the programme;
10
● CATI interviews with beneficiaries of the programme.
During the desk phase, the evaluation team reviewed the provided project’s documentation, including grant
proposal narrative, MEL plan, MEL and performance reports, PIT to strengthen its understanding of the project
design, implementation process, achieved results and the context in which it had been operating. Information
from the MEL and performance reports was used for planning the sampling approach and preliminary selection
of the most informative locations as units of the sample.
We will continue data collection with the key informant and in-depth interviews with the implementing partner,
CCs, partner organizations’ representatives, and local stakeholders to start with the big picture and then go into
details to understand the project implementation process, its effectiveness, and relevance, as well as identify
best practices, barriers, and challenges in realizing of the programme in emergency circumstances in various
project locations.
Focus group discussions and CATI survey will be held as the next step to collect data from the project’s
beneficiaries. Two online focus groups discussions and 600 of CATI interviews are planned for data collection.
Individual/group interviews will be based on the protocols adjusted towards each target group – implementing
partner representatives, CCs and partner organizations’ representatives, local stakeholders – and in line with
the analytical framework. Proposed protocols are presented in Annex 5 of this report. They may be further refined
during the data collection process in case of such a need.
Evaluators will take notes during the interviews/focus groups and will maximize full recording of the interviews in
Zoom or other video conferencing platform. Getting an informed consent of the respondents is the must for
starting the recording. In case when a respondent is opposed to the interview recording, only notes of the
evaluators will be used for analysis and documentation of the evaluation results.
3.3. Explanation of Sampling and Proposed Sample size
Qualitative purposive sample will be used to select respondents for KIIs, IIs, and FGDs. It will cover the following
target groups:
● the implementing partners (PAH country office and regional offices representatives);
● CCs representatives, partner organizations, volunteers;
● local stakeholders (local authorities in programme locations, management of the assisted social and
shelter facilities);
● beneficiaries of the programme.
For planning samples of CCs representatives, partner organizations, volunteers, and local stakeholders, primary
units of selection will be locations where PAH implemented the project. When selecting them, we will take into
account outstanding intensive cases to learn from those that are exemplars of best practices, as well as to
consider the barriers and unfavorable conditions. Informationally reach cases will be chosen to study the
evaluation questions in depth.
To ensure maximum variation, while selecting locations we will take into account the distribution of the project
activities by the geographical regions, type of the communities (urban/settlement or rural), type of implemented
activities. This heterogenous approach to sampling will yield the following kinds of findings:
1) Illustrative detailed stories from each prominent case to document uniqueness;
2) discovered shared patterns that emerge out of heterogeneity.
11
FGDs with beneficiaries of the programme will be conducted with homogeneous groups of participants. To fully
use benefits of this method of data collection we need to recruit people with similar background that will help
them to build rapport and freely share their experiences and thoughts.
For selecting individual respondents, we will consider their knowledge about the project and experience
regarding receiving and using provided aid. We plan to selects FGD participants from quantitative survey
respondents who will express willingness to provide their feedback.
Respondents for the quantitative survey will be randomly selected from the project’s beneficiary database and
we will survey them if they confirm receiving aid from PAH and agree to participate. We expect that an
achieved sample of randomly selected beneficiaries will be close by key characteristics to the distribution in the
population of the beneficiaries of this project. In a case where significant differences will be found between
them, weights might be calculated and applied. PAH representatives will be informed about it and a weighting
procedure will be agreed with them.
Key informants and in-depth interviews
22 interviews are planned as this stage with the following distribution by types of the respondents:
Types of respondents
# of
interviewees
PAH country office and regional offices
representatives 12
CCs representatives, partner organizations,
volunteers 5
Local authorities 2
Management of the assisted social and shelter
facilities 3
Total 22
11 KIIs will be conducted with PAH staff and 1 with USAID/BHA representative. 10 in-depth interviews will be
organized to maximize qualitative representation of different types of locations (communities) covered by the
project taking into account the abovementioned factors – geographical regions, type of community, type of
implemented activities. The draft sample structure based on the results presented in the Project’s documentation
and preliminary consultations with the PAH MEAL team is presented in the Annex 3 of this report.
FGD with beneficiaries of the programme
We will conduct two FGDs: one group with people who received PSS services and another one with beneficiaries
of MCVA, WASH and food assistance. 6-8 participants will take part in the focus group discussion (FGD) to be
conducted online via Zoom or other video conferencing platform.
Beneficiaries’ survey
CATI will be used as a data collection method for this survey. We plan
12
The questionnaire will be developed in collaboration with PAH MEL team. We expect that an interview (including
a respondent selection) with this tool would not exceed 20 minutes. We will conduct a questionnaire pretest and
interviewer briefing before commencing the fieldwork. The final version of the tool will be approved by PAH.
The following script will be used for respondents’ selection and to check the relevance of the provided contacts
for the survey purposes:
- randomly dialed numbers
- if there is a gender/age in the database, it is immediately checked whether it is this person's number. if
not, the number is marked in the database as belonging to someone else
- if there is a gender/age match, then the questions of block A follow. Now I have made it so that we ask
whether they remember any help at all (if not, a note is made in the database that nothing was
remembered for this number).
- If they remembered help, we ask them who it was from. This will include the recall rate without the PGA
prompt (and information about who else they received help from)
- Finally, a direct question about the PHA and what exactly they received. If the PHA did not recall, then
the contact database will indicate that the PHA did not recall this number
- then the questionnaire is filled out if the PHA is remembered
Quality assurance procedures will be a part of daily routine – a fieldwork supervisor will be able to check work of
each interviewer and make corrections if needed.
We plan to callback to 10% of respondents in order to check if the interview was conducted. Special attention
will be paid to interviews with inconsistencies reviled during data logical control.
3.4. Methods for analysis
The assessment will entail qualitative data analysis to uncover the themes relevant to the research
questions. The quantitative data analysis will be employed to find out answers to evaluation questions from the
beneficiaries’ prospective and confirm findings of the qualitative study and desk research.
The main evaluation activities will include mapping and triangulation of evidence through the review of the
Project’s documentation, monitoring data and report and analysis of the interviews and focus group discussions
results, quantitative survey data. To answer the evaluation questions, the evaluation team will obtain data from
more than one type of respondent and triangulate data. Agreement between respondents of different types on a
given question tends to suggest that the received data are accurate. Differences may indicate either inaccuracies
or the possibility that the project had different effects for different types of respondents studied or locations
covered.
We will use a case-based approach to illustrate changes and successful project results, including rich in detailed
descriptions of what were programme’s specific ways for achieving the successful results. This ‘how we learned
to do it’ focus in presenting project results will be helpful for future scaling up of the project experience in Ukraine
or abroad.
The report will analyze both different experiences of the individuals in the project and variations in program
processes at different sites.
Debriefing/validation
13
After submitting the draft evaluation report, we will conduct an online validation meeting with relevant PAH
program and technical staff to collect feedbacks and additional input that will serve to validate initial findings and
recommendations before they are made final.
Table of contents for a final report
The final report should have the following structure:
Table of Contents
List of acronyms and abbreviations
Executive Summary
Introduction
Description of the Intervention
Evaluation Scope and Objectives
Lessons Learned
Findings, Conclusions, and Recommendations
Report Annexes.
4. Protecting Data and Personal Identifying Information
There are no specific laws or norms regulating research involving human subjects in Ukraine, except for clinical
trials involving humans. Conducting social research in Ukraine you should follow general norms defined by
Ukrainian legislation (Constitution, Civic and Criminal Codes). It is widespread that social researchers follow
international ethical standards or a code of professional ethics of a sociologist approved by the Sociological
Association of Ukraine (http://sau.in.ua/app/uploads/2019/07/Kodeks-socziologa.docx).
In the evaluation of the PAH program we plan to apply all the necessary measures in order to correspond to
those norms:
- voluntary participation of respondents in the evaluation - respondents will receive all the necessary information
about the evaluation;
- respondents will provide consent for interview and its recording - confidentiality and anonymity will be ensured
Evaluators will read a brief statement about the purpose of the study and voluntary participation in the interview
and will answer respondents’ questions before proceeding with interview.
We will secure and protect the data collected from the respondents of this evaluation study, particularly
personal identifying information, taking the following actions:
● interviews log file and timetable – containing personal identifying information, e.g names, emails, and
phone numbers of the respondents, will be shared exclusively among evaluation team members and
PAH MEAL team;
● specific codes will be applied for all informants to anonymize the interview recording and the interview
notes’ files;
● if there will be a need to quote individual participant in the report, only key words may be presented,
taken in quotation mark, but without any identifying details – such as location, respondent’s position or
name;
● the data collected from respondents will be kept separately from the interviews’ log file and timetable;
● interviews and FGD recordings will be kept safely – on personal password-protected computers of the
evaluation team members - and erased after approval of the evaluation report.
14
5. Risks and possible limitations
The main factor that may influence realization of this evaluation is an ongoing security situation due to which
access to respondents might be deteriorated, as well as worsen evaluation team working conditions (e.g.
electricity blackouts). Drastic changes in the security situation may cause delays in the field data collection.
The survey of beneficiaries is the component prone to potential limitations most of all. Due to the fact that
respondents might receive assistance from various sources it might be difficult to recollect aid from the
USAID/BHA programme implemented by PAH. On the other hand, it might be difficult for beneficiaries to
remember details about assistance and, respectively, to provide an opinion about it.
Moreover, participation in the in-depth interviews for project assessment purpose may be new and worrying
experience for some stakeholders making them reluctant to take part in the evaluation.
Effective planning and support from PAH staff will help to find timely solutions for possible delays and problems
in communication with project’s technical staff, stakeholders and beneficiaries.
6. Resources and logistic support
The evaluation team will use Vox Populi resources for organization, data collection and analysis of the results in
this evaluation. We rely on support of the PAH team in finding most informative cases for the purposes of this
evaluation. An official letter of support from PAH might be needed to contact stakeholders and unroll effective
recruiting and data collection there.
15
Annex I: Logframe of the project Emergency multisectoral humanitarian assistance to conflict-affected people
in Ukraine
Principal Objective: To reduce suffering and improve well-being of people affected by the
conflict in Ukraine through emergency time-critical assistance and ensuring access to basic
essential services
Specific
Objectiv
es
Results Outputs Verifiable indicators Data Collection
Method
Objectiv
e 1: To
improve
psychos
ocial
wellbeing
of
conflict-
affected
men and
women,
boys
and girls
Result 1: 11
500 conflict-
affected
men and
women,
boys and
girls,
targeted by
the program,
improved
their
psychosocial
well-
being
Output 1: 11
500 conflict-
affected
men and
women,
boys and
girls,
targeted by
the program,
participated
in
psychosocia
l support
services
1. Number of individuals
participating in psychosocial
support services (BHA)
2. Percentage of people
reporting improvement of
psychosocial well-being, by
sex and age (CUSTOM)
3. Percent of beneficiaries
reporting that humanitarian
assistance is delivered in a
safe, accessible, accountable,
and participatory manner
(CUSTOM)
1. Project documents
and records review
2. Ex-ante and expost
structured
surveys with all PSS
beneficiaries targeted
by the programme
initially in Donetska
and Luhanska oblast
3. Endline study/PIM
study for beneficiaries
targeted by the
programme after
invasion OR selfreported
questionnaire
16
Objectiv
e 2: To
improve
ability of
highly
vulnerab
le
conflict-
affected
populati
on to
meet
their
basic
needs
Result 2:
5000 highly
vulnerable
conflict
affected
men and
women
improved
their ability
to cover the
basic needs
(including
but not
limited to
food needs
and access
to essential
WASH
NFIs)
Output 2:
5000 highly
vulnerable
conflict
affected
men and
women
assisted
with MPCA
Output 3: 1
110 000
USD in cash
transferred
to
beneficiaries
1. Total number of people
assisted through multipurpose
cash activities
2. Percentage of households
who report being able to meet
their basic needs of their
households
(all/most/some/none),
according to their
priorities
3.Percentage of beneficiaries
reporting that humanitarian
assistance is delivered in a
safe, accessible, accountable
and participatory
manner
4. Percent of beneficiary
households reporting adequate
access to household non- food
items
5. Percentage of households
by Food Consumption Score
(FCS) phase (Poor, Borderline,
and Acceptable);
6. Percentage of households
who have reduced essential
WASH related basic needs
expenditures
1. Project documents
and records review
2. Survey (structured
interviews with
beneficiaries)
3. Review of financial
documents
17
Objectiv
e 3: To
improve
access
to
dignified
hygiene
conditio
ns for
conflict-
affected
populati
on
Result 3:
Highly
vulnerable
conflict-
affected
men and
women, who
are
residents of
collective
centres for
IDPs/social
institutions
improved
access to
dignified
hygiene
conditions in
WASH
facilities in
line with
applicable
standards.
Result 4:
1350 highly
vulnerable
conflict-
affected
men and
women,
boys and
girls have
increased
access to
essential
WASH nonfood
items
(NFIs), as
defined by
Sphere or
Output 4:
Highly
vulnerable
conflict-
affected
men and
women, who
are
residents of
collective
centres for
IDPs/social
institutions
directly
utilizing
improved
sanitation
services
provided
with BHA
funding;
Output 5:
1350 highly
vulnerable
conflict-
affected
men and
women,
boys and
girls
received
WASH NFIs
assistance.
Sanitation:
1. Number of people directly
utilizing improved sanitation
services provided with BHA
funding (BHA)
2. Average number of users
per functioning toilet (BHA)
3. Percent of
latrines/defecation sites in the
target population with
handwashing facilities that are
functional and in use (BHA)
WASH Non-food items:
1. Total number of people
receiving WASH NFIs
assistance through all
modalities (without double
counting) (BHA).
2. Number of WASH NFIs kits
provided to collective centres
for IDPs/social institutions
3. Percent of households
reporting satisfaction with the
contents of the WASH NFIs
received through direct
distribution (i.e. kits) or
vouchers (BHA)
4.Percent of households
reporting satisfaction with the
quality of WASH NFIs received
through direct distribution (i.e.
kits), vouchers, or cash (BHA)
Sanitation: Review of
documents and
records of selected
institutions/KII with
managers of the
institutions.
WASH Non-food
items: 1. Regular
reporting 2. Survey
(Structured interviews
with beneficiaries)
18
national
standards
Objectiv
e 4: To
enhance
ability of
conflict-
affected
older
populati
on to
meet
their
basic
food
needs
Result 5:
5400 highly
vulnerable
conflict-
affected
men and
women have
enhanced
the ability to
meet their
basic food
needs
Output 6:
5400 highly
vulnerable
conflict-
affected
men and
women
received
food security
assistance.
1.Number of beneficiaries
receiving food assistance
2. Percent of households with
poor, borderline, and
acceptable Food Consumption
Score (FCS) (BHA).
3. Percent of households
reporting satisfaction with the
contents of the food kits
received through direct
distribution (CUSTOM)
4. Percent of households
reporting satisfaction with the
quality of food kits received
through direct distribution
1. Regular reporting 2.
Survey (Structured
interviews with
beneficiaries)
19
(CUSTOM)
5. Mean and median Reduced
Coping Strategy Index (rCSI)
score (BHA)
6. Percent of beneficiaries
reporting that humanitarian
assistance is delivered in a
safe, accessible, accountable,
and participatory manner
(CUSTOM)
Objectiv
e 5: To
increase
access
of
conflict-
affected
men and
women
to NFIs
Result 6:
1200 highly
vulnerable
conflict-
affected
men and
women have
increased
access to
NFIs
Output 7:
1200 highly
vulnerable
conflict-
affected
men and
women
received NFI
kits
1.Number and percent of
households receiving NFIs in
identified settlement(s) through
use of in-kind NFIs (BHA S&S
NFI)
2. Number and percent of
individuals reporting
satisfaction with the quality of
the NFIs received (BHA - S&S
NFI)
3. Number of NFI kits
provided to the collective
centres (CUSTOM)
1. Regular reporting 2.
Survey (Structured
interviews with
beneficiaries)
Assumptions:
- The large-scale military offensive does not escalate in proposed project sites and does not result
in interruption or suspension of project activities.
- Political situation enables to conduct project activities and reach the beneficiaries in selected
project locations.
- There is a sufficient number of recruited, trained and qualified staff to provide high quality
assistance relevant to the needs of the affected people
- Suppliers and/or local workers are reliable and transparent in the areas of project
implementation, including compliance with policies, law and procedures.
- Beneficiaries are not exposed to harm, abuse and/or violence during participation in project
activities as unintended negative effect of the action.
- Safety of beneficiaries.
20
- Prices and currency exchange rates will remain stable as foreseen during the period of project
implementation.
- Stability of the market and access of beneficiaries to market to meet their basic needs.
- The most vulnerable people are targeted by the project.
- Contractors/vendors/ suppliers responsible for provision of services are able to provide them to
beneficiaries according to the requirements.
21
Annex 2. Deliverables and Milestones, Schedule
Milestones, deliverables Date of
completion
Milestone 1
Milestone 2
Milestone 3
22
Annex 3. Selection of Stakeholders for in-depth interviews
Criteria for selection Indicative
number of indepth
interviews
Region
Dnipro 4
Kharkiv 3
Kyiv 3
Type of community
urban/settlement 5
rural 5
Type of activity
PSS 2
WASH 2
MCVA 3
Food 3
Preliminary List of Locations for Data Collection:
Community name Community type Region Target Group Number of
Interviews
23
Annex 4. Evaluation Matrix
The matrix below presents the evaluation questions from the ToR, judgement criteria, preliminary indicators, sources and methods and tools for
data collection. We will use only the indicators that appear most relevant during the data collection and analysis phase.
Assumptions to be
assessed/ Judgement
criteria
Preliminary indicators (which data to be collected) Sources of verification Methods and tools
EVALUATION QUESTION: To what extent were the outcomes and objectives achieved / are likely to be achieved?
EFFECTIVENESS
The project has
achieved the expected
results (outputs and
outcomes) under its
Principal Objective, as
well as Objectives for
separate activities.
● Evidence on the achieved project’s outputs and
outcomes
● Evidence on the effectiveness of the specific project’s
activities
● Perceptions/opinions on which activities were most
effective and why
● If there are any gaps remaining, in the achievement of
results (outputs/outcomes)
● Achievement of which results is expected in the nearest
future
● Unexpected results achieved
Project documentation – grant
proposal narrative, MEL plan,
MEL and performance reports,
PIT.
Insights and/or perceptions from
respondents – PAH
representatives, stakeholders,
beneficiaries.
Desk review
Key informants’
interviews
In-depth interviews
Focus group
discussions
Beneficiaries’ survey
EVALUATION QUESTION: What were the major factors influencing the achievement or non-achievement of the objectives?
EFFECTIVENESS
Factors contributing to /
undermining the
achievement of results
(output and outcomes)
of the project have
been recognized and
addressed where
possible.
● Evidence on the factors facilitating or impeding
achievement of the project’s outputs and outcomes
● Evidence that these factors have been recognized and
addressed where possible
● Perceptions/opinions on which factors are the most
critical and how to mitigate those undermining the
project’s effectiveness and employ facilitating factors
● Perceptions/opinions regarding interaction with PAH
representatives
Project documentation – grant
proposal narrative, MEL plan,
MEL and performance reports,
PIT.
Insights and/or perceptions from
respondents – PAH
representatives, stakeholders,
beneficiaries.
Desk review
Key informants’
interviews
In-depth interviews
Focus group
discussions
Beneficiaries’ survey
EVALUATION QUESTION: To which extent have the activity objectives, design and results addressed beneficiaries’ needs and priorities,
including across different population groups?
24
RELEVANCE
Project’s activities are
relevant to the
beneficiaries’ needs
and are implemented in
a relevant way.
Activities are adopted to
needs changing as a
result of changing
situation.
● Evidence confirming that project’s activities are relevant
to the beneficiaries’ needs and are implemented in a
relevant way (transparent and timely delivery,
consideration of security aspects)
● Evidence and Insights/opinions that activities are
adopted to changing needs of beneficiaries
● Insights/opinions confirming that project’s activities are
relevant to the beneficiaries’ needs and are implemented
in a relevant way
Project documentation grant
proposal narrative, MEL reports
and performance reports.
Insights and/or perceptions from
respondents – PAH
representatives, stakeholders,
beneficiaries.
Desk review
Key informants’
interviews
In-depth interviews
Focus group
discussions
Beneficiaries’ survey
25
Annex 5. Proposed Data Collection Tools / Protocols
26
Annex 5. Final Performance Evaluation. Terms of Reference