Within the framework of increasing access to quality and equitable health care and protection services for vulnerable Syrian refugees and host communities in Jordan, the UK’s Foreign, Commonwealth & Development Office (FCDO) funded a three-year project implemented by International Medical Corps (IMC) in partnership with Jordanian Hashemite Fund for Human Development (JOHUD). The multi-sectoral project ‘Specialized Protection Intervention for Vulnerable Refugees and Host Communities in Jordan’ was implemented between 2018 and2021 in six districts in Jordan: Mafraq, Irbid, Balqa, Ma’an, Jerash, and the Jordan Valley, in close collaboration with the Ministry of Health (MoH) and other governmental departments. The project aimed to improve access to quality services through the provision of mental health and psychosocial support (MHPSS), child protection (CP) and primary health care (PHC) with complementary services (nutrition and vaccination) for Syrian refugees and host communities.

This report presents the findings from an independent performance evaluation conducted between June and September 2021 by an external consultant. The evaluation aimed to analyse the final results of the project in the targeted areas, as well as to assess the efficiency, effectiveness, impact, and sustainability of the project after the implementation through the usage of quantitative and qualitative data.

The evaluation methodology included both primary and secondary data sources. The evaluation criteria used are based on the updated OECD-DAC standards, covering effectiveness, efficiency, impact, and sustainability^[1]. Additionally, specific humanitarian response standards, gender equality, and evaluation criteria which address coverage, coordination, quality and protection have also been used. The evaluation process included a desk review of project-related documents, a beneficiary-based survey, in-depth interviews and focus-group discussions with beneficiaries, and key informant interviews. Primary data collection was conducted in the six target locations where the project activities were implemented.

Quantitative data was captured during surveys with 1,026 beneficiaries of the project, while qualitative data was captured through individual interviews with 12 Key Informant Interviews (KIIs) and 8 focus group discussions (FGDs). Key informants (KIs) consisted of IMC project staff, JOHUD’s management team, and officials from the Mo H. The sample included 327 MHPSS beneficiaries, 424 CP beneficiaries, and 275 PHC – (JOHUD beneficiaries).

A desk review of project related reports revealed that project targets were significantly exceeded for the health, MHPSS and protection components, showing an increase in the beneficiaries' knowledge and skills related to protection and health concerns and issues, as well as an increase in the knowledge and skills of IMC staff and its partner and stakeholders, such as JOHUD and MoH. The project target indicators were assessed through the review of the project management tools (PMTs) and Log frame; the assessment showed that all the targets were achieved. Specifically, the project exceeded all targets for results in improved wellbeing, reduced protection needs, and strengthened the resilience of the most vulnerable refugees and host communities residing in Mafraq, Irbid, Balqa, Ma’an Jerash, Jordan Valley through child protection, health, and MHPSS interventions.

The key findings of the evaluation can be summarized as follows:

• The FCDO project met and exceeded the targets initially set in the logical framework.
• IMC Jordan’s holistic model of project design, management, and implementation created a strong network of specialized staff to work within the project. This contributed to the overall efficiency. There is evidence of quality activities being delivered on time, with efficient resource (information, human resources, financial, time) management that enabled achievement of planned outcomes.
• IMC, jointly with JOHUD, efficiently covered the hard-to-reach areas and delivered MHPSS, CP and PHC interventions and filled the most pronounced services accessibility gap for target population living in those areas.
• The project played a key role in linking the policy with the population needs and addressed the capacity gaps at the national level by training key health workforce cadre at MoH. IMC also successfully improved the capacities of the project staff as well as enhanced the program delivery and management capacities of its local partner JOHUD.
• These capacity building component of the project proved to be significantly contributing towards sustainable impact of the project but at the same time evaluation finds the continuous efforts are required to both reflect GoJ’s health policy as well as capacity to ensure integration of mental health at the PHC centres.
• Despite significant progress in increasing the capacity of local stakeholders, knowledge and funding gaps persist that might undermine the progress made in improving health, MHPSS and protection services for vulnerable communities. For example, the cessation of Mobile Medical Units (MMUs) in hard-to-reach areas targeted under this project will result in coverage gaps if no other partner or the MoH is able to sustain operations due to operational and budgetary constraints.
• IMC developed and implemented a capacity building plan for JOHUD as a local partner to ensure they are well trained on mental health (MH) and CP case management systems, minimum standards and protocols. IMC has also supported JOHUD in developing policies, Human Resources (HR), Best Interest Assessment (BIA), and built the capacity of the technical staff in CP, MHPSS and Health topics.

Mental Health and Psychosocial Support (MHPSS)

• Project efficiently responded to the needs of MHPSS services across the locations. The MH consultations covered psychiatric consultations, case management sessions, psychotherapy and psychoeducation sessions and home visits. More than half (52.6%) of the beneficiaries interviewed received MH consultations; while more than a quarter (27.5%) reported having received both MH and PSS, 17.1% received PSS only, and 2.8% reported other services.
• The project was effective in ensuring that MHPSS services are equally available across the locations both for men and women. In this regard, around 57% of the beneficiaries were women. these services were received by people of all ages. Around one-third of project beneficiaries (34.4%) receiving MHPSS services were below the age of 18 years, more than a quarter (27.8%) were between the ages of 19-35 years, 21.4% were between the ages of 36-49 years, and 16.5% were above 50 years of age.
• The project was successful in addressing the stigma related to accessing the MHPSS services (as reported by FGDs participants) – partly working closely with the communities, ensuring a high level of privacy, and integrating MH services with PHC centres of MoH. Most of the MHPSS beneficiaries continued to receive services until their care plan was achieved.
• Within the satisfaction survey carried out under the evaluation, 75.1% (74.8% males and 75.5% females) of respondents reported receiving effective MHPSS services.
• Internal and external referrals were an integral part of IMC’s services provided under the FCDO project. The survey respondents rated the effectiveness of the internal referrals for psychoeducation, diagnosis/prescription of psychotropic medications by a psychiatrist, and psychotherapy by psychologist as 82.6%, 76.1%, and 71.3%, respectively. However, the effectiveness of the referrals made to other agencies was reported as less effective, where 60% of the survey respondents reported satisfaction. These responses show the level of satisfaction of services received by beneficiaries from other agencies. Participants in the FGDs also indicated that referral pathways established by IMC were highly effective in terms of their connectivity and responsiveness (both internal and external referrals).
• The project was very effective in adapting and addressing the cultural requirements of the beneficiaries (agreed by 91.7%), staff management and availability also contributed towards effectiveness (agreed by 87.4%), the impact of the case management services proved to be effective (agreed by 82.3%). On average, around 83% of beneficiaries agree that the MHPSS services and activities have been managed and implemented effectively.
• The project was found to be highly effective in addressing complex mental health needs or combination of mental health conditions including the cases who experienced sexual violence, anxiety, depression, and obsessive-compulsive disorder.
• The project helped MHPSS beneficiaries through resolving their mental health concern (52%), enabling them to understand their mental health conditions and psychological wellbeing, and 3% motivated to engage normally with family.
• The project has left a lasting impact by training MoH staff in terms of improving attitudes towards the importance of MHPSS and how to deal with the cases and concerns, raising awareness and identification MH signs/symptoms and now they could identify and refer people requiring MHPSS interventions.

Child Protection

• The majority of CP respondents (81.8%) were parents/caregivers who participated in the survey on behalf of the child; 12.5% were children aged 12 and above and 5.7% consisted of responses from both the parent and child. The majority, 86% of the CP beneficiaries reported CP CM as efficient enough to help them identify protection concerns and receive related services. Among all CP beneficiaries interviewed more than half (56%) of them reported services to be satisfactory – the results found desirability bias amongst CP beneficiaries who received cash assistance versus those who did not.
• More than half of CP beneficiaries (57.8%) reported that the case management care plan was efficient in ensuring the full participation of children in CP services, and in providing recommendations and communication with parents/caregivers.
• The cash assistance program was rated as the most efficient method in resolving child protection concerns, specifically for those engaged in child labour and exposed to multiple concerns, giving vulnerable families autonomy over how to meet their own needs.
• Parents and caregivers in the survey sample reported that children with challenges and specific concerns received effective referral services to child protection and more specialized services, when required.
• More than half (57%) of the beneficiaries reported the CP services as effective as they helped the children to improve their situation in terms of increased participation in community activities, as well as higher enrolment in school, which reduced child labour cases as well.
• CP services were found to be inclusive and accessible for children regardless of age, sex, and nationality.
• Participants reported considerable levels of positive effects because of the participatory processes and practices used by caseworkers who supported children’s participation in decisions concerning their protection concerns and personal
• The CP services provided the communities, parents/caregivers, and children, with more knowledge, awareness, and understanding of the psychological health of children.

Primary Healthcare (PHC) – Delivered by JOHUD

• The project intervention model was innovative and targeted the most vulnerable and excluded communities living in hard-to-reach areas – the project effectively identified and addressed vulnerabilities through the engagement of the local partner, JOHUD.
• Outreach with PHC services was confirmed by around two-thirds of the beneficiaries. Around two-thirds (61.5%) confirmed that JOHUD’s different services under PHC helped them in meeting their health needs.
• The quality of the services by JOHUD was reported by around 71% of beneficiaries as good or very good. Among those who accessed different services by JOHUD, 69% reported the services as efficient to address their health concerns.
• Around two-third, 64.7%, of the sample reported that the MMU was effective or very effective for the provision of complimentary primary health and allied services.
• As we see the coverage was around70% which appeared to be a challenge due to the roving nature of only one MMU that was covering a widespread population.
• PLWs and children were covered for nutritional screening as well as follow-ups for immunization that helped the GoJ to ensure children in hard-to-reach areas are effectively managed to receive essential immunization.
• The MMU modality proved to be highly effective in reaching out to populations living in remote and distant areas where small population groups were covered through roving dates.
• Around 74% of the PHC beneficiaries reported having received services that helped them to resolve their health concerns and the same percentage of beneficiaries reported that they are now more aware of health care issues related to mothers and children.
• JOHUD specifically highlighted that their collaboration with IMC helped them to develop capacities, as a local organization, to get prepared and respond to different vulnerable population needs.

Given the three components of the FCDO project (MHPSS, CP (including CA) and Health) as key interventions of IMC, the evaluation identified key lessons learned from findings. The following are the key lessons learned.

Mental Health and Psychosocial Support (MHPSS)

• The project was highly successful in extending and making available the MHPSS to a majority, (83.5%) of beneficiaries who have not faced any limitations or challenges in participating in MHPSS services and activities.
• The key lesson learned was related to the cost of transportation and the interruption of availability of medication, as highlighted by some beneficiaries.
• Mental health beneficiaries who reported to cover non-communicable diseases along with the MHPSS interventions to enable them to manage their mental conditions more effectively. Project many consider, as a lesson learned, to cover NCDs consultations for those MHPSS beneficiaries who are also suffering from NCDs.
• Due to project activities, MHPSS beneficiaries are reporting to overcome the social stigma associated with the receipt of specialist mental services, but at the same time, some beneficiaries are reporting social stigma while approaching the mental health clinics at PHC when other people from their communities are also visiting the PHCs. The project may consider it as a lesson learned to devise social advocacy component to work at the community level to change the attitudes and behaviour of community towards mental health and its access.

Child Protection

• CP beneficiaries recommend the increase in the number of child friendly spaces so they could be easily reachable to most people.
• Knowledge centres are needed to enable vulnerable children to access the internet and technology because online learning is important but their parents in many cases are unable to provide their children with access to the internet and iPads or smartphones. (Feedback from parents/caregivers of CP beneficiaries).
• A key informant from the project team indicated that domestic violence, weak economic situation and lack of recreation opportunities are the main challenges that expose children in target areas to risks.
• Cash assistance has achieved a short-term goal for most of the beneficiaries. However, there should be a system to monitor the long-term impact in terms of how cash assistance was used and whether it fulfilled the planned objectives of mitigating the risks that the child was exposed to While, usually other referrals are done along with the social protection package for the long term solutions as livelihood, longer term cash assistance, vocational training sessions etc.
• The vulnerability criteria and the procedures used for cash assistance were set up building upon the in-depth technical assessment which takes place for each individual after assessing the child elements and provide the necessary services, covering level of child protection risks, case complexity and severity as well as the longer and/or short-term impacts on the beneficiaries safety and well-being. In this process, the case managers refer the most vulnerable Syrian families to a committee who would determine the eligibility of the case and select those most vulnerable and are entitled for the cash assistance. It is recommended to organize, at the beginning of the project, a focus group discussion or community awareness session in each geographical area about the vulnerability assessment criteria and procedures. 

Primary Healthcare (PHC) – Delivered by JOHUD

• MMU was an innovative idea to reach vulnerable communities living in hard-to-reach areas, and it effectively provided access to primary health care and allied services but at the same time, evaluation learned that MMU, due to its being roving and only source, was having limitations to cover widespread population. The project may consider for future to increase the availability of MMU by increasing the frequency of visits and number of working MMUs in hard-to-reach areas.
• Evaluation learned that working with JOHUD contributed to the capacity building in terms of managing and operating MMU for hard to reach areas as JOUD specified that they plan to use the FCDO project design as a model in their future interventions.

Based on the evaluation findings and lessons learned, the evaluation team provided some recommendations for IMC summarized as follows:

1. Raise awareness about the community-based feedback and response mechanism for better sharing information with beneficiaries and to raise their awareness about it. Even though the project design included functional feedback and complaints system, the evaluation found few beneficiaries were aware of it.
2. Establish knowledge centres in the target locations to allow children access to learning services in addition to all available child protection activities, such as community centres.
3. Target a greater number of persons with disabilities in future interventions through the inclusion of children with disability to access and participate in the project activities, namely child protection activities, need to be deliberate.
4. Raise awareness about Vulnerability Assessment Criteria and procedures through organizing focus group discussions or community awareness sessions in each geographical area.
5. Encouraging MoH and local authorities to prioritize mental health  There should be a component of advocacy in similar future projects.
6. Maintaining the project’s services and activities is significant to sustain the benefits to the target populations. It is recommended to continue to provide the same services in terms of MHPSS, CP, and primary healthcare to the target population in order to fulfil the needs of the beneficiaries due to the lack of alternative similar services in these areas.
7. Restructuring and re-activation of the role of the advisory committees representing the community, including beneficiaries, civil society organizations, and governmental institutions to advise the project management regarding health and protection services, activities, and policies.

Below are some recommendations for donors summarized as follows:

1. Sustaining the efforts towards capacity building of MoH for self-reliance in provision and integration of MHPSS services at PHC level that is in line with the National Mental Health Policy and Action Plan, along with the Substance Use plan.
2. Building the capacities of the Community Based Organizations (CBOs) to provide awareness sessions, identify MH concerns, provide psychosocial support and safe
3. Integrating mental health and Non-Communicable Diseases (NCDs) in future interventions since the evaluation found that there is high demand for NCDs with some linkages between mental health and NCDs among Syrian refugee population, which poses a unique challenge to health system in Jordan for provision and continuity of care and access to services.
4. Giving more priority on funding the health care interventions that take into consideration the approaches aimed to helping the local authorities to develop capacities that enabling them to fulfil the health care needs of refugees and vulnerable communities in Jordan.
5. Ensuring easy access to services through mobile units or by covering the transportation costs. This feedback has been received from CP and MHPSS, and health beneficiaries who were living far from the service points.

[1] https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm