Job Description

Terms of Reference

Endline Evaluation Consultancy Services for CARE Somalia’s Adolescent Girls’ Education in Somalia Project, funded by UK’s Girls’ Education Challenge and USAI

1. SUMMARY

CARE Somalia (henceforth referred to as “CARE”) is seeking to procure the services of a local or international evaluation company to manage the endline evaluation of Adolescent Girls’ Education in Somalia (AGES), an initiative funded jointly by the United Kingdom’s (UK) Foreign, Commonwealth and Development Office (FCDO) through its Girls’ Education Challenge- Leave No Girl Behind (GEC/ LNGB) initiative and USAID. The GEC is a global initiative focusing on ensuring the expansion of education opportunities for marginalised girls at primary and secondary level, using rigorous evaluation practices. The AGES project seeks to improve learning and transition outcomes for 80,475 severely marginalised girls in conflict-affected areas of Banaadir, Hirshabelle, Jubaland and South West states of Somalia. AGES supports formal, non-formal and accelerated education for some of the most affected out-of-school girls, including internally displaced populations (IDPs), girls with disabilities, minorities, married and divorced girls, adolescent mothers, and those in child labour. AGES has high demands in terms of the quantity and quality of data to be collected to generate robust evidence which can be used by multiple stakeholders.

This tender includes two components:

(1)     Data collection, including: Training of enumerators; data collection; quality assurance of fieldwork; data cleaning and collation; and regular reporting to CARE.

(2)     Evaluation design, data analysis and reporting, including: Modifications to baseline, 1st and 2nd midline quantitative tools; design of qualitative tools; design of the analysis framework; technical guidance to the data collection company on data collection methodologies, in case the two components are awarded to separate companies; data analysis; reporting; and preparation of dissemination products.

CARE will accept bids inclusive of both components or only one of them.

The endline evaluation will be an independent and rigorous study, which will enable the project to identify its effectiveness on achieving the expected outcomes and intermediate outcomes; assess to what extent the expected outcomes have  been achieved by different subgroups of the population (equity); identify the extent to which different components of the intervention  have contributed (or not) to the said outcomes and intermediate outcomes, testing the project’s Theory of Change (ToC); and assess delivery processes (efficiency and economy). The evaluation will ultimately assess if the AGES project investment represents Value for Money (VfM) for the UK, US and Somali governments, while also identifying valuable lessons learned for girls’ education programming in fragile and conflict-affected contexts.

The endline evaluation will use a pre-post design with a mixed methods approach. Quantitative data will be obtained from learning assessments conducted with three longitudinally tracked student cohorts (1, 4 and 5) representative of treatment interventions; a household survey conducted with students’ families; a school survey; head counts; and classroom observations. Quantitative results will be triangulated with qualitative data obtained through focus group discussions, interviews, and participatory exercises with girls. The endline evaluation will also include an analysis of contextual changes influencing outcomes, using secondary data collected by other development partners and monitoring data collected by the project during its implementation period. Findings from the contextual analysis will inform the interpretation of the endline results and the preparation of recommendations based on those for future programming and system strengthening.

2. BACKGROUND TO THE GEC PROGRAMME AND THE AGES PROJECT

2.1. GEC Programme Background

FCDO leads the UK’s work to end extreme poverty. FCDO is tackling the global challenges of our time including poverty and disease, mass migration, insecurity and conflict. FCDO’s work is building a safer, healthier, more prosperous world for people in developing countries and in the UK too.

FCDO is working to reach the Sustainable Development Goals (SDGs) by 2030. Progress on girls’ education is critical to the achievement of these targets. SDGs 4 and 5 specifically relate to education and achieving gender parity. SDG 4 specifically notes ‘inclusive and quality education for all and promote lifelong learning’.

USAID has joined efforts with FCDO’s Girls’ Education Challenge in its commitment to address barriers to the achievement of equitable education outcomes for all. In doing so, USAID contributes to the US Government’s Strategy on International Education, which commits American aid towards the achievement of SDG 4, expanding access to education for marginalised and vulnerable populations, such as children and youth affected by crises and conflict; improving learning outcomes and prepare girls for a productive future; and facilitating girls and women’s empowerment through gender-transformative educational opportunities. USAID’s partnership with FCDO’s GEC is aligned with the principles of its Education Policy, which prioritizes leveraging resources, promoting equity and inclusion, and focusing investments on improving learning and educational outcomes. In the specific case of AGES, it is expected that the programme will contribute to the expected outcomes of USAID Somalia’s Country Cooperation Development Strategy (CDCS), USAID’s Youth in Development Policy, USAID’s Vision for Action on Child Marriage, and USAID’s Vision and Approach on Localization.

Globally, 31 million primary age girls have never been to school[1]. The majority of these girls come from the poorest and most marginalised communities in the most disadvantaged locations, ethnic groups, etc.[2] Over the last 20 years, primary enrolments for girls have improved along with boys but completion rates are equally low for both sexes. At the secondary level, the differences between boys’ and girls’ participation rates really start to show. Significant disparities exist within countries, with the poorest girls from rural areas most severely subject to educational disadvantage – even at the primary level.[3] In the specific case of Somalia, such disparities are already observed in early primary grades, where the gross enrolment rate stands at 14%.[4] The GEC is helping the world’s poorest girls improve their lives through education and supporting better ways of getting girls in school and ensuring they receive a quality of education to transform their future.

PricewaterhouseCoopers LLP (PwC) and alliance partners have been contracted as the dedicated Fund Manager (FM) and is responsible for the day-to-day operation of the GEC. This includes establishing the recipient tendering process, supporting bidders, sifting and scoring proposals, monitoring Value for Money (VfM) and making project funding recommendations for FCDO approval. The FM also manages the relationships with the selected projects and oversees their Monitoring, Evaluation, and Learning operations.

Through the GEC, the UK government provided £355m between 2012 and 2017 to the FM to disburse to 37 individual projects across 18 countries across sub-Saharan Africa and South Asia to help girl’s education. In 2016 the GEC Transition window has been set up with additional UK funding to support the original GEC beneficiaries continue their journey through stages of education and further transit to upper grades and improve their learning[5]. In Somalia, the US government has contributed $14.4m to the GEC-funded AGES project to further expand its reach and impact.

CARE Somalia is implementing AGES, one of the GEC-funded projects. AGES’ intervention focuses on providing tailored, sustainable solutions to develop literacy, numeracy, and key life skills (financial literacy and knowledge of reproductive health) for 80,475 of the most marginalised girls and female youth in South Somalia, including girls with disabilities, those from minority groups, IDPs, married and divorced girls and adolescent mothers, and to boost positive transitions into higher education and dignified work, contributing to prevent child, early, and forced marriage and unions (CEFMU).

2.2.  Project Background

Operating in Somalia since 1981, CARE currently works through three main programs: the first, CARE’s Gender Equality & Education program, builds the capacity of government institutions to provide gender responsive education services while building adolescents and young people’s life skills and mobilizing families and communities to support girls’ and boy educational pathways. It also strengthens the capacity of feminist organisations and supports their collective advocacy and action. Our second program area, the Climate Justice, Food, Water, and Nutrition contributes to increased food security and an improved nutritional status of vulnerable population through increasing climate-smart agricultural and livestock production, promoting the organisation of smallholder farmers and pastoralists into savings groups and cooperatives and promoting their access and linkage with finance, value chains and business development services. It also increases the availability of water for household consumption and agricultural use, helps to improve children’s and pregnant and lactating mothers’ nutritional status, and supports community contingency planning and the strengthening of early warning systems. Thirdly, the Humanitarian Assistance program provides life-saving humanitarian assistance in Food, health, nutrition, WASH and education, promotes the use of information provided by early warning systems and promotes the leadership of women and local organisations in humanitarian response and decision-making.

Led by CARE in partnership with three local NGOs, AGES works with schools, communities, individual students, religious leaders, the Federal Ministry of Education, Culture and Higher Education (MOECHE) and State Ministries of Education (MoEs) in Hirshabelle, Jubaland and South-West, as well as the Banaadir Regional Administration to:

1. Increase the availability of flexible, quality learning opportunities tailored to the needs of ultra-marginalised out-of-school girls:

-          Accelerated basic education (ABE) including compressed primary content;

-          Increased access to quality formal education through agreements with education providers and teacher training (reading, numeracy, child protection and gender-equitable practices);

-          Non-formal education (NFE/life skills course) for older girls (ages 15-25), teaching literacy, numeracy, financial literacy, savings, and basic business planning.

2. Facilitate a social norm change movement towards broader life opportunities for adolescent girls and boys:

-          Religious leaders trained on Islamic principles to support gender equity and education for girls;

-          Community leaders and Community Education Committees (CECs) sensitised and mobilised to work with parents to identify and support extremely marginalised girls to enrol in learning opportunities and attend classes regularly;

-          Mothers and mothers-in-law mobilised through village savings and loans associations (VSLAs) and enrolment drives;

-          Girls trained on leadership skills, menstrual hygiene management, savings and forming peer support networks at Girls’ Empowerment Forums (GEFs);

-          Boys mentored through Boys’ Empowerment Forums (BEFs);

-          Messages reinforced via radio and social media;

-          Linkages between female youth enrolled in NFE with youth VSLAs, financial institutions, and apprenticeship opportunities, shifting perceptions and options for livelihoods.

3. Build institutional capacity: Support to strengthen policy frameworks on NFE and their implementation; improving monitoring of education services; develop decentralised management capacity; strengthen mainstreaming of inclusive approaches and safeguarding mechanisms; and increase technical capacity for planning and management of non-formal education services.
4. Facilitate the engagement of girls and female youth in collective action to claim rights through support to interventions led by GEFs and GEF networks.
5. Support increased access of female youth to other development opportunities, including employability skills through technical-vocational education and training (TVET); economic empowerment opportunities; and participatory accountability processes.
6. Increase girls and female youth’s access to critical services, such as legal services, governance institutions and safety nets through humanitarian support.

AGES is also providing the means for girls who are unable to attend school consistently to study remotely, through guided lessons and remote support; mobilising communities to follow up on dropouts and ensure their return to school; providing hygiene supplies to participants and schools; and supporting the implementation of psychosocial first aid through mentors, teachers, and peers.

The use of flexible learning tracks by the AGES project, complemented by remote education, allows girls engaged in labour, married and divorced girls, and pastoralist girls to attend learning sessions despite the limitations on their time. The work with religious leaders and Quranic teachers addresses the resistance to the inclusion of older and married girls and issues with their mobility and approval from family to attend classes, particularly in areas where recent influence from armed groups has resulted in severe curtailing of women’s freedom.

Physically disabled girls with limited mobility benefit from (1) access to ABE/ non-formal education courses close to home; (2) renovations to schools which include accessible features such as ramps; (3) access to remote learning allowing them to learn from home; (4) increased awareness of parents and community members of the importance of education for the disabled. Girls with special learning needs are benefitting from scholarships to attend special needs schools. Girls facing severe anxiety and depression are supported through the combination of psychosocial first aid and support networks through the GEFs.

Girls from minority clans and ethnic groups develop their self-confidence and aspirations and build connections beyond their own clans through participation in the GEFs. The participation in GEF networks have expanded the engagement of the most marginalised adolescent girls and female youth in civic action. Furthermore, teachers have been trained to support girls who are lagging behind and first-generation learners, enhancing the likelihood of enrolment and retention in ABE, NFE and formal education. Girls from displaced and ultra-poor families as well as orphaned children are overcoming financial barriers through the participation of parents, older adolescent girls, and female youth in VSL. Through the expansion of AGES activities funded by USAID, female youth also have the opportunity to further expand their economic empowerment through VSL networks and participation in skills training, as well as enhanced access to safety nets.

See Annex 1 for an overview of the project’s outcomes, intermediate outcomes, outputs, and associated indicators as well as the key activities under each output.

2.3. Overview of the project implementation timescales

Project start-date:        September 7, 2018

Project end-date:         August 31, 2024

Experiences and Qualifications

Qualifications: the selected evaluation company is required to clearly identify and provide CVs for all those proposed in the Evaluation Team, clearly stating their roles and responsibilities for this study. Bidders are required to provide CVs by component, i.e. (1) data collection and (2) evaluation design, analysis and reporting. In case the same team is proposed for both components, bidders should indicate that this is the case.

The proposed evaluation team should include the technical expertise and practical experience required to deliver the scope of work, with regards to:

Component 1: Data collection

• Data collection planning: The team should include skills and expertise required to design, plan and conduct large scale quantitative and qualitative data collection in fragile contexts;
• Relevant subject matter knowledge and experience: Knowledge and experience required on conducting research with children, the education sector, disability and gender to ensure that the research methods and approaches used in the field are as relevant and meaningful as possible given the aims and objectives of the project and the context in which it is being delivered;
• Data collection management: Manage a large-scale and complex research process from end-to-end, including remote management of teams;
• Primary research: Gender-sensitive implementation of primary quantitative and qualitative research in potentially challenging project environments, such as fragile and conflict affected states – this could include the design of longitudinal household panel surveys, advanced versions of EGRA /EGMA tests, in-depth interviews, focus groups, participatory qualitative exercises with children, etc.;
• Country experience: It is particularly important that the team has the appropriate country knowledge /experience required to conduct the research;
• Ability to operate in project regions: The bidder should be able to mobilise experienced enumerators able to collect data in targeted areas and knowledgeable about local dynamics;
• Experience in electronic data collection and phone-based, virtual data collection;
• Quality assurance, data management and data cleaning:  Ability to supervise the collection, entry (if required), cleaning and management of large datasets. This includes backtranslation of surveys; field supervision of teams and remote supervision/ checks whenever applicable; systematic daily checks of incoming electronic data during data collection; database management; and transcription, translation and verification of qualitative datasets; and
• Safety considerations: Ensuring the whole study process adheres to best practice for research with children, including the implementation of CARE’s child protection policy and procedures to ensure safety of participants. Note that the selected bidder is expected to be able to show that they have a child protection policy in place to safeguard children that the research team would be in direct contact with through the research activities.                  

Component 2: Evaluation design, data analysis and reporting

• Study design: The team should include skills and expertise required to design, plan and conduct rigorous, mixed-methods impact evaluations in fragile contexts;
• Skills in quantitative and qualitative data collection and analysis, drawing findings from multiple sources and handling potential contradictions between datasets;
• Relevant subject matter knowledge and experience: Knowledge and experience required on conducting research with children and adolescents, the education sector, disability and gender to ensure that the evaluation design and research methods are as relevant and meaningful as possible given the aims and objectives of the project and the context in which it is being delivered;
• Evaluation management: Manage a large-scale and complex research process from end-to-end;
• Primary research: Gender-sensitive design, management and implementation of primary quantitative and qualitative research in potentially challenging project environments, such as fragile and conflict-affected states – this could include the design of longitudinal household panel surveys, adapted versions of EGRA /EGMA tests, in-depth interviews, focus groups, participatory qualitative exercises with children/ adolescents, etc.;
• Country experience: It is particularly important that the team has the appropriate country knowledge /experience and ability to interpret findings from a contextual perspective, as required to conduct the research;
• Information management: design and manage sex- and disability-disaggregated data and information systems capable of handling large datasets for MEL purposes, including cross-referencing different datasets;
• Statistical analysis: a range of statistical modelling and analysis of impact data; highly proficient user of SPSS or STATA; and qualitative data analysis techniques, preferably considering emerging codes;
• Data management and data cleaning: Ability to supervise the collection, entry (if required), cleaning and management of large datasets. In case the two Components are awarded to different bidders, the company selected for Component 2 will provide technical guidelines to the company contracted to deliver Component 1 on data management and cleaning.
• VfM assessment of education projects: Education economics expertise to conduct cost-benefit and cost-effectiveness analyses as part of the assessment of the project’s VfM; and
• Safety considerations: Ensuring the whole study adheres to best practice for research with children including the implementation of child protection policy and procedures to ensure safety of participants. Note that all bidders are expected to be able to show that they have a child protection policy in place; although Component 2 does not include direct contact with children, a child protection policy is essential to ensure an appropriate approach to data analysis and reporting.

The day–to–day project management of this study will be under the responsibility of Paul Odhiambo, M&E and Knowledge Manager, AGES project/ CARE Somalia.

4.2. Deliverables and Schedule

Expected Tasks

Component 1: Data collection

1. Develop a detailed work plan for data collection, data entry (where relevant – it is anticipated that most of the data will be collected electronically), data cleaning, transcription, and translation.
2. Translate data collection tools and training protocols from English to Somali. Note that back-translation will be required for accuracy. The selected bidder is also expected to be able to use translations in Af-Maay where applicable, allowing students to respond in their mother tongue to the girl survey.
3. Upload all relevant data collection tools into an electronic format.
4. Lead the training of enumerators and team leaders, technically supported by the evaluation team (Component 2) and CARE. The data collection company will follow the guidelines and the training plan created by the evaluation team leading on Component 2; and will implement quality control protocols and guidelines for enumerators previously reviewed and approved by the external evaluation team and CARE’s M&E team.
5. Prepare all training and data collection tools and related supplies for enumerators, including mobile phones/ tablets; printouts of assessments and reference materials.
6. Conduct all data collection in the field for each tool described above, including testing of tools and actual data collection.
7. Support in data collection related logistics, including but not limited to deployment of enumerators and supervisors to the target areas; and where required coordination with CARE to access each area with approval from local authorities.
8. Upload datasets completed by the team on a daily basis for verification.
9. Create electronic data entry forms (databases) as necessary in case of data entry.
10. Qualitative data will be completely transcribed in Somali and translated into English. Spot-checking back-translation will be required for qualitative data.
11. Conduct data entry, where necessary, simultaneous to data collection to prevent data delays.
12. Conduct all necessary data quality and ethical control measures.
13. Provide a complete set of physical and electronic data collection documents, filed, and organized as per guidelines provided by the external evaluation company, upon completion of data collection and entry. This includes but is not limited to filled surveys, questionnaires, FGD/ interview notes, daily survey logs, voice files (for electronic qualitative data collection), pictures, etc. Please note that CARE’s guidelines do not authorize pictures of respondents due to the need for confidentiality.

Component 2: Evaluation design, data analysis and reporting

1. Review the project’s MEL framework, other relevant project documents and external literature related to the Somali context and in particular, the education sector in country;
2. Submit an inception report (draft and final), that outlines the research methodology, including data collection tools, detailed analysis framework and detailed work plan outlining all tasks to be completed by each of the members of the study team;
3. Finalization of tools, including adaptations post-piloting and back-translation supervision;
4. Technical support for enumerator training content and toolkits (qualitative and quantitative) and supervise/monitor training on data collection tools and their application;
5. Follow up on and directly supervise enumerators’ work in the field during data collection (qualitative and quantitative) and provide the necessary technical support to the data collection team to ensure quality of data, including overnight checks of electronic datasets;
6. Validate all datasets, working in coordination with the data collection company, and collate data as necessary for analysis;
7. Analyse the collected data in accordance with the agreed analysis framework.
8. Prepare a comprehensive draft and final study report, according to the format provided by the Fund Manager, and inclusive of the GEC Outcomes Spreadsheet and logframe (with all disaggregates).

Skills Required

Follow us on social media