Job Description

Independent Evaluation Company services for ‘Girls Education Empowerment Project in Somalia’ (GEEPS) Endline Study

Summary

CARE Somalia is seeking to procure the services of an evaluation company / consultants (consultants) to conduct the endline study of Girls Education Empowerment Project in Somalia (GEEPS) funded by Global Affairs Canada. GEEPS has high demands in terms of the quantity and quality of data to be collected, in order to generate robust evidence to be used by multiple stakeholders.

The evaluation company / consultants will be responsible for the overall design of the endline study including the refinement of the methodology and tools as required, manage data collection services, carry out qualitative and quantitative data analysis, and prepare a comprehensive endline report for GEEPS. The external consultants will provide an independent and rigorous evaluation to enable the project to assess progress against outcomes at baseline, and to what extent the intervention was effective in contributing to gender transformative processes that contributed to the improved education (learning outcomes) and empowerment of girls, adolescent girls, and young women, including those with disabilities.

GEEPS overall evaluation require the use of mixed-methods, pre-post design, including longitudinal tracking of a in-school and out-school student cohort to establish project impact overtime.  The evaluator is expected to follow the same girls interviewed at baseline, as this greatly increases the power and robustness of statistical comparisons between two rounds of data collection. The evaluator is also expected put in place an efficient and systematic cohort tracking processes to ensure that enumerators follow up with girls and households interviewed at baseline. This requires the evaluator to develop a robust a recontact protocol .

Project Background 

CARE is a humanitarian non-governmental organization committed to working with poor women, men, boys, girls, communities, and institutions to have a significant impact on the underlying causes of poverty. CARE seeks to contribute to economic and social transformation, unleashing the power of the most vulnerable women and girls.  CARE has been providing emergency relief and lifesaving assistance to the Somali people since 1981. The main program activities since then have included projects in water and sanitation, sustainable pastoralist activities, civil society and media development, small-scale enterprise development, primary school education, teacher training, adult literacy and vocational training. CARE works in partnership with Somali and international aid agencies, civil society leaders and local authorities. 

The Girls Empowerment and Education Project in Somalia (GEEPS), or Hayanka Hablaha in Somali, is a three year initiative (activities beginning early 2021 through end of 2024) that seeks to address the systemic and pervasive gender inequalities that prevent adolescent girls in Somalia from the full and equitable enjoyment of their rights to a safe and inclusive education and learning in Jubaland, Puntland and Somaliland. GEEPS is a collaboration between CARE Somalia, the Federal Ministry of Education, Culture and Higher Education of Somalia (MOECHE) and State Ministries of Education (MOEs).

GEEPS supported  Ministries of Education in 30 urban locations in Jubaland and 40 primarily rural locations in both Puntland and Somaliland, to implement an integrated and innovative package of interventions that intend to 1) Increase equitable access to safe, secure, quality, inclusive education and learning by adolescent girls, including girls with disabilities (GwDs) and girls from traditionally marginalized groups and those heavily affected by conflict and displacement;  and 2) Enhance quality, gender responsiveness and inclusiveness of the Somali education system at all levels addressing the needs of marginalized girls, including those with disabilities, in service provision. GEEPS projects to directly reach 27,077 girls and 8,280 parents and further improve the lives of 63,062 girls and 15,972 parents already currently engaged in the UKAID-funded Girls Education Challenge projects implemented by CARE in Puntland, Jubaland and Somaliland.

GEEPS Theory of Change

Annex 1 outlines the project’s logic model, listing the intermediate and immediate outcomes, outputs and activities. GEEPS seeks to contribute to improved learning outcomes for girls and adolescent girls, including those with disabilities, through increased national / state system harmonization, quality standard-setting, gender equity, and gender mainstreaming in the education sector in fragile and conflict affected situations in Jubaland, Puntland and Somaliland. This project is underpinned by the theory that this can be accomplished if; 1) the gender, social, and economic related barriers that limit girls’ access to education and learning are addressed at the community, family and individual level; and 2) the education system in Somalia as a whole is more inclusive, gender responsive and able to provide quality education to those who are highly marginalized not only because of their sex and age but also because of their disability, minority, marital or educational status and /or pastoralist livelihood, in areas affected by conflict and crisis. The project posits that this ultimate change will be facilitated when barriers are reduced, systems strengthened, and adolescent girls are able to acquire more knowledge and skills in a supportive and inclusive environment.

Two intermediate outcomes guided the project’s interventions towards achieving the ultimate outcome.  The first intermediate outcome ensures that there increased equitable access to safe, quality and inclusive education by marginalized girls including girls with disabilities (GWDs) and those most affected by conflict, displacement and social discrimination. Improving community and family willingness, and social and economic capacity, to support gender-equitable and inclusive practices and address harmful social norms, while also supporting the capacity of individual adolescent girls to claim their rights, will contribute to the increased empowerment of adolescent girls by fostering a supportive enabling environment.

The second intermediate outcome focused on the enhancement of the gender-responsiveness and quality of the school system through a multi-pronged approach, working at multiple levels: Ministries of Education (MoEs) at state, regional and district levels; Community Education Committees, head teachers and classroom based teachers. This approach seeks to strengthen the capacity of the MoEs to identify and respond to girls’ needs and address gendered practices which may be discriminatory and exclusionary, improve access and retention, while also supporting capacity building of the actors on the ground to identify and address girls’ needs, including those of girls with disabilities and from displaced or minority groups.

Project Baseline

GEEPs baseline study was conducted over two rounds between May and October 2021, with the objective of providing a reference point to measure changes, milestone achievements, and impact resulting from project interventions. The baseline study was also used to inform revisions to project design and implementation modalities with an intention to better support the different needs of marginalized subgroups within the targeted communities.

The methodology employed a mixed methods approach, incorporating both quantitative and qualitative research methodologies to gather primary and secondary data. Due to the worsened COVID-19 situation in Somalia, the first round of data collection with in-school girls was conducted remotely using phone-based interviews, while the second round with former out-of-school girls was conducted in person.

The study utilized the Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) to assess students from Grade 1 to Grade 5 (both out-of-school and in-school cohort girls) and from Grade 6 to Grade 8 (benchmark girls) on their functional abilities in basic literacy and numeracy tasks. Adapted versions of EGRA and EGMA were used for remote data collection in the first phase of the baseline, while regular assessments were used for in-person data collection. In addition to these assessments, school surveys, teacher surveys, classroom observations, household surveys, and education officer surveys were also conducted. The quantitative survey results were complemented by qualitative interviews with parents, teachers, Community Education Committee (CEC) members, Women’s Rights Organizations (WROs), and vignettes from girls and teachers. This comprehensive approach aimed to gather a wide range of data sources to fully understand the baseline conditions and the potential impacts of the GEEPS project on the targeted population.

Purpose and Objectives of the Endline Study

The primary objective of the endline study is to compare progress against baseline and subsequent evaluation values for project indicators (as set out in the Performance Measurement Framework - PMF). Using findings from the baseline and subsequent evaluations, data from the endline will be used:

• To identify factors affecting expected outcomes and if/how those align against the intermediate outcomes and outputs selected by the project, thus testing its Theory of Change (ToC);
• To demonstrate accountability for the funding received to GAC, other Canada Government Departments, Canada taxpayers, Canada media.
• By partners, stakeholders, and the Government to learn lessons from the project for the purpose of informing education programming in country.
• By other donors, academic institutions and education networks to inform the wider policy debate concerning the education of girls and marginalised girls.

Endline Evaluation Questions

The evaluation will address the following evaluation questions:

1. Process – Was the project successfully designed and implemented?
2. Impact and Effectiveness- Learning

•  What impact did the project have on the learning of marginalised girls( as defined by the project)?
• Has the intervention been equally effective in improving learning outcomes across different sub-groups, particularly those identified as the most marginalized at the baseline?
• Are there particular skills for which no gains were realised? Why?
• What worked (and did not work) to increase the learning of marginalized girls?

2. Impact and Effectiveness- Retention

• What impact did the project have on the enrollment of marginalised girls, including girls with disabilities and other sub-groups?
• What worked (and did not work) to increase the learning of marginalized girls?

3. Sustainability – How sustainable were the activities funded by the GAC and was the project successful in leveraging additional interest and investment?
4. To what extent do evaluation findings support the Theory of Change? This includes but is not limited to the following questions:

• Increasing Access to safe, secure quality inclusive education

1. a)Is the project able to create equitable access to safe, secure quality inclusive education?
2. b)What are the key factors that contributed to improvements in access to safe, secure quality inclusive education and learning by girls, including those with disabilities and affected by displacement, in conflict and disaster-affected areas targeted by the project?

• Stakeholders Efforts to Ensure Girls' Education Access and Equality

1. c)To what extent did the enrollment drives (door-to-door campaigns , awareness campaigns , gender-sensitive media campaigns ) contribute to enhancing the knowledge and skills of stakeholders in addressing gender barriers to education and promoting girls' access to inclusive, quality education?

• Breaking Barriers thorough enhanced schools’ capacities

1. d)In what ways did the actions taken by Community Education Committees (CECs) to address social, cultural, and physical barriers for girls, including those with disabilities, enhance the overall inclusivity and accessibility of education in the project areas?
2. e)What specific impact did the girl-centered renovations in school infrastructure have on reducing physical barriers and creating a more conducive learning environment for adolescent girls, including those with disabilities?
3. f)How did the establishment of a crisis modifier enable early action and rapid response to new humanitarian needs, particularly in protecting the development gains of the project in enhancing the capacity of schools and vulnerable households to reduce barriers for girls' participation in education? with disabilities (GwDs), to participate in education?

• Amplification of Girls and Women Voices

1. g)To what extent did the mentorship through Girls' Empowerment Forums enhanced the leadership skills and advocacy abilities of girls aged 10-19 in addressing barriers to education and enrollment?
2. h)What were the perceived benefits and impact of the exchange visits between Girls' Empowerment Forums and female role models in empowering girls to assert their right to education and assume leadership roles?
3. i)How did the mentorship of boys on gender equity and positive masculinity in Boys' Empowerment Forums contribute to creating a more supportive and equitable environment for girls education?
4. j)What changes were observed in the attitudes and behaviors of girls, adolescent girls, and women in the targeted areas following their participation in the empowerment activities, and how did these changes impact their ability to make their voices heard in decision-making processes?

• MoE Integration efforts of ultra-marginalized adolescent girls into education systems

1. k)To what extent were education stakeholders, including MOE efforts, able to integrate adolescent girls, especially those from highly marginalized groups, into formal and non-formal education systems?
2. l)How effective was the training of MOE trainers and coaching of REOs, DEOs, and Gender Focal Points in addressing and supporting gender-responsive school management?
3. m)What impact did the support provided to MoEs for gender-responsive budget planning and establishment of resource centers for girls with disabilities have on promoting equitable treatment and inclusion of children with disabilities in education?
4. n)To what extent were inclusive education policies developed? And how have these policies addressed the equitable treatment of girls and children with disabilities in the education system?

• Creating Gender Responsive Learning Environment

1. o)To what extent has the training on gender-responsive school management and working with male teachers influenced the implementation of inclusive and safe environments within schools and communities?
2. p)How effective were the action plans developed by the CECs in addressing gaps in gender equity within the school system?
3. q)What measurable effects have been observed in the quality, inclusivity, and gender-sensitivity of curriculum delivery following the training and coaching of teachers?
4. r)In what ways have the GEF/BEFs influenced gender-sensitive, participatory school management?

• Enhanced monitoring and reflection systems and processes

1. s)How did the training of Regional and District Education Officers impacted on the collection and aggregation of data on Education Sector Strategic Plan indicators, and what changes resulted from this intervention?
2. t)What were the outcomes of the support provided to Ministries of Education Gender Focal Points in conducting bi-annual dialogues to assess progress on marginalized girls’ learning, participation, and completion rates?
3. To what extent has the project contributed to a broader process of shifting gender and social norms for adolescent girls, particularly older adolescents?
4. Are there any unintended consequences (positive or negative) of the intervention in terms of learning, transition, sustainability and protection?

Evaluation Approach and Methodology

The project will use a mixed-methods longitudinal design tracking a sample of cohort girls identified at the baseline across 55 sampling sites. The analysis should consider the exposure levels of sampled girls, using household survey data and monitoring data to generate exposure variables. The overall analysis of the samples will assess the efficiency of the project approach, while the analysis of the exposed sub-sample will assess the effectiveness of the project’s intervention to date.

The endline evaluation will utilise both quantitative and qualitative study methods to collect primary.

and secondary data. These include literature review, quantitative surveys, Key Informant

Interviews (KIIs) and Focus Group Discussions (FGDs).

Literature Review

A review of available documents relevant to the assignment will provide a clear understanding of the project. Some of the documents to be reviewed include the GEEPS Baseline Report, Monitoring, Evaluation, Accountability and Learning (MEAL) Plan, GEEPS Project Implementation Plan, Activity Plan, Project Implementation Reports, MOECHE Education Sector Strategic Plans, World Bank reports and UN reports, among others.

Quantitative Data Collection Tools

The project evaluation strategy will rely heavily on longitudinal surveys. These surveys are designed to gather and track essential output and outcome indicators for the project, this allows the project to measure the overall effectiveness of project interventions. The surveys will also assess the extent to which targeted girls and households have been exposed to project interventions and determine if the project reached its intended target groups and with what impact.

The study will undertake surveys in the categories outlined below using in-person data collection method. These surveys will be administered to both in-school and out-of-school girls. Teacher surveys, school surveys, and classroom observations will also be administered to provide insights into teaching practices and potential gender disparities. Stakeholder surveys will target education officials and the Community Education Committee (CECs), the stakeholder surveys will further enhance the understanding of the broader context. By utilizing these tools collectively, the project can quantify and assess Performance Measurement Framework (PMF) indicators at various levels of the hierarchy.

Experiences and Qualifications

Professional Skills and Qualifications

Qualifications: bidders are required to clearly identify and provide CVs for all those proposed in the Evaluation Team, clearly stating their roles and responsibilities for this baseline.  The consultants’ proposed evaluation team should include the technical expertise and practical experience required to deliver the scope of work and baseline evaluation outputs, in particular, with regards to:

• Study design: the team should include skills and expertise required to design, plan, and conduct mixed-methods impact evaluation, using longitudinal techniques.
• Skills in quantitative and qualitative data collection and analysis, drawing findings from multiple sources and handling potential contradictions between data sets; using justified approaches for analysis of qualitative data. 
• Relevant subject matter knowledge and experience: knowledge and experience required on conducting research with children, the education sector, disability, gender and gender equality to ensure that the evaluation design and research methods are as relevant and meaningful as possible given the aims and objectives of the project and the context in which it is being delivered;
• Evaluation management: manage a medium-scale and complex research process from end-to-end including baseline studies;
• Primary research: gender-sensitive design, management and implementation of primary quantitative and qualitative research in potentially challenging project environments, such as fragile and conflict affected states – this could include the design of longitudinal household panel surveys, advanced versions of EGRA /EGMA tests, in-depth interviews, focus groups, participatory qualitative exercises with children, etc.;
• Country experience: it is particularly important that the team has the appropriate country knowledge /experience and ability to interpret findings from a contextual perspective, as required to conduct the research;
• Information management: design and manage sex- and disability-disaggregated data and information systems capable of handling large datasets for MEL purposes, including cross-referencing different datasets;
• Statistical analysis: a range of statistical modelling and analysis of impact data; highly proficient user of: SPSS or STATA; and qualitative data analysis techniques.
• Data management and data cleaning.  Ability to supervise the collection, entry (if required), cleaning and management of large data sets.  Digital data collection processes are preferred.
• Safety considerations: Ensuring the whole evaluation process adheres to best practice for research with children including the implementation of child protection policy and procedures to ensure safety of participants. Note that all bidders are expected to be able to show that they have a child protection policy in place to safeguard children that the research team would come into contact with through the research activities.

The day–to–day project management of the endline will be the responsibility of the MEL Advisor, CARE USA as well as CARE Canada who will provide input on the tools; review training materials and proposed approaches to analysis; review emerging datasets; and provide input to the interpretation of findings.

Project milestones

Bidders are required to include in their detailed work plans the milestones set out below.  The project is under a tight timeframe to complete the study and report by 30th April 2024.

Skills Required

Attachment

Follow us on social media