Call for Proposals - Final Evaluation of the Action “Community Action for Full Inclusion in Education”, Armenian Caritas BNGO

BACKGROUND

The action to be evaluated has the following Overall Objective (OO) and Specific Objectives (SO):

OO: Children with disabilities who are currently out of education enjoy their human rights for inclusive education, as specified in Article 24 of the UNCRPD ( UN Convention on the Rights of people with Disabilities)

SO1: Ensure that children with disabilities who are currently out of education in the region of Shirak receive quality education in their neighbourhood mainstream schools

SO2: Strengthen community capacity to monitor and enforce enrolment of disabled children to school and report on cases of its violation

The target groups include:

(1) An estimated 200 children with disabilities in school age who are out of education in Shirak region

(2) 12 schools (6 urban, 6 rural) in Shirak region (7 communities: 1 urban and 6 rural) with a particular focus on school authority, teachers (350) and all children (3150)

(3) The parents of children with disabilities who do not attend school in Shirak region (estimated 300)

(4) Regional Department of Education in Shirak region

The main activities are:

Activity 1.1. Selection of 12 Schools

Activity 1.2. Conduct a Mapping survey to identify children out of school in the region of Shirak.

Activity 1.3. Establish a Case file for every disabled child out of education.  

Activity 1.4. Aggregate the case files into a Regional roster (with all case files integrated).

Activity 1.5. Translate into Armenian the International Index for Inclusion tool.

Activity 1.6. Provide training to 12 school administration and teachers in Shirak on the Index for inclusion and support schools and their communities to introduce it.

Activity 1.7. Provide counselling to the families of disabled children out of school.

Activity 1.8. Conduct public awareness campaign on the need of inclusive education for all.  

Activity 2.1. Develop a monitoring system to ensure ongoing collection of information about children with disabilities out of education and their school enrolment status.

Activity 2.2. Produce bi annual reports highlighting cases of violation and submit them to the relevant authorities for resolution

For more information please refer to the logframe of this Action and to the project documents that will be made available upon assignment.

DESCRIPTION OF THE EVALUATION ASSIGNMENT

Objectives of the evaluation 

Systematic and timely evaluation of its programmes and activities is an established priority of the European Commission. The focus of evaluations is on the assessment of achievements, the quality and the results of Actions in the context of an evolving cooperation policy with an increasing emphasis on result-oriented approaches and the contribution towards the implementation of the SDGs.

From this perspective, evaluations should look for evidence of why, whether or how these results are linked to the EU intervention and seek to identify the factors driving or hindering progress.

Evaluations should provide an understanding of the cause and effect links between: inputs and activities, and outputs, outcomes and impacts. Evaluations should serve accountability, decision making, learning and management purposes.

The main objectives of this evaluation are to provide the relevant services of the European Union and the implementing partners with:

• an overall independent assessment of the past performance of the Action to be evaluated, paying particular attention to its results measured against its expected objectives; and the reasons underpinning such results;

• key lessons learned, conclusions and related recommendations in order to improve future Actions.

In particular, this evaluation will serve to further develop the supporting actions in the field of inclusive education.  Therefore, the evaluation shall focus on the learning aspects in order to be a good basis for shaping the follow up activities in the respective field.

The main users of this evaluation will be the EU Delegation to Armenia, the Austrian Development Agency, Caritas Austria, Armenian Caritas and the National Association of School Parents Council Members in Armenia; in other words those that are involved in the implementation and the financing of the Action to be evaluated.

Requested services

Scope of the evaluation

The evaluation will assess the Action using the five standard DAC evaluation criteria, namely: relevance, effectiveness, efficiency, sustainability and the perspectives of the impact. In addition, the evaluation will assess two EU specific evaluation criteria:

the EU added value (the extent to which the Action brings additional benefits to what would have resulted from Member States' interventions only);

the coherence of the Action itself, with the EU strategy in Armenia and with other EU policies and Member State Actions.

The evaluation team / evaluator shall furthermore consider whether poverty reduction, RBA, and gender were mainstreamed; the relevant SDGs and their interlinkages were identified; the principle of Leave No-One Behind and the rights-based approach methodology was followed in the identification/formulation documents and the extent to which they have been reflected in the implementation of the Action, its governance and monitoring.

The specific Evaluation Questions as formulated below are indicative. Based on the latter and following initial consultations and document analysis, the evaluation team/evaluator will discuss them with the Evaluation Manager and propose in their Inception Report a complete and finalised set of Evaluation Questions with indication of specific Judgement Criteria and Indicators, as well as the relevant data collection sources and tools.

Once agreed through the approval of the Inception Report, the Evaluation Questions will become contractually binding.
Specific Evaluation Questions

Relevance

• To what extent are the objectives of the Action still valid for the partner country, the partner organization and the beneficiaries?

• Are the expected outputs of the Action consistent with the outcomes and the impact (as part of the analysis of the logframe matrix/programme theory and the presentation of the theory of change and its underlying assumptions)?

• What is the relevant support parents, children, teachers and other stakeholders wish for the future?

Effectiveness

• What were the major factors influencing the achievement or non-achievement of the outcomes/outputs? Also consider any which were possibly beyond the control of the Action.

• To what extent have all Action’s stakeholders collaborated as planned?

Efficiency

• Was the Action implemented in the most efficient way (time, personnel resources)?

(Perspectives of the) Impact

• What has already changed in the lives of the girls and boys with disabilities (disaggregated data on gender, as well as on the number of girls and boys from poor and extremely poor families)?

• To what extent became the cultures and practices in target schools more inclusive?

Sustainability

• What did the implementation organisations or any actor involved in the Action learned from the Action and how has this learning been utilised and disseminated?

• What are the changes in the attitude of different involved stakeholders/beneficiaries to inclusive education?

Phases of the evaluation and required outputs

The evaluation process will be carried out in three phases:

• Inception & Desk

• Field

• Synthesis & Reporting

The main content of each output is described in Chapter 5.

The different phases of the evaluation are explained in more detail in Annex I.

EXPERTISE REQUIRED

Key Qualifications should be:

• Relevant academic degree (master level) in social sciences and/or education  

• A minimum of three years’ experience and expertise in the field/sector of education, most ideally of inclusive education

• Conducted at least three evaluations in the last five years, ideally in the relevant field

• Knowledge of Armenia with focus on topics such as (inclusive) education

• Experience in project cycle management

• Experience in project level evaluations

• Familiarity with donor funded projects, preferably with EU

• Experience preparing and analysing a theory of change

• Experience in social science methods

• Excellent oral and written Armenian & English skills

• Sound MS Office and IT skills

The consultants must not have been involved in the design, implementation or monitoring of this project.

DURATION / TIMETABLE

REPORTING

The outputs must match quality standards. The text of the reports should be illustrated, as appropriate, with maps, graphs and tables; a map of the area of Action is required (to be attached as Annex).

Language

All reports shall be submitted in English.
The final report, the final executive summary and the slide presentation shall in addition also be submitted in Armenian.

Formatting of report
All reports will be produced using Font Arial or Times New Roman (letter size 11 and 12 respectively), single spacing, double sided. They will be sent in Word and PDF formats.

Procedure and Logistics
The Evaluator covers all travel expenses within Armenia; as well as all expenses related to printing, copying, data collection and other support services. The Evaluator is solely responsible for the quality of the work to Armenian Caritas.

Armenian Caritas reserves the right not to pay the Contractor or withhold part of the payable amount if one/more requirement(s) established for this assignment are not met or if the deadline set for the accomplishment of the tasks is missed.

TERMS OF APPLICATION

The deadline for the submission of the application is the  15 th  of November 2018. Applications should include:

• CV

• Cover Letter: max 300 words, should explain why you think you are qualified for this post and also indicate when you can start to work.

• Brief Concept Note not exceeding 3 pages describing the approach and suggestions for the evaluation.

• Proposed budget of all inclusive fee in AMD, including separate lines  for the Consultancy fee and travel costs (e.g travel, translation, data collection assistance, printing etc,), as deemed necessary for the assignment.

Qualified candidates should send the requested documents to info@caritas.am and in CC to Gayane Norikyan (g.norikyan@caritas.am).

Applications that do not include all the required documents will be disqualified.

Applications received after the deadline will not be accepted.

Annex I: Phases of Evaluation

Inception & Desk Phase

This phase aims at structuring the evaluation and clarifying the key issues to be addressed.

The phase will start with initial background study, to be conducted by the evaluators from home. It will then continue with a kick-off session in Gyumri between the project implementing team and the evaluator. Half-day presence of the evaluator is required. The meeting aims at arriving at a clear and shared understanding of the scope of the evaluation, its limitations and feasibility. It also serves to clarify expectations regarding evaluation outputs, the methodology to be used and, where necessary, to pass on additional or latest relevant information.

In the Inception phase, the relevant documents will be reviewed. Further to a desk review, the evaluator, based on the Intervention Logic, will develop a narrative explanation of the logic of the Action that describes how change is expected to happen within the Action, all along its results chain, i.e. Theory of Change. This explanation includes an assessment of the evidence underpinning this logic (especially between outputs and outcomes, and between outcomes and impact), and articulates the assumptions that must hold for the Action to work, as well as identification of the factors most likely to inhibit the change from happening.

Based on the Intervention Logic and the Theory of Change the evaluators will finalise i) the Evaluation Questions with the definition of judgement criteria and indicators, the selection of data collection tools and sources, ii) the evaluation methodology, and iii) the planning of the following phases.

The methodological approach will be represented in an Evaluation Design Matrix, which will be included in the Inception Report. The methodology of the evaluation should be gender sensitive, contemplate the use of sex- and age-disaggregated data and demonstrate how actions have contributed to progress on gender equality.

The work plan for the overall evaluation process will be presented and agreed in this phase; this work plan shall be in line with that proposed in the present ToR. Any modifications shall be justified and agreed with the Evaluation Manager.  

The in-depth desk phase is phase is when the document analysis takes place. Selected interviews with the management and other stakeholders may be realised. The activities to be conducted during this phase should allow for the provision of preliminary responses to each evaluation question, stating the information already gathered and its limitations. They will also identify the issues still to be covered and the preliminary hypotheses to be tested. During this phase the evaluation team shall fine-tune the evaluation tools to be used during the Field Phase.

On the basis of the information collected, the evaluation team should prepare an Inception Report; its content is described in Chapter 5.

Field Phase

The Field Phase aims at validating / changing the preliminary answers formulated during the Desk phase and further completing information through primary research.

If any significant deviation from the agreed work plan or schedule is perceived as creating a risk for the quality of the evaluation or not respecting the end of the validity of the specific contract, these elements are to be immediately discussed with the Evaluation Manager and, regarding the validity of the contract, corrective measures undertaken.

During the field phase, the evaluator shall ensure adequate contact and consultation with, and involvement of the different stakeholders; with the relevant government authorities and agencies. Throughout the mission the evaluator will use the most reliable and appropriate sources of information, respect the rights of individuals to provide information in confidence, and be sensitive to the beliefs and customs of local social and cultural environments.

At the end of the field phase, the evaluation team will summarise its work, analyse the reliability and coverage of data collection, and present preliminary findings in a meeting with programme management.

At the end of the Field Phase an Intermediary Note will be prepared; its content is described in Chapter 5.

Synthesis Phase

This phase is devoted to the preparation by the contractor of two distinct documents: the Executive Summary and the Final Report, whose structures are described in the Annex II; it entails the analysis of the data collected during the desk and field phases to answer the Evaluation Questions and preparation of the overall assessment, conclusions and recommendations of the evaluation.

The evaluator will present, in a single Report with Annexes, their findings, conclusions and recommendations in accordance with the structure in Annex II; a separate Executive Summary will be produced as well (see Annex II).

The evaluator will make sure that:

• Their assessments are objective and balanced, statements are accurate and evidence-based, and recommendations realistic and clearly targeted.

• When drafting the report, they will acknowledge clearly where changes in the desired direction are known to be already taking place.

The evaluator will deliver and then present in Gyumri the Draft Final Report to discuss the draft findings, conclusions and recommendations. One half day of presence is required.  

The Evaluation Manager consolidates the comments and sends them to the evaluation team for the report revision

The evaluator will then finalise the Final Report and the Executive Summary by addressing the relevant comments.

Annex II: Structure of the Final Report and of the Executive Summary

The main sections of the evaluation report shall be as follows:

1. Introduction		A description of the Action, of the relevant country/region/sector background and of the evaluation, providing the reader with sufficient methodological explanations to gauge the credibility of the conclusions and to acknowledge limitations or weaknesses, where relevant.
2. Answered questions / Findings		A chapter presenting the answers to the Evaluation Questions, supported by evidence and reasoning.
3. Overall assessment (optional)		A chapter synthesising all answers to Evaluation Questions into an overall assessment of the Action. The detailed structure of the overall assessment should be refined during the evaluation process. The relevant chapter has to articulate all the findings, conclusions and lessons in a way that reflects their importance and facilitates the reading. The structure should not follow the Evaluation Questions, the logical framework or the evaluation criteria.
4. Conclusions and Recommendations
	4.3 Lessons learnt	Lessons learnt generalise findings and translate past experience into relevant knowledge that should support decision making, improve performance and promote the achievement of better results. Ideally, they should support the work of both the relevant European and partner institutions.
	4.1 Conclusions	This chapter contains the conclusions of the evaluation, organised per evaluation criterion. In order to allow better communication of the evaluation messages that are addressed to the Commission, a table organising the conclusions by order of importance can be presented, or a paragraph or sub-chapter emphasizing the 3 or 4 major conclusions organised by order of importance, while avoiding being repetitive.
	4.2 Recommendations	They are intended to improve or reform the Action in the framework of the cycle under way, or to prepare the design of a new Action for the next cycle. Recommendations must be clustered and prioritised, and carefully targeted to the appropriate audiences at all levels, especially within the Commission structure.
5. Annexes to the report		The report should include the following annexes:   The Terms of Reference of the evaluation   The names of the evaluators (CVs can be shown, but summarised and limited to one page per person)   Detailed evaluation methodology including: options taken, difficulties encountered and limitations; detail of tools and analyses.   Evaluation Matrix   Intervention logic / Logical Framework matrices (planned/real and improved/updated)   Relevant geographic map(s) where the Action took place   List of persons/organisations consulted   Literature and documentation consulted   Other technical annexes (e.g. statistical analyses, tables of contents and figures, matrix of evidence, databases) as relevant   Detailed answer to the Evaluation Questions, judgement criteria and indicators

Full call of proposals read here.