Author Archives: ITA

Seeking new solutions to educational measurement challenges: Pakistan’s leadership as a LMTF Learning Champion

hs2

Posted By Kate Anderson, Senior Policy Analyst, Center for Universal Education at the Brookings Institution

In April 2012, I was invited to Lahore to present on the topic of global learning metrics at the Quality-Inequality Quandary seminar hosted by the South Asian Forum for Education Development (SAFED).  This was one of the very first consultations of the Learning Metrics Task Force (LMTF), which would eventually go on to reach more than 1,700 people in 118 countries through its global consultation efforts. From those very early days, Pakistan has been at the forefront of rethinking how education and learning can be measured and ultimately improved for all children. This is not an easy task in a country where education is devolved to the provinces and there is no central agency responsible for monitoring learning outcomes. However, this challenge was seen as an opportunity when a group of agencies and organizations in Pakistan came together last year under the auspices of the LMTF 2.0 Learning Champions.

The Learning Metrics Task Force (LMTF) was convened in 2012 to make recommendations on how the education community can track progress on learning at the global level. Convened by the UNESCO Institute for Statistics and the Center for Universal Education (CUE) at the Brookings Institution, the LMTF was positioned within the context of other efforts to inform the broader education and development agendas, such as the Sustainable Development Solutions Network, the UN Secretary General’s Global Education First Initiative, the EFA Steering Committee, Open Working Group, and the UNSG’s High Level Panel. 

The Task Force reviewed existing empirical evidence and global discourse on learning and conducted a broad, public consultation in three phases. Teachers, education ministry staff, and youth comprised the majority of the more than 1700 consultation participants in 118 countries. Through this consultation and dialogue with a high-level Task Force, the LMTF came to consensus on a framework of seven learning domains, recommendations for global measurement areas, and a process by which to support countries to improve their assessment systems in order to improve learning outcomes. When the LMTF consultations were finished and the final recommendations report published, we were contacted by numerous colleagues who had participated in the LMTF consultations and wanted to continue this dialogue at the country level.

In some countries, it was the first time there was an inclusive national dialogue on learning. In others, the previous conversations on learning had been focused only on one or two domains, typically reading and mathematics. The LMTF provided a platform to discuss learning more broadly. For the LMTF Secretariat, we knew that we had explored only the tip of the iceberg on the controversial and often divisive topic of learning assessment, and much more work needed to be done at the global, national and local levels. The third technical report of the LMTF  describes some of the issues raised in these national consultations. Participants around the world converged on some of the same issues, and we heard over and over that much data is collected that does not lead to improved learning. They described a lack of technical capacity for assessment, including among teachers, as a key barrier to measuring and improving learning. They also mentioned that the domains captured in national examinations are limited and therefore curtail the content covered in the classroom, as teachers feel pressure to teach only the subjects covered in the exams. Participants expressed a desire to think through these challenges and potential solutions with other countries around the world that are grappling with similar issues.

In response, the LMTF began to think about ways to leverage the collective expertise of those who participated in the first phase of the Task Force to support a nationally-driven but globally-informed process to critically look at learning and assessment issues. Our colleagues in Pakistan were at the top of our list to reach out to for advice on how to support countries. We eventually decided on the Learning Champions model, in which a group of countries grappling with issues of learning and assessment could come together, learn from each other and be supported by Task Force members.

In July 2014, 15 countries applied and were selected as Learning Champions under the auspices of LMTF 2.0. National stakeholders will be working over the next year to adapt LMTF recommendations to their national contexts and priorities in Argentina (Buenos Aires), Botswana, Canada (Ontario), Colombia (Bogotá), Ethiopia, Kenya, Kyrgyz Republic , Nepal, Pakistan, Palestine, Rwanda, Senegal, Sudan, Tunisia, and Zambia. A key component of the Learning Champions initiative is broad inclusion in guiding policy decisions, including but not limited to teachers, students, government officials, civil society, and development agencies. Countries will share what they are learning with the Task Force and other Learning Champions, in addition to other countries in their regions and the global education community.

Learning Champions are seeking to develop new solutions to their unique educational challenges. In Pakistan, this means reviewing current assessment practices at the national and provincial levels and converting the LMTF recommendations in to practical strategies and assessment tools from early childhood through upper secondary education. ITA is leading this effort along with the Inter-Board Committee of Chairmen, the National Education Assessment System in Islamabad; the Provincial Education Assessment Centres from Sindh; the Policy Planning and Implementation Unit in Balochistan; the Provincial Institute for Teacher Education; the Kashmir Education Assessment Center; the Punjab Examination Commission in Punjab; and the Aga Khan University Institute for Educational Development.  In all of the Learning Champion countries, there remains much to be learned on the different ways assessment helps (and in some cases hurts) learning and we hope that the lessons from these 15 countries can be used to inform efforts to expand learning assessments in order to improve all children’s learning experiences, in Pakistan and throughout the world.

 

There’s No One-size-fits-all Approach to Improve Learning

learning-metrics_0

During the struggle for freedom in Zimbabwe, I always remember a certain general who used to say, “The challenge is not finding the problem – that’s easy. The hard part is identifying the solution.” This is as true in war as it is in every other human endeavor.

With regard to education, we know we have many problems. This is demonstrated well in the latestGlobal Monitoring Report. We have millions of children not in school (30 million of them in Africa), and millions more who are in school but not learning. This is a huge problem of inequality and wastage. So what are the solutions?

Addressing the Global Data Gap on Learning

Since July 2012, I have had the opportunity to contribute to a group that has been actively working on this issue. The Learning Metrics Task Force (LMTF) was convened to build consensus among diverse stakeholders in education on how learning should be measured and tracked globally.

From the outset, the group recognized that effective measurement of learning is most certainly not the same as improving learning, and that it is only one part of the solution to the many problems facing global education.

Still, we agreed that without good data on learning we cannot fully define the problem, and in turn we cannot find a solution. Knowing where we stand on learning outcomes is essential for identifying the best ways forward for improving learning achievement for students (e.g. support for teachers, improved learning materials for specific subjects). So we set about developing a set of recommendations on how to address this particular gap: the lack of a common definition and metrics for learning outcomes that can be tracked globally.

To join the LMTF mailing list, please contact [email protected].

The final task force recommendations released in September 2013 were developed based on input from 30 member organizations, 186 technical experts, and public consultations engaging more than 1,700 individuals in 118 countries. But these summary figures do not tell the whole story. In fact, the road to consensus among task force members was not at all straightforward. With so many different sectors, contexts, and geographies represented, of course we did not always agree. However, the disagreements pushed the group to be more innovative in its thinking.

No preference for specific tests

For instance, the task force agreed that it would not privilege the areas of learning that are currently measured through large-scale international assessments, but rather consider them together with other areas that are less well measured or even defined. The task force would also not endorse any one test or any one method for collecting data on learning outcomes. We learned through consultation that there is strong interest globally in exploring other areas of learning beyond literacy and numeracy, and other ways to measure learning beyond paper-and-pencil tests. So the task force recommended a holistic framework of seven learning domains, seven areas of measurement for global tracking, and that multiple methods be considered when designing systems to assess learning opportunities and outcomes.

Still, as much as the task force tried to take all perspectives into account, divergent views on the these topics remain, and the task force welcomes continued dialogue and collaboration as it moves into the next phase of work.

One size doesn’t fit all

As a former education minister and current chair of the Association for the Development of Education in Africa (ADEA), I led the last LMTF working group tasked with exploring how measurement of learning can be implemented to improve education quality. In the brief five months allotted, we only just scratched the surface of this complex topic. But one thing we heard time and again in our consultations with more than 50 ministers of education, permanent secretaries, and their representatives1, was the pivotal role of teachers in any potential solution for the learning crisis. They also confirmed what most of us already suspected—that a one-size-fits-all approach would not work if we are to support countries in measuring and improving learning levels.

As the task force wrapped up its first phase, we agreed that we had much more work to do if our ultimate goal is to help increase levels of learning worldwide.

Accordingly, for the next phase of LMTF work over the next two years, the task force will shift its focus: from global-level recommendations to actively supporting countries.

And from a strict focus on measurement we will  expand to explore how measurement data can be used to improve learning outcomes—including data that teachers and schools collect to inform their instruction.

Focus on Africa

In Africa, the LMTF recommendations are influencing discussions on education quality. ADEA in particular will support African education ministries in developing an Inter-Country Quality Node (ICQN) on Teaching and Learning under the joint leadership of Rwanda and Uganda. This effort is supported by the African Union Commission, which is an LMTF member, and by the Ministers of Education of Rwanda and Uganda, who have committed to providing leadership, as well as financial and human resources to get this important work underway.

The goal is to have a forum for countries to exchange ideas and support each other as they explore ways to advance the ideas captured in the LMTF recommendations. Initially the ICQN will focus on countries in East Africa, but as with other Nodes, the reach will eventually extend to other countries in the region. Previous Nodes on topics such as peace education and technical and skills development, and use of African languages have been successful in bringing together representatives of education ministries from different African countries to engage in joint problem-solving and share information on innovative educational experiences in Africa.

Task force partners in other parts of the world—from Latin America to Asia—are also taking up activities to advance the LMTF recommendations. By helping to build a global network of actors focused on access plus learning in education, the task force is working toward better collaboration among existing agencies, stronger and more relevant support to countries, and a coalition of stakeholders with a common vision of learning for all.

What’s Next for the Learning Metrics Task Force?

palestine_student

Measurement, if done well, can play a critical role at many levels in improving education quality and learning. Effective teachers measure learning in the classroom to adjust and individualize instruction. Head teachers, school administrators and school district leaders measure learning to target resources and improve school quality. Many national governments also measure learning to diagnose the overall health of their education systems and develop policies to improve the teaching and learning process and ultimately student learning outcomes. Civil society actors, donors and development agencies use assessments to measure the effectiveness of programming and advocate for effective education policies and practices.

Global Commitment to Improving, and Measuring, Learning

There has been a longstanding and broad-based commitment among the world’s education ministers to improving, and hence measuring, learning outcomes.  In particular, Goal 6 of Education for All commits world leaders to: “Improving all aspects of the quality of education and ensuring excellence of all so that recognized and measurable learning outcomes are achieved by all, especially in literacy, numeracy and essential life skills”.

In the first phase of the Learning Metrics Task Force (LMTF 1.0), the task force heard from nearly 2,000 people, including more than 50 ministers of education, permanent secretaries or their representatives through a broad consultation process. The vast majority of consultation participants, echoing previous commitments to EFA Goal 6, articulated the importance of improving learning outcomes and the role of measurement in helping to inform actors, from teachers to ministers, about the efficacy of their interventions. Debates were heated, however, about how to best assess learning to improve outcomes. What competencies young people in all countries should develop, what approaches to measurement should be used, and how issues of equity should be addressed were all examples of the tough questions discussed. Ultimately, the task force weighed a wide range of (often conflicting) input and after much debate came to agreement on a series of recommendations.

Recap of LMTF 1.0 Recommendations

The consultative process of LMTF 1.0 was structured around three guiding questions: What learning is important globally? How should it be measured? How can measurement of learning improve education quality?

The collective response in the form of final recommendations was launched during the 2013 U.N. General Assembly meetings as a contribution to the U.N. secretary-general’s Global Education First Initiative (GEFI):

1. Global Focus on Access plus Learning. The task force calls for a return to the vision of the EFA agenda, aligned with the priorities set out by GEFI in order to shift the past focus in the Millennium Development Goals (MDGs) from access to access plus learning. The collection and appropriate use of better data on learning is central to this effort.

Areas of Measurement Recommended for Global Tracking


2. Learning Competencies. The task force calls upon education systems to offer opportunities to children and youth to master a set of competencies in seven domains of learning, which are considered important for all children to learn in addition to the wide range of other learning objectives countries may have for their students.

3. Learning Indicators for Global Tracking. The task force recommends a small set of learning indicators within seven areas of measurement (see figure) to be tracked in all countries. These areas of measurement represent fundamental learning opportunities from early childhood through lower secondary. Some indicators within these areas currently exist, while others need to be developed.

4. Supporting Countries. The task force recommends that support be provided to countries that desire it in strengthening their assessment systems in ways that help improve learning levels. An international, multi-stakeholder partnership would ensure better collaboration among existing agencies and fill essential gaps in support to countries.

5. Equity. Measurement of learning must include an explicit focus on equity, with particular attention to rising inequality within countries. Data on child characteristics (e.g. sex, socioeconomic status, geographical location, disability) together with information on such things as school conditions and teacher quality should be used to ensure equitable learning opportunities.

6. Assessment as a Public Good. Measures for the indicators recommended for global tracking must be considered a public good, with tools, documentation and data made freely available. No country should be precluded from measuring these particular learning outcomes due to financial constraints.

7. Taking Action. The task force calls on all actors to play a role in advancing these recommendations.

The task force acknowledged that it had completed its work as originally set out; however, it also recognized the high demand from stakeholders to take advantage of the momentum and interest gained thus far. In response, the task force has agreed to make the transition into a new stage of work, with a focus on implementing its recommendations.

LMTF 2.0: Debates and Next Steps

With a diverse group of engaged and committed stakeholders representing both the global south and north, and high interest from multiple governments to adapt and try out the recommendations to improve national learning levels, task force members held a “next steps” meeting in November 2013. The task force agreed to transition to a new phase of work—being referred to as LMTF 2.0—and to continue collaborating as a multi-stakeholder group for two more years, running from January 2014 to December 2015. A number of questions around the overarching goal, working model, and results desired were debated during the meeting and are summarized below.

Overarching Goal

The task force acknowledged that the group had only just begun to answer the third guiding question from LMTF 1.0: How can measurement of learning improve education quality? To address this question adequately, task force members were clear that much more attention and energy was needed at the country level, concurrent with the ongoing global-level work.

There was vigorous debate about whether the task force should expand its scope to address improving learning broadly, including the wide range of strategies required—teacher development, teaching and learning materials, school nutrition, etc. Task force members acknowledged that measurement is only part of the answer. Arguments for expanding the scope included the importance of shifting the emphasis in the next phase to improving learning and not representing assessment as an end to itself. Furthermore, members recognized that LMTF is the main global initiative specifically focused on improving learning, and so some argued that it should expand its remit to fully cover the issue. Others argued that expanding to other areas that help improve learning, in addition to better student assessment, would likely be too broad and unmanageable and could possibly encroach on the mandates of other existing initiatives (e.g. UNESCO’s Teachers Task Force for EFA).

Ultimately, the task force decided that the overall goal of LMTF 2.0 should be to help countries improve learning levels, and that it should focus its activities on how to support better assessment systems and better use of assessment data to do just that. Any work at the global, regional and of course country level that the task force takes up as part of LMTF 2.0 should be in service of this overarching goal.

Working Model

Task force members acknowledged that the next phase of work requires a different working model from before. In LMTF 1.0, the task force began by collecting input from technical working groups and broad public consultations on the three guiding questions listed above. Then discussion and debate on the consultation feedback followed, and finally the task force reached consensus on its recommendations. In contrast, LMTF 2.0 will focus on implementing those recommendations, which requires a much more decentralized approach. An expanded partnership of LMTF member organizations will: take on aspects of the work based on their core mandates and competencies; collaborate with others as appropriate; and regularly share, discuss and update each other on progress. Over the next two years, the task force will serve as a network on improving learning with multiple stakeholders engaged; however, it will also be important to develop a clear plan for transitioning the work after two years, either through embedding it within partner organizations or some other approach.

Five Results to Achieve

The task force will work to achieve the following five main results at the end of two years:

  1. Technical. Indicators in each of the areas of measurement recommended for global tracking are developed (to different degrees of readiness for implementation).
  2. Institutional. At least 10 “Learning Champions” use task force recommendations to support country-level work on learning assessment and use of assessment data to improve learning.
  3. Political. Task force work supports ability of access plus learning to feature in the post-2015 development agenda.
  4. Assessment as a Public Good. A strategy is developed for advancing an agenda in which learning data is supported as a global public good.
  5. Knowledge Sharing. Actors and experts in learning assessment share knowledge and coordinate efforts.

Developing Indicators for Global Tracking

Among the LMTF 1.0 recommendations, the task force proposed seven areas of measurement as desirable and feasible for tracking at the global level (see figure). Technical work is now required to develop indicators under each of these seven areas. Accordingly, a number of LMTF partners have already begun organizing to undertake this work.

In addition, the task force has opted to add one more area of measurement: equity.  Measuring equity is not a stand-alone effort but one that must cut across all the other seven areas; however, a methodology and process for doing this is needed. Hence, a group of organizations will focus on equity, examining how the LMTF global indicators can provide critical information on characteristics that lead to disparities in learning, in order to support equitable improvements in learning outcomes (e.g. gender equality in education). The aim is to expedite the work over the next two years to be ready (or nearly ready) for implementation post-2015. Members agreed it is also important to take into account and support the linking of assessment results from existing regional and international assessments, as well as citizen-led assessments, which can contribute to a healthy ecosystem of assessments for all children, those in and out of school.

Supporting Countries

Consultation participants in LMTF 1.0 identified many challenges to improving learning, and in particular, the difficulty of developing and sustaining robust assessment systems at the country level and most importantly effectively using assessment data to help guide actions that improve learning. In particular, a range of difficulties were identified, including lack of national ownership of the assessment system; lack of national institutions with sufficient technical capacity; insufficient political will to assess learning regularly and make the results publicly available; lack of information about how to use data to guide actions that improve learning; and scarcity of neutral sources of information on the advantages and disadvantages of the various assessment tools available. Many also identified the desire to learn from each other, recognizing that countries have different strengths and weaknesses.

To help address these challenges, the task force invites “Learning Champion” countries, states, districts and cities to join LMTF 2.0 and work together to advance the goal of improving learning outcomes. Learning Champions will have the opportunity to connect with and learn from each other as well as other education systems and LMTF partners in the region and across the globe; receive technical guidance on navigating the many options related to learning assessment; and play a leading role in developing global good practice on assessment and learning. In turn, LMTF partners will look to Learning Champions to provide country-level expertise and feedback during the development of indicators described above, as well as potentially pilot new assessment instruments relevant to identified national priorities. The task force will also work with relevant regional bodies to facilitate coordination and communication.

Mitigating Unintended Consequences

The task force recognizes the potential unintended consequences of measurement, such as the overuse and misuse of high-stakes standardized tests, corruption and cheating, and the potential advantage for those who can afford tutoring and courses to boost test scores. Still, these consequences are not inevitable, and there are ways to address these concerns—for instance, ensuring that decisions on measurement are country-driven rather than prescribed at the global level; ensuring that data used for accountability are fair and accurate representations of education quality and learning; and ensuring that at least some test items are freely available to low and middle-income countries (e.g. a free bank of common items countries can incorporate into their existing assessments). In its ongoing work, the task force will continue to provide recommendations to mitigate the harmful effects of testing and focus on leveraging the power of better information to make decisions that improve learning for all children and youth.

Opportunities to Participate

As in the previous phase, LMTF 2.0 will continue to work through an open, multi-stakeholder approach. Education ministries interested in signing on as Learning Champions are invited to submit letters of interest by May 15, 2014. In the coming weeks, there will be additional opportunities for organizations and individuals to get involved as partners and working group members. Sign up to receive these updates by emailing[email protected].

The Critical Voice of Youth in Education Advocacy and Global Citizenship

syrian_refugees008

Last week, at the Countdown to 2015 Summit in Washington, DC, “The Education We Want: An Advocacy Toolkit” was launched, featuring real-life stories of youth who have successfully advocated for expanding national education programs to reach the most marginalized children and youth. The toolkit outlines practical steps in a user-friendly and engaging format to help youth carry out their own advocacy campaigns. As part of a final push to get all children into school by 2015, the Youth Advocacy Group (YAG)—a group of young leaders from around the world who have been working to strengthen momentum for global education as part of the UN secretary-general’s Global Education First Initiative(GEFI)—will disseminate the toolkit to youth around the globe. Their plans also include staging youth “takeovers” of governments worldwide, following up on their successfulMalala Day UN Youth Takeover on July 12, 2013.

With the sunset of the current Millennium Development Goals in 2015, the many stakeholders working on global education issues are focused on two things: 1) delivering on the promise of universal primary education, and 2) simultaneously ensuring that universal access plus quality teaching and learning are part of the next global development framework. The voices of youth have been critical in these debates, in no small part due to the leadership of the YAG.

The YAG youth demonstrate remarkable passion, leadership skills, and eloquence when speaking about the barriers to quality education in their communities and in the world. Their ability to apply diverse skills—including so-called 21st century skills such as collaboration, problem-solving, digital literacy, and creativity—to the major challenges facing our world is the quintessence of what it means to be a “global citizen.” In fact, the Youth Advocacy Group is co-convening a Global Citizenship Working Group along with UNESCO and theCenter for Universal Education here at Brookings to dig deeper into what it means to be a global citizen and how education programs can foster and track these skills and values, which is one of the three GEFI priorities. Defining and tracking the competencies related to being a “citizen of the world” was also one of the recommendations of the Learning Metrics Task Force (LMTF).

There is now an opportunity to include global citizenship education in the post-2015 development agenda as part of the knowledge, skills, and competencies that all learners require in the 21st century. However, there is currently a lack of consensus about what skills and values constitute global citizenship. There is also much debate on the terminology used to describe these competencies, as the term “citizenship” is usually associated with a political entity. However, some important groundwork has already been done by UNESCO and the LMTF.

Through meetings on this topic by UNESCO, the following core competencies have emerged as likely outcomes of global citizenship education:

  • knowledge and understanding of specific global issues and trends, and knowledge of and respect for key universal values (e.g., peace and human rights, diversity, justice, democracy, caring, non‐discrimination, tolerance);
  • cognitive skills for critical, creative and innovative thinking, problem‐solving, and decision‐making;
  • non‐cognitive skills, such as empathy, openness to experiences and other perspectives, interpersonal/communicative skills, and aptitude for networking and interacting with people of different backgrounds and origins; and
  • behavioral capacities to launch and engage in proactive actions.

The LMTF also discussed this topic with education ministries, teachers, civil society actors, and other stakeholders  and these conversations revealed a similar set of competencies with an additional emphasis on climate change, environmental awareness, leadership and digital literacy.

Based on the results of these consultations and a review of existing efforts—together with the LMTF’s recommended seven domains of learning—the global citizenship working group that YAG, CUE and UNESCO are convening will work over the next year to build consensus on the core competencies of global citizenship. Members of the Youth Advocacy Group in particular expressed an interest in shaping how learning related to global citizenship is assessed. However, they cautioned that the traditional ways of testing may not be appropriate for measuring global citizenship and could stifle innovation and creativity. In this absence of reliable measurement tools, there is an opportunity to move beyond traditional methods and redefine how learning is measured in the context of global citizenship. This includes looking at new ways of measurement that are more engaging to children and youth, including through technology. Without a collective effort on measurement by the actors involved in global citizenship education, the education community risks having a continued focus on testing and learning of only cognitive or academic skills, such as reading and numeracy.

The working group process will include: exploring the different definitions and competencies related to global citizenship; identifying ways in which these competencies are currently measured, with an emphasis on educational outcomes; building consensus on core competencies of global citizenship that are relevant in all countries; and proposing new and innovative ways of assessing learning in this area.

The Youth Advocacy Group is well-positioned to lead the conversation among youth on this complex topic, as the success of the project is dependent on their energy, innovative spirit, and willingness to tackle the most difficult problems of our time.