Monitoring the extent to which education systems are meeting the UN Sustainable Development Goals in education (SDG 4) is an essential part of their achievement. To support countries in monitoring learning progress towards achieving SDG 4, the UNESCO Institute for Statistics (UIS), in partnership with the Australian Council for Educational Research (ACER) and funded by the Bill & Melinda Gates Foundation, has developed a new tool: the Assessment for Minimum Proficiency Levels (AMPL).

The objective of AMPL is to enable countries to produce and report on SDG indicators 4.1.1a and 4.1.1b. Its comparative advantage over other assessments is its cost- and time-effectiveness.

AMPL has been developed in two phases

AMPL was initially developed in 2021, as part of the Monitoring Impact on Learning Outcomes (MILO) project, to be administered in English and French at the end primary education (AMPL-b) in six sub-Saharan African countries (Burundi, Burkina Faso, Cote d’Ivoire, Kenya, Senegal and Zambia). Its successful implementation in the middle of the COVID-19 crisis proved that measuring learning could be done in a cost-effective affordable way.

In 2022, AMPL was administered in Sierra Leone and, following its translation into Urdu, in two provinces of Pakistan (Khyber Pakhtunkhwaand Islamabad Capital Territory) by the World Bank.

In 2023, a new module to assess learning at the end of lower primary (AMPL-a) was developed and administered in Gambia and Zambia. Kenya, Lesotho and Zambia administered AMPL-b. India piloted both assessments in English and Hindi with the aim of administering AMPL-a and AMPL-b in 2024.

Implementation of AMPL by tool and language

The AMPL can be administered as a standalone assessment, as was done in 2023, or it can be integrated into existing national or cross-national assessments, as was done with the MILO study.

The AMPL implementation results in the four sub-Saharan African countries in 2023 showed that the proportion of students meeting at least the minimum proficiency levels was low. Across these countries, more students reached the minimum level in mathematics than in reading. Girls and boys had the same learning levels in mathematics, but girls generally outperformed boys in reading.

Percentage of students who achieved the minimum proficiency level, by level and subject

The AMPL tools include questionnaires for students and school leaders to help explain learning levels with reference to students’ individual, home and school characteristics. Predictably, students’ socioeconomic background correlates with their learning outcomes. For instance, students with a more nutritious diet, from families with higher wealth, with literate parents had higher proficiency in mathematics and reading, on average, than other students.

AMPL was conducted in English and administered in schools where English was the language of instruction, yet over 94% of students indicated that the main language spoken as home was not English.

AMPL helps develop country capacity to measure education outcomes

The process of implementing AMPL helps develop participating countries’ capacity to administer large scale assessments, according to the rigorous AMPL technical standards.

“Our national assessments are going to benefit a lot in the future from the experience that we have gained from participating in AMPL.”

“We are going to do e-assessments, it’s going to be the first time national assessments are being done this way and most of the learning we are going to be borrowing from the experiences of AMPL.”

Not only does the AMPL provide robust results on primary school students’ learning levels, but it can also be applied in a wide range of education systems and contexts. It can be translated into other languages, supporting specific monitoring and reporting needs.

A particular challenge in measuring learning at early grades is that many students cannot read or write well enough to complete a written assessment. To address this, an innovation of the project is the measurement of listening comprehension and decoding skills using pre-recorded audio stimuli, which participating countries found particularly helpful to adopt in their national assessments:

“I think that [the listening comprehension component] is going to strengthen our national assessment in the future, in the sense that we have also learned that we can test other skills that we have until now been overlooking.”

The AMPL-a listening comprehension tasks were provided to countries in high-quality audio files but all participating countries chose to re-record the audio files in local accents.

“it’s best when you give [the assessment in the] country accent that tests the knowledge, skills and attitudes that this child has, is actually based on the correct accent.”

Developed further, students’ learning progress could be monitored up to the end of lower secondary school, bringing better visibility to our progress towards SDG indicator 4.1.1c as well. As one national team member commented:

“The capacity building sessions … were very helpful and it’s something that I think should be encouraged going forward. We look forward to having more of such sessions so that … capacity is built for [our] team”.

The post A new tool to fill the data gap on learning: AMPL appeared first on World Education Blog.

Source: 2024 SDG 4 Scorecard by UIS and GEM Report

Eight in ten countries have now set benchmarks against at least one indicator, a demonstration of the increased recognition of this new global way of monitoring progress towards SDG 4. This high participation demonstrates that the process is fulfilling one of its key objectives: to increase ownership of the agenda at the national level.

Benchmarks also help keep the global SDG 4 process relevant by being flexible as new education priorities emerge: in 2023, countries set benchmarks for an eighth indicator, school internet connectivity, in response to the priority assigned to digital transformation at the UN Transforming Education Summit in 2022.

Countries’ benchmarks demonstrate governments’ willingness to be held accountable for progress, first and foremost to their own people. While the SDG Summit showed that progress towards all global education targets was well off track, the 2024 SDG 4 Scorecard shows that progress towards national targets is also insufficient. Countries are making good progress in connecting schools to the internet and in raising teacher qualifications, but progress on the six other benchmark indicators is not on course. For instance, two thirds of countries with data have made no or slow progress towards their upper secondary completion rate targets since 2015. Countries are even moving backwards on closing gender gaps in upper secondary completion and on public expenditure on education.

The 2024 SDG 4 Scorecard was launched today at the UNESCO Conference of Education Data and Statistics, which will also recognize and celebrate countries’ contributions to the benchmarking process. The event is also an opportunity to drive the political process that supports benchmarking – a process not all countries are yet acquainted with and one that is particularly novel in an international context, and especially in education. Countries were invited to set national targets as part of their participation in the implementation of a global agenda. New ways of working are needed at the international level to collaborate with countries, provide them with transparent updates on the assessment of their progress and give them the chance to contest findings, seek clarifications or propose corrections.

This 2024 Scorecard edition also shows that the benchmark process requires data of better quality. This requires work on definitions, such as on what it means to be a qualified and trained teacher. It also requires better shared understanding of the appropriate data sources to monitor some indicators. These actions and others will help fill notable data gaps on some of the most important policy-related indicators. For example, there are no data for 32% of country-indicator pairs and not enough data to estimate trends for another 14% of them. Efforts are particularly needed to measure proficiency in reading and mathematics, for which data are missing in 73% of the pairs.

Data can inject momentum into policy agendas and inform policy planning and development. More work is needed to use the SDG 4 Scorecard as a starting point to explain what policies are linked to slow or fast progress. All of us working on education share a responsibility to help countries reach the national benchmarks they are committed to achieve.

The post 2024 SDG 4 Scorecard now out: see how countries are progressing towards their national targets appeared first on World Education Blog.

The Conference has three core objectives this week. It aims to:

1. Establish the process for an international community of practice among education statisticians and set a broad agenda for the Technical Cooperation Group (TCG) on SDG 4 Indicators.
2. Communicate, discuss, and reach consensus on key issues regarding concepts, definitions, methodologies, and operational aspects of SDG 4 indicator measurement in the form of recommendations and guidelines for adoption as international standards to improve comparability.
3. Debate the impact of technological developments on education statistics and ways in which the community of education statisticians can benefit from opportunities and address challenges.

The UIS, which is hosting the conference, is the official source of data on SDG 4 and works to fill data gaps along with national statistical offices, line ministries and other statistical organisations, as this blog series on education data has explained in the past few days.

Since the goal was set, the UIS has introduced new approaches to fill data gaps, which have enabled it to improve the share of countries reporting on government spending on education from 68% to 90%. A new model visualized on the VIEW website has been used to compile multiple data sources on out-of-school children, increasing the share of countries for which we have data from 62% to 98%. This innovative approach provided new numbers on how many children were out of school in 2022 in countries such as Ethiopia, Kenya and Nigeria that had not reported data for over a decade, although of course there is uncertainty for areas where conflict is hampering data collection.

Despite the huge amount of effort countries have put into monitoring SDG 4 indicators, however, almost half are not measuring children’s learning levels as they progress through school, leaving 680 million children’s achievement uncounted. Some regions suffer particularly large learning data gaps, with 93% of children in Central and Southern Asia and 62% in sub-Saharan Africa and Eastern and South-Eastern Asia of children without ever having had their reading skills assessed at the end of primary or lower secondary school since 2015 reported in the SDG database.

In 2016, the Technical Cooperation Group on SDG 4 Indicator (TCG) was established to look at exactly how to monitor progress towards SDG 4. It sets and approves new indicators, including the seven SDG 4 benchmark indicators monitored in the new annual SDG 4 scorecard. It ensures a balanced representation of UNESCO member states and has five thematic working groups, all of which will have a session at the Conference this week covering issues such as administrative data, household surveys, ISCED, teachers, education expenditure and SDG 4 benchmarks.

While the International Labour Organisation recently celebrated the 100-year anniversary of its labour statistics conference, education statisticians have not benefited from a regular open forum where questions about improving comparable data can be explored. It is therefore the appropriate moment – halfway to 2030 – for a wider gathering to set a new agenda for education statistics.

Many of the issues have been detailed in the series of blogs that have featured on this site the last few days, but the top issues are listed here below:

1. Education data ecosystem: There is a need to assess coverage and efficiency of data collection efforts and to establish better synergies that might combine different types of data sources (such as administrative data and household surveys) to expand data coverage. Plans need to include efforts to develop the capacity of Member States.
2. ISCED: This framework, which maps every national education system to facilitate comparisons between countries, requires constant improvements. It is a valuable tool that has inspired the UIS to create the ISCED-T, a framework to capture the characteristics of teacher training programmes in order to improve the way we compare teacher quality from one country to another.
3. Administrative data and capacity development: As covered in this blog, some long-established processes for collecting data need to be improved to help fill data gaps, to improve population estimates, to improve capacity building at the country level and assess support needed, and to prioritise a future education monitoring agenda post-2030.
4. Teacher indicators: There is a lot of uncertainty still around the basic definitions being used to measure teacher quality, such as how to define when and if teachers are qualified or trained. The Conference will debate new definitions to be considered, and new methodologies that can help monitor this issue, including mapping teacher policies.
5. Expenditure: There are multiple sources of data for government expenditure on education for the same country and year. There is a need to explore ways to combine different sources, and how to obtain more regular private expenditure data.
6. Household surveys and the integration of their data: As per this blog, the emphasis on equity in the SDG agenda has drawn household surveys into the spotlight. They have also been useful for filling gaps on core indicators such as out-of-school and completion rates, as done on the VIEW website. The Conference will look at validating indicators that rely on multiple sources, developing an inventory of household surveys, and harmonizing questions in surveys so that their results can be combined and compared.
7. Learning outcomes: There remain considerable challenges as the headline of this blog suggests, especially related to data gaps. The Conference will provide a forum to discuss standards and criteria that assessments need to fulfil so that their results can be reported. It will also discuss operational and cost issues, as efforts for collective action need to be accelerated.
8. SDG 4 benchmarks: The SDG 4 benchmarks and SDG 4 Scorecard, the 2024 edition of which was released this morning, is the new way of monitoring progress towards SDG 4. The conference will ensure all Member states are familiar with the process, discuss processes for raising queries, how to update benchmarks, and how to ensure alignment between national, regional and global targets.
9. Technology: Digital technology development is changing approaches to education monitoring. Big data has potential implications for monitoring SDG 4; protocols for its use and the potential of AI for policy-related indicators will be among the topics to discuss on the third day of the Conference.
10. Partnerships: Finally, the Conference will also discuss the potential of collaborations, notably with regional organizations and the broad family of UN agencies, to support monitoring some of the multiple angles of SDG 4.

The post Learning levels unknown for over half a billion children. A new education data ecosystem is needed appeared first on World Education Blog.

Working out how to monitor the ambition in our global education goal, SDG 4, required a certain amount of innovation back in 2015. One of the key suggestions made at the time was that ‘the more data can be combined, the more useful they are’. Data integration, in other words.

Even before the SDGs, other sectors had faced similar challenges to combine different data sources and types together. Wasting and stunting, for instance, which are ways of measuring malnutrition, were calculated thanks to a Joint Child Malnutrition Estimates group in 2011. In health, multiple administrative and survey data sources were combined by the UN Inter-agency Group for Child Mortality Estimation, which created a new model to generate annual estimates for under-5 mortality, and by the Inter-Agency Group for Maternal Mortality Rates.

Which parts of SDG 4 do we monitor with data integration?

Data integration can either involve merging different sources of the same type, or by merging different types of sources. This requires education statisticians to increasingly work out how to incorporate these sources in the estimation of indicators. It is not always simple.

An example of the first is learning outcomes from different assessments, which are the same source, but often have slightly different methodologies. This requires integration in the sense that the results are not immediately comparable and may require further analysis. An example of the second is the out-of-school rate, which can rely on both administrative and survey data, as seen on the VIEW website, or teacher continuous professional development, which can draw on administrative and learning assessment data.

Distribution of SDG 4 global and thematic indicators, by potential data source

Integrating data to monitor completion rates

In 2020, a review of the Inter-agency and Expert Group on SDG Indicators approved the completion rate at three levels of education (primary, lower secondary and upper secondary) as a global indicator. It was one of only six among more than 200 proposals to be successful. Estimating completion rates requires some form of integration in order to be flexible around the fact that many children enrol late, and some may repeat years, especially in poorer countries.

One of the benefits of combining multiple survey data sources is that it can fill in the gaps as a result of infrequent survey cycles or sampling errors. The approach taken borrows from similar solutions in health statistics but has been adapted to the education context. It estimates an underlying trend. Late completion is explicitly modelled by specifying the magnitude of the delay as a function of age. Age misreporting concerns are also addressed. By addressing various data quality concerns associated with survey data, these estimates are also less sensitive to individual surveys, the year in which they were conducted, and the type of survey that happens to be the latest available in a given country.

Combining data sources to estimate out-of-school rates.

The need for a methodology that combines data sources to estimate out-of-school rates was recognized 20 years ago, when it was acknowledged that ‘some sort of composite approach may be needed for estimating time series and producing estimates for the most recent year’.

In the absence of such an approach in the past, measurements were done using enrolment records from school censuses. But there were three challenges with this approach: enrolment records are often incomplete or inaccurate; those records needed to be combined with population estimates, which come from a different and often inconsistent source; and schools were not always able to determine students’ ages accurately.

In recent years, many of these countries have carried out household surveys which, despite their own weaknesses, can help fill some gaps and address challenges related to age and population.  A model was accordingly developed to add these sources to the administrative ones in order to get a better picture. The results of this model were reported for the first time In September 2022, and visualized in the VIEW website. Thanks to the new approach, new out-of-school rates were produced for countries such as Nigeria and Ethiopia that hadn’t had data reported for over a decade. The latest data release using this approach has estimated there are still 250 million children, adolescents and youth out of school.

What are the challenges associated with data integration?

When combining data, the methods must be understood so that that they can be accurately interpreted. Even more critically, although these models can only be estimated at a global, central level, it is important to ensure that countries participate in the process and engage with it. This is important not only to make sure they feel ownership of the results but also to help develop the capacity of national statisticians so that they can feed into the model. As things stand, there is no systematic mechanism for countries to seek clarifications, understand the methods underpinning the estimates, contest results that contradict their own understanding of the actual situation, but also proactively contribute data sources and ideas for the development of the models.

There are also technical  issues that need to be ironed out and which will also be discussed at this week’s conference, such as how female and male rates should be estimated, and how to align the estimates of the out-of-school and completion rate models.

What further developments are needed?

We need to formalize good practice for the way estimates are reported with guidelines similar to GATHER, an approach followed in health statistics.

We need to build the participation of countries in these new models. Countries should review model results in a systematic way, familiarize themselves with the rationale and implications, identify errors and seek clarifications, contribute ideas to potential areas of model development, and provide additional and up-to-date data sources. Familiarizing ministries of education and the expert community with estimate-based SDG 4 indicators as a new way of monitoring progress requires extensive communication. The suggestion is that the same approach as in health is taken in education so that the UNESCO Institute for Statistics covers the models in the workshops it is already running with countries. An inventory of surveys that will support data integration, ensuring countries are involved in the data inputs used, is also recommended.

A joint model combining the out-of-school and completion rates should be developed. The GEM Report and the UIS are currently working to develop a model that integrates the completion and out-of-school rate estimates to ensure they are consistent with each other.

New ways of integrating data should be considered to estimate other elements of SDG 4. The suggestion is that similar models could be used that draw in data, for instance on children who are over-age for their grade, those learners in non-formal education, and more.

The post Data integration: How do we measure progress towards SDG 4 – Part 4 appeared first on World Education Blog.

This is a part of a series of blogs, aiming to inform about some of the core challenges and solutions to collecting quality data which will be discussed in depth next week at the first ever Conference on Education Data and Statistics, convened by the UNESCO Institute for Statistics (UIS).

When the SDG 4 goal on education was set in 2015, it moved the agenda from a focus on getting children into school to ensuring that they are also learning. It called on the international community to assess whether students meet at least a minimum proficiency level in reading and mathematics at grades 2/3, at the end of primary and at the end of secondary school. Yet, half-way to the 2030 deadline for our SDG 4 goal, still only 34 countries are reporting at the end of grades 2/3, almost 100 (or around half) at the end of primary school, and 85 at the end of secondary school. Why?

As for all data used at the global level, and as discussed in the previous two blogs just posted on administrative data and household surveys, the outputs need to be comparable across countries and representative of each country. They need to be compliant with minimum standards of quality, so that there is full understanding of whether students are being compared like-for-like. Ideally, the measurement needs to serve not just the purpose of reporting; it needs to also develop national capacity to carry out learning assessment and use the results to improve curriculum, teacher education and assessments. Last but not least, countries need to report the results.

The world has come a long way in a short amount of time. Prior to 2015, when SDG 4 was set, there was no global consensus on how to define minimum proficiency. Today, not only is there an indicator but there are also standards of what should be measured at each one of three key steps of students’ learning progression and criteria of what a good measurement looks like. It is now also possible to use a variety of tools instead of only one. Already in 2018, the major cross-national studies at global (e.g., PIRLS, TIMSS) or regional level (e.g., PILNA, SEA-PLM, PASEC, LLECE, SACMEQ) agreed how the proficiency levels they were measuring mapped on to the global their proficiency levels match to the global minimum proficiency level.

Challenges

A number of challenges stand in the way of realizing these objectives and increasing country coverage.

Comparability and quality: No one country is the same. Not every country wants, and has a policy, to measure learning at each one of the three levels. Not every country aims to teach the same content in its schools as those in other countries. There are so many test formats and sampling decisions to make that a minimum set of procedural standards is needed to make sure that results are robust and comparable. The UIS has produced guidance on what background information should be provided to help assess the comparability of results.

Timeliness: The turnaround time between learning being assessed and results being reported stretches so long in some cases that the purpose of monitoring is not served, while at the same time results cannot even be used to influence policy.

Costs: The cost of assessment is low relative to the cost of not measuring learning. Assessment systems, after all, have positive impacts that go way beyond simply producing statistics. Still, assessments are relatively costly, especially for the poorest countries. This makes some countries reluctant to invest without external support. But such support tends to be short-term, fragmented, and often not taking the best interest of countries into account. Rarely does such support build institutions and develop capacity.

There is a high chance that low country coverage of the indicator on minimum proficiency in early grades, 4.1.1a, will lead to it being dropped from the list of global SDG indicators in 2025. Dropping an indicator from the global list for pragmatic reasons (i.e., because countries are not reporting) does not mean that the indicator loses its relevance: the out-of-school rate is not a global SDG indicator, yet it continues to receive global attention. But it shifts attention to the lack of a coherent long-term approach to help countries develop their capacity to assess and monitor learning in early grades of primary and beyond.

When the standards were set in 2018, it was agreed that some other assessments were not suitable but could one day be considered for reporting if they worked towards these standards. There is still potential for these assessments that currently do not meet the standards – because they were not designed to be comparable, or they measure proficiency at a level below the minimum, or governments are not prepared to report their results – to meet them in the future, even though this might be too late to increase coverage in time, given how long it takes to administer assessments.

But this is also a critical moment to reflect what approach to assessment will empower countries. Measurement should not be done for the sake of measurement. It is supposed to be a tool to help countries develop. UIS has proposed that funding of learning assessment needs to move to countries: depending on their income and capacity, they should be eligible for a funding entitlement, and it is then countries that should decide which of the measurement options that meet standards best meet their capacity objectives and are cost efficient. These are issues that will be discussed during the Conference this week in Paris.

Various tools have been used to promote comparability and ownership

Apart from promoting consensus by getting existing assessments to map their descriptions of proficiency to the global minimum proficiency level, the UIS has also promoted comparability in at least three other ways.

The first is statistical. Aptly called, Rosetta Stone, named after the famous archaeological discovery that enabled translation between different written languages, this approach harmonized data sets from two regional assessments in Latin America and francophone Africa and one international assessment to test how robust the consensus-based approach was.

The second set of tools, policy linking and pairwise comparison of items, are not statistical. They rely on a panel of national experts to judge how the national assessment aligns with the global minimum proficiency level.  Still in their piloting phase, these tools can empower countries to use their national assessments for comparable global reporting.

The third tool, Assessments for Minimum Proficiency Level, or AMPL, is a more comprehensive approach. It is a set of 20 questions that enable countries to report to the global indicator. The questions can be added to any assessment that governments are already carrying out. It is low-cost approach that has been able to produce results within months. It has been used already in nine African countries (Burkina Faso, Burundi, Côte d’Ivoire, Gambia, Kenya, Lesotho, Senegal, Sierra Leone and Zambia), has been administered in Urdu in Pakistan, and it has been piloted in India in English and in Hindi. The huge potential of this tool will be presented at the Conference this week.

Solutions

Solutions, then, fall out of the above challenges:

• Assessment harmonization and reporting handbook: With the progress made in recent years, it is time to compile and regularly update a handbook with all the information on eligibility criteria for reporting. Among the elements to be included would be the following: alignment to standards and frameworks; representativeness; administration comparability; process transparency; and participation feasibility (costs, schedules, capacity building, and overall burden for a country).
• Assessment accreditation system: Accordingly, it is also the right time to introduce a clear and transparent accreditation system. Assessment providers, including government organizations, will be able to apply to having assessments vetted for their fitness of purpose to report on SDG indicator 4.1.1. Based on the handbook, a checklist will contain the standards and eligibility criteria with which applicants need to comply.

The post Learning data: How do we measure progress towards SDG 4 – Part 3 appeared first on World Education Blog.

Historically, data to monitor progress in education have come from education institutions and ministries based on records (see our previous blog on administrative data). However, the increasing availability of household and other surveys over the past 30 years means that they have become a complementary (and, in a few cases, as in the case of equity, almost exclusive) source of data on education indicators that needs to be accommodated by national education statistics systems.

What information do household surveys contribute?

Household surveys collect information from members of households about various aspects of their lives. They collect data on school attendance, literacy, and education spending, among other pieces of information. Two major cross-national household surveys, DHS and MICS, tend to take place every 5 years and cover areas such as child development to ICT skills. Labour force surveys provide information on adult education. School surveys can generate data on school health and nutrition.

One of the major benefits of using data from household surveys is that they can be broken down by individual and household characteristics, such as socioeconomic background (e.g. sex, ethnicity, disability, and income or wealth). The WIDE database on inequalities in education shows just how useful such data can be for visualizing education gaps and for informing policy responses.

If nationally representative, household surveys can also collect information that administrative data cannot, such as self-reported skills and training taking place outside of the educational system. The more regular and comparable they are, the richer they are as a source for monitoring SDG 4, providing trends over time. This is proved by the extent to which survey data feeds into the indicators published by the UIS.

SDG 4 indicators that may be derived using household survey data, by provider

Six challenges to using household surveys and potential solutions

There are challenges to using household surveys more consistently, however, which can be reduced to six core areas. The lack of harmonization means that their findings may not be comparable to those from other countries. Comparability can also be challenged by different reference periods or incomplete coverage of education providers. The quality of the background information that enables some of the data to be broken down by factors such as wealth, migration and disability, is sometimes insufficient.  In addition, many countries do not make their data accessible, limiting cross-country analysis. In addition, outcomes, notably those related to literacy and learning, are conceptualized differently across surveys, some relying on self-reporting and others on direct measures, while they may be administered to different population groups. Similarly, while household surveys provide critical information on household spending, respondents may be reticent to share financial information. Surveys also differ in the types of expenses they capture.

Solutions

Against the challenges, there are, thankfully, solutions, many of which will be tabled at the first ever Conference on Education Data and Statistics being convened by UNESCO on 7-9 February in Paris. To maximize the potential of surveys, the Conference will present the need for a comprehensive strategy for all actors that involves collaboration, capacity building, and standardization. Within this strategy, there are responsibilities and roles at the global and national levels.

At global level

We should:

• Raise awareness among policymakers, researchers, and the public of the opportunities that surveys offer for generating education statistics and advocate for:
  • the importance of high-quality survey-based data for evidence-based decision-making; and
  • the need for continued financial and technical support for household surveys in education.
• Develop a household survey data repository with the collaboration of member states, ensuring accessibility while maintaining data security and privacy.

At national level

In their national household survey programs, countries should strive to establish standardized, modular survey instruments. They should ensure covering all major national education programs and foster more dialogue between their national statistical office, the education ministry and any technical partner developing survey questionnaires. They should, for instance:

• Ensure that the questions on education programs are aligned with the International Standard Classification of Education (ISCED).
• Ensure that survey questions related to education attendance are clearly aligned with specific school years and the reference periods for SDG 4 indicators. Record more information to calculate children’s age precisely at the beginning of the school year.
• Develop standardized definitions and measures for socioeconomic factors like household wealth, migration, and disability to enhance comparability across surveys, referring to international existing guidelines such as the Washington Group on disability questions.
• Integrate simple enumerator-assessed literacy tests alongside self-assessed measures. Expand such assessments to all youth and adults, not just those below a certain level of educational attainment.
• Ensure that questions on types of education expenditure follow global guidelines for comparability – and link education expenditure to individual students within households for more precise data.

Countries should also develop guidelines for data collection and processing to ensure consistency and comparability of education indicators.

Lastly, countries are encouraged to share survey microdata for public use.

In conclusion, household surveys offer valuable insights into education indicators, but their effective implementation and utilization face numerous challenges. Addressing these challenges is essential to ensure reliable and meaningful data for monitoring progress towards SDG 4.

Read the first blog in the series:

Administrative data: How do we measure progress towards SDG 4 – Part 1

The post Household survey data: How do we measure progress towards SDG 4 – Part 2 appeared first on World Education Blog.

This is part of a series of blogs, aiming to inform about some of the core challenges and solutions to collecting quality data which will be discussed in depth next week at the first ever Conference on Education Data and Statistics, convened by the UNESCO Institute for Statistics (UIS).

Over half of the data to report back on our education goal is administrative data collected by governments. This is data collected by line ministries and other national authorities. They are typically collected through annual school censuses, compiled in education management information systems, and used as a key resource for day-to-day operations.

These same data also feed into our global understanding of progress towards SDG 4 through the UIS, which administers an annual formal survey to countries. The UIS assures data quality and comparability including through back-and-forth discussions with Member States. These data are then complemented with other sources and published online so that they can be used in monitoring publications, such as the GEM Report, to give a good picture of how we are progressing towards our various targets.

Four challenges in compiling and producing internationally comparable data

A country’s capacity to respond to the UIS questionnaire relies entirely on data availability at the national level. Even when national data are available, they may not be sufficient or appropriate. For instance, school response rates may be low and not representative enough to use. Countries may not agree with the values of the indicators produced by the UIS. While data are regularly produced in many countries and SDG regions for most indicators, there remain significant gaps in areas, such as teacher qualifications.

Measuring target 4.c on teachers is an example. There is a 75% coverage of indicators on teacher training, but the other indicators on teachers have lower coverage: 50% coverage for the indicator on teacher attrition, 30% on professional development and less than 20% for teacher salaries relative to other professionals. These rates reflect low reporting rates by countries to the UIS survey. In 2013–17, for instance, at least two-thirds of data fields on teachers were not filled out by countries. The UIS has since tried to fill gaps by using OECD, ILO data as sources, but important gaps remain.

Figure 1. Percentage of population in countries covered with at least one data point on teacher indicators, 2018–2022

The second challenge is related to quality. For data to be comparable, and to feed into contribute to policy debate at the international level, they must all fit against predefined quality standards, such as ISCED. Currently, all SDG 4 indicators are conceptually clear, have an internationally established methodology and the standards are available.

A good quality of analysis may help fix instances where national indicator definitions may not align with global standards. For instance, differences remain between how different countries understand whether a teacher is trained or qualified, a key discussion point at the Conference next week. The UIS has assembled a comprehensive database aiming to define a global minimum standard of teacher academic qualifications needed to teach at specific levels of education, and the TCG has endorsed a Global standard for teacher’s academic qualification which should help improve global comparability of data on this issue.

The UIS has also collected information on initial teacher training programmes, through the innovative ISCED-T process, which should eventually contribute to a new global minimum standard of a trained teacher.

Some data works with ratios, such as enrolment numbers which need to be mapped against the population. Biases can arise however, when different data sources are used. For example, there may not be clarity or agreement on the population data used: countries may use national population data instead of UN Population Division data, which is the default source used by the UIS. A similar situation arises with financial indicators when national data are used instead of IMF and World Bank estimates.

Developments and solutions

A few key developments, led by the UIS and partners, have been trying to address some of the above issues:

1. Implement a new population data policy: In April 2023, the UIS introduced an important change to its population data policy, allowing countries to request that the UIS use national population data instead of UN Population Division data to increase national ownership of statistics disseminated by the UIS. The policy specifies criteria that national population data must meet to be used (such as showing a time series from 2000-2023 and having data disaggregated by sex and age). The hope is that the implementation of this policy will be gradually expand in future UIS data releases.
2. UIS dynamic template: This tool helps reduce the burden of long questionnaires on Member States. It produces international comparable indicators immediately after entering data into the template and provides historical data and indicators for each country. The template also helps countries understand in a transparent manner how indicators are calculated following the international methodologies and has, therefore, become a capacity development tool for countries as well that could be used for their national indicators and subregional disaggregation. More than 40 countries used the template to report data to the UIS in 2023, helping reduce historical data gaps and making data collection and validation processes more efficient.
3. Capacity development for better administrative data: The UIS works with countries to develop their capacity to collect and analyse data for policy development. It advises on the alignment of annual school census forms so that the right data are being collected for national, regional and global education frameworks, including whether they are compatible with international standards and flexible enough to accommodate future data needs. A new tool being launched at the Conference, LASER, highlights data gaps and capacity development needs per country.

These issues will feature during the Conference, including at dedicated sessions on administrative data and on teacher data. Three core areas for advancing the agenda will be tabled, notably:

• Expand the use of the UIS dynamic template to more countries and provide support to them.
• Develop a maturity model of an education management information system to assess and guide countries to move to advanced systems.
• Develop standard items and formats with all variables needed to estimate SDG 4 indicators.

On teacher data specifically, discuss a revision of the indicator framework:

• Implement ISCED-T to develop global standards for teacher training.
• Agree global definitions of minimum trained teacher requirements at each level.
• Consider developing policy indicators on attracting, preparing, and retaining teachers, which are not currently part of the framework

The post Administrative data: How do we measure progress towards SDG 4 – Part 1 appeared first on World Education Blog.