The demand for large-scale assessments in higher education, especially at an international scale, is growing. A major challenge of conducting these assessments, however, is that they require understanding and balancing the interests of multiple stakeholders (government officials, university administrators, and students) and also overcoming potential unwillingness of these stakeholders to participate. In this paper, we take the experience of the Study of Undergraduate Performance (SUPER) in conducting a large-scale international assessment as a case study. We discuss ways in which we mitigated perceived risks, built trust, and provided incentives to ensure the successful engagement of stakeholders during the study’s implementation.

We examine the effects of computer-based versus paper-based assessment of critical thinking skills, adapted from English (in the U.S.) to Chinese. Using data collected based on a random assignment between the two modes in multiple Chinese colleges, we investigate mode effects from multiple perspectives: mean scores, measurement precision, item functioning (i.e. item difficulty and discrimination), response behavior (i.e. test completion and item omission), and user perceptions. Our findings shed light on assessment and item properties that could be the sources of mode effects. At the test level, we find that the computer-based test is more difficult and more speeded than the paper-based test. We speculate that these differences are attributable to the test’s structure, its high demands on reading, and test-taking flexibility afforded under the paper testing mode. Item-level evaluation allows us to identify item characteristics that are prone to mode effects, including targeted cognitive skill, response type, and the amount of adaptation between modes. Implications for test design are discussed, and actionable design suggestions are offered with the goal of minimizing mode effect.

The wide-scale global movement of school education to remote instruction due to Covid-19 is unprecedented. The use of educational technology (EdTech) offers an alternative to in-person learning and reinforces social distancing, but there is limited evidence on whether and how EdTech affects academic outcomes. Recently, we conducted two large-scale randomized experiments, involving ~10,000 primary school students in China and Russia, to evaluate the effectiveness of EdTech as a substitute for traditional schooling. In China, we examined whether EdTech improves academic outcomes relative to paper-and-pencil workbook exercises of identical content. We found that EdTech was a perfect substitute for traditional learning. In Russia, we further explored how much EdTech can substitute for traditional learning. We found that EdTech substitutes only to a limited extent. The findings from these large-scale trials indicate that we need to be careful about using EdTech as a full-scale substitute for the traditional instruction received by schoolchildren.

The wide-scale global movement of school education to remote instruction due to Covid-19 is unprecedented. The use of educational technology (EdTech) offers an alternative to in-person learning and reinforces social distancing, but there is limited evidence on whether and how EdTech affects academic outcomes, and that limited evidence is mixed.1,2 For example, previous studies examine performance of students in online courses and generally find that they do not perform as well as in traditional courses. On the other hand, recent large-scale evaluations of supplemental computer-assisted learning programs show large positive effects on test scores. One concern, however, is that EdTech is often evaluated as a supplemental after-school program instead of as a direct substitute for traditional learning. Supplemental programs inherently have an advantage in that provide more time learning material.

Recently, we conducted two large-scale randomized experiments, involving ~10,000 primary school students in China and Russia, to evaluate the effectiveness of EdTech as a substitute for traditional schooling.3,4 In both, we focused on whether and how EdTech can substitute for in-person instruction (being careful to control for time on task). In China, we examined whether EdTech improves academic outcomes relative to paper-and-pencil workbook exercises of identical content. We followed students ages 9–13 for several months over the academic year. When we examined the impacts of each supplemental program we found that EdTech and workbook exercise sessions of equal time and content outside of school hours had the same effect on standardized math test scores and grades in math classes. As such, EdTech appeared to be a perfect substitute for traditional learning.

In Russia, we built on these findings by further exploring how much EdTech can substitute for traditional learning. We examined whether providing students ages 9–11 with no EdTech, a base level of EdTech (~45 min per week), and a doubling of that level of EdTech can improve standardized test scores and grades. We found that EdTech can substitute for traditional learning only to a limited extent. There is a diminishing marginal rate of substitution for traditional learning from doubling the amount of EdTech use (that is, when we double the amount of EdTech used we do not find that test scores performance doubles). We find that additional time on EdTech even decreases schoolchildren’s motivation and engagement in subject material.

The findings from the large-scale trials indicate that we need to be careful about using EdTech as a full-scale substitute for the traditional instruction received by schoolchildren. There are two general takeaways: First, to a certain extent, EdTech can successfully substitute for traditional learning. Second, there are limits on how much EdTech may be beneficial. Admittedly, we need to be careful about extrapolating from the smaller amount of technology substitution in our experiments to the full-scale substitution in the face of the coronavirus pandemic. However, these studies may offer important lessons. For example, a balanced approach to learning in which schoolchildren intermingle work on electronic devices and work with traditional materials might be optimal. Schools could mail workbooks to students or recommend that students print out exercises to break up the amount of continuous time schoolchildren spend on devices. This might keep students engaged throughout the day and avoid problems associated with removing the structure of classroom schedules. Schools and families can devise creative remote learning solutions that include a combination of EdTech and more traditional forms of learning. Activities such as reading books, running at-home experiments, and art projects can also be used to break up extensive use of technology in remote instruction.

In some accountability regimes, teachers pay more attention to higher achieving students at the expense of lower achieving students. The overall goal of this study is to examine, in this type of accountability regime, the impacts of a pay-for-percentile type scheme in which incentives exist for all students but which are larger for improving the achievement of lower achieving students. Analyzing data from a large-scale randomized experiment in rural China, we find that incentives improve average achievement by 0.10 SDs and the achievement of low-achieving students by 0.15 SDs. We find parallel changes in teacher behavior and curricular coverage. Taken together, the results demonstrate that incentive schemes can effectively address teacher neglect of low-achieving students.

Purpose: Although China has instituted compulsory education through Grade 9, it is still unclear whether students are, in fact, staying in school. In this paper, the authors use a multi-year (2003–2011) longitudinal survey data set on rural households in 102–130 villages across 30 provinces in China to examine the extent to which students still drop out of school prior to finishing compulsory education.

Design/methodology/approach: To examine the correlates of dropping out, the study uses ordinary least squares and multivariate probit models.

Findings: Dropout rate from junior high school was still high (14%) in 2011, even though it fell across the study period. There was heterogeneity in the measured dropout rate. There was great variation among different regions, and especially among different villages. In all, 10% of the sample villages showed extremely high rates during the study period and actually rose over time. Household characteristics associated with poverty and the opportunity cost of staying in school were significantly and negatively correlated with the completion of nine years of schooling.

Research limitations/implications: The findings of this study suggest that China needs to take additional steps to overcome the barriers keeping children from completing nine years of schooling if they hope to either achieve their goal of having all children complete nine years of school or extend compulsory schooling to the end of twelfth grade.

Originality/value: The authors seek to measure the prevalence of both compulsory education rates of dropouts and rates of completion in China. The study examines the correlates of dropping out at the lower secondary schooling level as a way of understanding what types of students (from what types of villages) are not complying with national schooling regulations. To overcome the methodological shortcomings of previous research on dropout in China, the study uses a nationally representative, longitudinal data set based on household surveys collected between 2003 and 2011.

Book Synopsis:

As the glittering skyline in Shanghai seemingly attests, China has quickly transformed itself from a place of stark poverty into a modern, urban, technologically savvy economic powerhouse. But as Scott Rozelle and Natalie Hell show in Invisible China, the truth is much more complicated and might be a serious cause for concern.

China’s growth has relied heavily on unskilled labor. Most of the workers who have fueled the country’s rise come from rural villages and have never been to high school. While this national growth strategy has been effective for three decades, the unskilled wage rate is finally rising, inducing companies inside China to automate at an unprecedented rate and triggering an exodus of companies seeking cheaper labor in other countries. Ten years ago, almost every product for sale in an American Walmart was made in China. Today, that is no longer the case. With the changing demand for labor, China seems to have no good back-up plan. For all of its investment in physical infrastructure, for decades China failed to invest enough in its people. Recent progress may come too late. Drawing on extensive surveys on the ground in China, Rozelle and Hell reveal that while China may be the second-largest economy in the world, its labor force has one of the lowest levels of education of any comparable country. Over half of China’s population—as well as a vast majority of its children—are from rural areas. Their low levels of basic education may leave many unable to find work in the formal workplace as China’s economy changes and manufacturing jobs move elsewhere.

In Invisible China, Rozelle and Hell speak not only to an urgent humanitarian concern but also a potential economic crisis that could upend economies and foreign relations around the globe. If too many are left structurally unemployable, the implications both inside and outside of China could be serious. Understanding the situation in China today is essential if we are to avoid a potential crisis of international proportions. This book is an urgent and timely call to action that should be read by economists, policymakers, the business community, and general readers alike.

Meet the Authors:

Scott Rozelle: Scott is the Helen F. Farnsworth Senior Fellow and the co-director of the Rural Education Action Program in the Freeman Spogli Institute for International Studies at Stanford University. His research focuses almost exclusively on China and is concerned with: agricultural policy, including the supply, demand, and trade in agricultural projects; the emergence and evolution of markets and other economic institutions in the transition process and their implications for equity and efficiency; and the economics of poverty and inequality, with an emphasis on rural education, health and nutrition. He has been published in top academic journals and has authored over 500 articles, chapters, and books. 

 

Natalie Hell: Natalie is a writer based in the San Francisco Bay Area. As part of Stanford University’s Rural Education Action Program, she has worked on Chinese education and health issues for seven years. This work has included managing numerous large-scale field studies and conducting extensive qualitative interviews across rural China. She holds a bachelor’s degree in Chinese language and political science from Williams College, and is a fluent Mandarin speaker.

---

A growing body of literature has documented that community-based early childhood development (ECD) interventions can improve child developmental outcomes in vulnerable communities. One critical element of effective community-based programs is consistent program participation. However, little is known about participation in community-based ECD interventions or factors that may affect participation. This paper examines factors linked to program participation within a community-based ECD program serving 819 infants and their caregivers in 50 rural villages in northwestern China. The results find that more than half of families did not regularly attend the ECD program. Both village-level social ties within the program and proximity to the program significantly predict program participation. Increased distance from the program site is linked with decreased individual program participation, while the number of social ties is positively correlated with participation. The average program participation rates among a family’s social ties is also positively correlated with individual participation, indicating strong peer effects. Taken together, our findings suggest that attention should be given to promoting social interactions and reducing geographic barriers among households in order to raise participation in community-based ECD programs.

The effectiveness of educational technology (EdTech) in improving the outcomes of poor, marginalized students has primarily been documented by studies conducted in developing countries; however, relevant research involving randomized studies in developed country contexts is relatively scarce. The objective of the current study is to examine whether an in-school computer assisted learning (CAL) intervention can improve the math performance (the primary outcome) and academic attitudes (secondary outcomes) of rural students in Taiwan, including a marginalized subgroup of rural students called Xinzhumin. We also seek to identify which factors are associated with the effectiveness of the intervention. In order to achieve this, we conducted a randomized control trial involving 1,840 sixth-grade students at 95 schools in four relatively poor counties and municipalities of Taiwan during the spring semester of 2019. According to the ITT analysis, the O-CAL intervention had no significant ITT impacts on the primary outcome of student math performance as well as on most secondary outcomes of the overall treatment group (who on average used the software for only about one quarter of the protocol’s minimum required time of 30 minutes per week, indicating that compliance was low). However, the LATE analysis revealed significant improvements in the math performance of the 30% most active students in the treatment group (who used the software for about two thirds of the minimum required time). Effect sizes of active users overall (0.16 SD-0.22 SD) increased in accordance with increases in usage and were larger for active Xinzhumin users specifically (0.21 SD-0.35 SD). A wide range of student-level and (in particular) teacher-level characteristics were associated with the low compliance to the intervention, which are findings that may help inform educational policymakers and administrators of the potential challenges of introducing school-based interventions that depend heavily on teacher adoption and integration.  

In response to the COVID-19 epidemic, many education systems have relied on distance learning and educational technologies to an unprecedented degree. However, rigorous empirical research on the impacts on learning under these conditions is still scarce. We present the first large-scale, quantitative evidence detailing how school closures affected education in China. The data set includes households and teachers of 4,360 rural and urban primary school students. We find that although the majority of students engaged in distance education, many households encountered difficulties including barriers to learning (such as access to appropriate digital devices and study spaces), curricular delays, and costs to parents equivalent to about two months of income. We also find significant disparities across rural and urban households.