Assessing and Credentialing Vocational High Schools

 Problem

Vocational high schools in China serve as an alternative to academic high school and typically provide two years of in-the-classroom training and a one-year, school-supervised, on-the-job internship. Over the past two decades, vocational high schools have been expanding rapidly. Between 1990 and 2011, annual investment in secondary vocational schooling increased six-fold and enrollment increased by 14.2 million. However, these schools vary widely in quality, and some schools may employ practices that are not beneficial to their students.

Why do some vocational schools fail to provide fair, safe, or human capital-enhancing experiences? Part of the problem is due to a lack of coordination and organization within China’s vocational schooling system, which is characterized by decentralized decision-making and multiple channels of authority. There are no fewer than four different streams of vocational education programs, run through two entirely separate and often competing ministries. In such a complex and poorly coordinated system, schools are rarely monitored or held accountable for providing quality training. Schools further lack information to match students into internship opportunities that fit their skill levels and interest. After schools match students into internships, they often fail to provide students with adequate support and supervision.

In addition, when large sums of money and poor monitoring are combined, there is a chance for opportunistic behavior. Perhaps unsurprisingly, not all vocational schools have carried out policy plans and educational initiatives in ways consistent with the objective of increasing student learning. Ultimately, while there may be high quality vocational schools that provide students with good educational experiences—both in the areas of basic education (math, science, language, English, and information and communications technology skills) and in technical skills (their major)—there are many schools that do not.

Unfortunately, no standard assessment procedures exist to delineate low quality from high quality schools. While there are clear and long-standing assessment procedures for every other level of education in China (primary school, junior high school, academic high school and college), vocational school quality remains poorly monitored.

Phase 1

Objective

All other levels of the Chinese education system have rigorous assessment and credentialing systems in place to hold schools accountable to clear quality standards.

The goal of Phase 1 of this project was to assess which vocational high schools are offering students a “high quality” education and which are falling short. In order to do so, REAP collected and analyzed data on school resources, teacher qualifications, curricula, student achievement levels, value-added gains in basic and technical skills, and dropout rates. We also compared the high quality schools with schools identified by the provincial Department of Education as “key schools.”

Approach

In order to meet this objective, REAP used a simple but proven approach: a paired baseline and endline assessment. After randomly selecting 180 vocational programs in one central province from comprehensive listings of vocational institutions run by the Department of Education and Department of Human Resources, we surveyed and assessed two cohorts of students. We focused on standardized metrics such as tests of academic abilities and practical skills; school resources, including teacher qualifications, finances, facilities, and equipment; school curricula; and dropout rates collected by following a cohort of students (instead of relying on administrative data). From this data, REAP developed a list of top quality vocational programs.

Phase 2

Objective

REAP carried out a paired baseline and endline assessment to develop a list of top quality vocational programs in one central province.

Assessment is useful for understanding the quality of individual schools. However, on its own it is powerless to actually motivate change.

In every other level of Chinese education, including primary schools, junior high schools, academic high schools, and universities, there are not only rigorous assessment tools, but also rigorous credentialing systems in place to both hold schools accountable to clear quality standards. Credentialing systems motivate change because they tie assessments of quality to tangible benefits. Schools that are able to receive credentials through this sort of program are rewarded for their high quality. Schools that fail evaluations in a credentialing system have substantial motivation to improve. Credentialing systems not only provide incentives to schools for improved performance, they also provide information to government, industry and the public at large about which schools are of high quality. This allows all actors engaging with vocational schools to make better, more informed choices. The objective of Phase 2 of this project is to evaluate whether a pilot credentialing system can effectively raise the quality of vocational schools.

Approach

To reach this goal, we are conducting a Randomized Controlled Trial (RCT) among our sample of vocational programs. We first randomly selected half of the programs in our Phase 1 sample to be our “treatment” group. These schools will be enrolled in a credentialing system. The remaining half of our sample programs were designated the “control” group. These schools will proceed with “business as usual” with no intervention from our research team.

The provincial Department of Education invited principals from the “treatment” programs to a central meeting. During the meeting, the treatment principals were informed that they had been enrolled in a new pilot credentialing system. Under this credentialing system, all treatment schools will be evaluated using the assessment tools REAP developed in Phase 1 of this project. Schools that meet a set of criteria (contributing to student learning, maintaining low rates of school dropout, and meeting legal standards for internship behavior) will have increased opportunities for government funding and industry collaboration. 

At the end of the school year, REAP will return to all the programs in our original sample and assess them, using the same measures employed in the Phase 1 assessment. We will use the endline measures to evaluate whether the credentialing system (administered only to treatment programs) led to measurable increases in school quality. By comparing changes in the performance of treatment and control programs, we will be able to determine whether a credentialing system effectively increases school quality.

Results

Are schools providing high quality education? Phase 1 results:

REAP is using a Randomized Controlled Trial (RCT) to evaluate whether a credentialing system can effectively motivate positive change in rural vocational schools.

Average learning gains were low overall, though highly variable.

• In both general and specific skills, average learning gains were not significantly significant from zero. In other words, the students in our sample vocational high schools are not learning any general or specific skills.
• There is a wide variation between the best performing (top 10%) and worst performing (bottom 10%) schools. This indicates that some schools are successful in teaching their students general and specific skills, while others are not.

Dropout rates are highly variable between vocational schools.

• The dropout rate among the top 10% of schools was only 1%. In contrast, the dropout rate among the bottom 10% of schools was 66%. 
• The average dropout rate across all schools was 16.3% among Grade 1 students and 6.2% among Grade 2 students.

The majority of student internships are not compliant with legal standards.

• 68% of students with internship experience reported that their internship had no relation to their major.
• 39% of students with internship experience had no teacher accompany them on their internship.
• 7% of students with internship experience had been sent to internships before age 16, the legal internship age.

• Students in the key schools rank in the 41st and 47th percentile in their general and specific skills, respectively. In contrast, students in non-key schools rank at the 59th and 56th percentiles.
• There is no significant benefit to students from attending a key school, as measured by student learning, non-cognitive skills, or self-reported satisfaction with schooling.