VT Census Case Studies : OECD - Programme for International Student Assessment (PISA)

Created by ziemer on 12 Aug 2015

Brief Overall Description of the Dataset:

“The Programme for International Student Assessment (PISA) is a triennial international survey which aims to evaluate education systems worldwide by testing the skills and knowledge of 15-year-old students. To date, students representing more than 70 economies have participated in the assessment.”

Link: www.oecd.org/pisa/pisaproducts/

Date Inventory Completed: 6/18/15

Screening

  • Is the data collected opinion-based?
  • Is the data collection recurring (must be collected at least annually)?
  • Is there data available for 2013?
  • For Education: Is the data collected at least the school level? 
  • Can we access the data by August 15th?
  • Can the data be linked to other education/workforce datasets (e.g., K-12, higher education, workforce)?

 

Purpose

  • What is the purpose of the organization collecting the data?

The Organization for Economic Co-operation and Development (OECD) aims to “promote policies that will improve the economic and social well-being of people around the world.” They work with governments to understand what drives economic, social and environmental change.

  • Why is it collected and how does the organization use it?

PISA is used to evaluate education systems worldwide.

  • Who else uses the data?

Governments and countries

  • Who do they sell the data to?

NA


Method

  • What is the data collection method? 

 

Paper-and-pencil and Computerized assessment

 

  • What is the type of data collected? 

Assessment

  • If designed, who created the questions?

All PISA participating countries are invited to submit questions to the international contractors; in addition, the international contractors write some questions. The questions are reviewed by the international contractors and by participants and are carefully checked for cultural bias.

  • What is the raw source of the collected data (prior to any aggregation)? 

Students and school principals

Description

 

  • What is the general topic of the data (1-2 words)?

Student assessment

 

  • What are the earliest and latest dates for which data is available?

2000-2012

 

  • Is data collected and available periodically?

Yes the test is given every three years

 

  • How soon after a reference period ends can a data source be prepared and provided? 

Unknown

Selectivity

  • What is the universe (e.g., population) that the data represents?

To date, students representing more than 70 economies have participated in the assessment. Around 510,000 students in 65 economies took part in the PISA 2012 assessment of reading, mathematics and science representing about 28 million 15-year-olds globally. Of those economies, 44 took part in an assessment of creative problem solving and 18 in an assessment of financial literacy.

Accessibility

  • How is the data accessed? 

Download SAS and SPSS files

  • Is it open data?

Yes

  • Any legal, regulatory, or administrative restrictions on accessing the data source?

No

  • Cost? - One time or annual or project based payment?

NA

Does this dataset appear to meet our needs for the Census study? NO

Explanation: Only national level is available


Full Inventory

Description

  • What is the general contents of the data source?

Student assessment

  • Features
    • What is the temporal nature of the data: longitudinal, time-series, or one time point?

Time Series

    • Geospatial? If Yes, at what level?

Country

Metadata

  • Is there information available to assess the transparency and soundness of the methods to gather the data for our purposes?

Yes

  • Is there a description of each variable in the source along with their valid values?

Yes

  • Are there unique IDs for unique elements that can be used for linking data?

Yes

  • Is there a data dictionary or codebook?

Yes - https://www.dropbox.com/s/waxjyo5qi9eiy24/Notes%20on%20log%20file%20data%20-%20releaseditems.pdf?dl=0

Selectivity

  • What unit is represented at the record level of the data source?

Student

  • Does this universe match the stated intentions for the data collection? If not, what has been included or excluded and why?

Yes

  • What is the sampling technique used (if applicable)? 

Countries elect to be a part of PISA. The PISA Governing Board then approves membership according to certain criteria. Participants must have the technical expertise necessary to administer an international assessment and must be able to meet the full costs of participation. Schools in each country are randomly selected by the international contractor for participation in PISA. At these schools, the test is given to students who are between age 15 years 3 months and age 16 years 2 months at the time of the test, rather than to students in a specific year of school. This average age of 15 was chosen because at this age young people in most OECD countries are nearing the end of compulsory education. The selection of schools and students is kept as inclusive as possible, so that the sample of students comes from a broad range of backgrounds and abilities.

  • What was the coverage?

Unknown - assumed to be 100%

Stability/Coherence

  • Were there any changes to the universe of data being captured (including geographical areas covered) and if so what were they?

Yes, different countries elect to be a part of PISA each cycle

  • Were there any changes in the data capture method and if so what were they? 

Some countries started using computerized assessment as opposed to paper-and-pencil.

  • Were there any changes in the sources of data and if so what were they? 

Unknown

Accuracy

  • Any known sources of error?

Unknown

  • Describe any quality control checks performed by the data’s owner.

Tests are checked for missing responses, reliability, dimensionality, test targeting, etc.

 

Accessibility

  • Any records or fields collected, but not included in data source, such as for confidentiality reasons)? 

Yes, there is a list of cases in which the OECD, on the basis of technical advice from the PISA Consortium, removed or annotated national data in the report because of technical anomalies or because the data did not meet the OECD technical standards for PISA.

  • Is there a subset of variables and/or data that is must be obtained through a separate process? If yes, is there a separate legal, regulatory, or administrative restrictions on accessing the data source? Cost? - One time or annual or project based payment?

No

Privacy and security

  • Was consent given by participant? If so, how was consent given?

Unknown

  • Are there legal limitations or restrictions on the use of the data? 

No

  • What confidentiality policies does the source have? 

NA

 

Research

  • What research has been done with this dataset? (e.g., impact of policies, predictors of student success)

Reading and Math performance by country, school characteristics that lead to optimal student performance, student learning

  • Include any links to research if provided:

http://www.oecd.org/pisa/pisaproducts/pisain-depthreports.htm

  • List any other data use notes provided by the supplier.

NA