MATHia Data Science Challenge FAQs

Yes! The winning team will receive $3,000 (to be split across all team members) with the first runner up receiving $1,000 (to be split across all team members).

You will be working with log data (aka process data or event stream data) for this competition. Each line of the data represents an action a student takes within MATHia (e.g., requesting a hint, entering an answer, clicking to advance to the next page, etc.). Specifically, the following variables are included for each student action:

• Student ID
• School ID
• Class ID
• Session ID
• Time stamp
• Workspace name
• Problem name
• Step name
• Action taken (attempt, hint request, etc.)
• Input (i.e., the answer a student gave)
• Outcome (did they get the answer correct)
• Workspace progress (did they complete the workspace)

Each individual student has about 30,000 rows of data for the workspace used for this competition. We anticipate the final dataset participants will use to exceed 1,000,000 rows of data. Teams will receive training, validation, and, at the end of the competition, test datasets.

Test sets will be shared with participants late in the competition without outcome variables. Teams will run their model and submit their predictions to the judging panel. The judging panel will then evaluate the performance metrics of these submissions.

Final submissions will be evaluated by a panel of judges. Judges will be instructed to rate the predictive performance of the model (80% of the final score) as well as the relevance of the model to the competition goals (20% of the final score). See the question below for more specifics on how we will measure overall performance. For the relevance score, our team of judges will examine the 1-2 page summary teams will submit where they will explain how this model might be used by Carnegie Learning, educators, or students to facilitate better math instruction, evaluation, or learning. A specific rubric for this 1-2 page summary and the presentation will be provided to teams.

Teams will be asked to build a model that can predict three different outcomes: one is continuous and two are ordinal. For the continuous outcome (student learning), we will use Root Mean Squared Error. (RMSE) and R-squared as the primary metrics for evaluation as well as Mean Absolute Error (MAE) as a secondary metric. For the ordinal outcomes (motivational and metacognitive outcomes), we will use Quadratic Weighted Kappa (QWK) to evaluate these outcomes. A specific rubric with a breakdown of how these three scores contribute to the overall score will be provided to teams.

Teams will be asked to share their model and work with Carnegie Learning at the final presentation, and teams will have the opportunity to publish their model in partnership with the research team.