Real requests. Imperfect data. Decisions you can defend.
Investigate how New York City service requests move toward recorded closure. Build a probability model, test it on later data, and explain what the evidence can tell us.
The timeline below is proposed. Canvas will confirm deadlines, dataset releases, team assignments, and submission links. The public practice tools on this page are available now.
01 / The question
What can a service request tell us?
NYC 311 connects residents with non-emergency services and information. Its public service-request records cover concerns such as noise, sanitation, housing, and streets. Different agencies and complaint categories follow different processes. Missing values, changing categories, and uneven workloads are part of the problem.
For each request, estimate the probability that it lacks a recorded closure timestamp within seven calendar days of creation, using only the approved features in the frozen course dataset.
Return one p_slow_7d probability between 0 and 1 for every supplied row_id. Then assess whether these estimates could support aggregate workload monitoring, and where they might fail.
Understand the measurement.
Recorded closure is an administrative event. It does not prove that a resident's problem was solved. Seven days is our course horizon, not a universal city service deadline. Historical fields may have changed after intake, so this is a retrospective benchmark.
What you will practice
01
Investigate
Audit data, ask useful questions, and make informative figures.
02
Experiment
Compare baselines and models using a credible temporal split.
03
Explain
Connect results to a decision, with evidence and limitations.
A data-forensics exercise and a small labeled onboarding sample. Learn the schema and investigate data-quality problems.
Main release
Build & validate
Labeled training and visible development data. Use these to fit models, diagnose errors, and compare approaches.
Staff evaluation
Generalize
Your submitted pipeline runs on unseen requests. Evaluation records and labels remain with the teaching staff.
How the label works
0
Recorded closure is at or before the seven-day boundary.
1
Recorded closure is later, or the closure timestamp is missing for a sufficiently mature record.
Staff require 14 days of maturity: the seven-day outcome window plus a reporting buffer. Immature records and invalid required timestamps are excluded. The course uses New York local wall-clock values and seven calendar days; labels are frozen with the release. The buffer does not eliminate every later data revision.
Illustration 01 · Synthetic requests
The outcome window is not the maturity window.
All three requests are checked on day 21, so all are mature. Filled dots mark recorded closures; the open dot means no closure is recorded by the snapshot. The seven-day boundary decides the label. The extra reporting buffer does not turn the target into “closure within 14 days.” Closure exactly at day 7 counts as label 0.
Approved feature families
Creation day, hour, and derived calendar features
Agency and problem
Location type, borough, and community board
Submission channel
The released data dictionary is the final feature contract. Version 1 omits problem_detail. row_id is a join key, never a predictor.
Keep outcome information out
Closure timestamps and current status
Due dates, resolution text, and update timestamps
Original IDs, addresses, and exact coordinates
External outcomes or lookup tables
Outcome fields may appear in the separate forensics exercise to teach auditing. That does not make them legal model inputs.
Worked example · Entirely synthetic
Follow one request from row to prediction.
Imagine a street-condition request with the attributes below. This invented example follows the version-1 schema; it is not a real resident's request, a course data row, or a hidden test record. Its timestamps and prediction are illustrative too.
1. Read the model inputs
One row of train_features.parquet, displayed vertically for readability. The file has these nine columns.
Column
Example value
How to read it
row_id
00000000000000000000000000000001
32-character course join key. Keep it as text, including leading zeros; never use it as a predictor.
created_day
2025-10-01
New York local creation date. You may derive calendar features such as day of week.
created_hour
10
The 10 a.m. hour. Source minutes and seconds are not supplied as features.
agency
DOT
Agency category as represented in the snapshot.
problem
Street Condition
Problem category, not a free-text description of the outcome.
location_type
missing
A Parquet null, not the literal word “missing.” Handle it explicitly; do not invent a location.
borough
BROOKLYN
A coarse location category, not an address.
community_board
01 BROOKLYN
Categorical text. Do not treat the “01” as a continuous numeric measurement.
channel
ONLINE
The request's submission channel.
2. Understand how staff assign the label
The following fictional staff-side timestamps are shown only to explain the target. They are not columns in the modeling feature file and must not be added to your predictors. All times below are New York local wall-clock values.
Created: October 1, 2025, at 10:30. Staff use the full source timestamp to calculate the label, before reducing the released features to day and hour.
Seven-day boundary: October 8 at 10:30. A recorded closure at or before this moment would give label 0.
Recorded closure: October 10 at 09:00. This is after the boundary, so the request did not have a recorded closure within seven days.
Example snapshot: October 20 at 12:00. The request is more than 14 days old, so it meets the maturity rule.
Result: slow_7d = 1. The later closure does not turn this label into 0. If the closure timestamp were still missing at this mature snapshot, the label would also be 1. Neither case proves when the resident's underlying problem was actually solved.
3. Connect features, label, and prediction
For training and visible development, the label is in a separate CSV. Join it to the feature row by row_id, not by row position. The target slow_7d is used to learn and evaluate; it is never an input feature. The smaller onboarding CSV combines features and label for convenience.
Illustration 02 · Synthetic prediction and score
How a separate held-out request is evaluated.
First, fit: training features + training labels → learned preprocessing and model.
01 / InputHeld-out featuresApproved fields only. No label for this row.
02 / PredictFitted pipelineApply what was learned from training data.
03 / OutputProbability 0.70Return row_id and p_slow_7d.
04 / ScoreCompare with label 1The scorer joins the label; this row's loss is 0.09.
Labels help fit the model; they do not accompany a row being predicted. For your honest development evaluation, fit on training and score on development. For hidden evaluation, staff refit on the allowed training + development data and keep hidden labels in a separate scorer, outside the submitted program.
Suppose a model fitted on training data returns p_slow_7d = 0.70 for this request. It estimates a 70% probability of lacking recorded closure within seven days—not a 70% chance that the problem is still physically unresolved, and not a prediction of how many days closure will take. The 0.70 is made up for this walkthrough, not calculated from these timestamps or produced by a trained course model.
This row's squared error:(0.70 − 1)² = 0.09. The dataset's Brier score is the average of this quantity across all evaluated rows. This single-row calculation illustrates the formula, not a valid model comparison: compare models on held-out development predictions, not their training scores. Submit probabilities, not thresholded 0/1 class decisions. In hidden evaluation, staff run your pipeline on approved features without giving it the corresponding labels, then score its outputs separately.
03 / Your milestones
One evolving project, six checkpoints.
The challenge spans Weeks 2–11, beginning with the onboarding release in Week 2 and finishing on November 13. Each phase builds on the previous one, with a lighter period around the midterm. All dates below are proposed for Fall 2026; the published Canvas assignments control deadlines and late-work rules.
Build on your work; do not start six separate reports.
After the individual forensics exercise, maintain one evolving team notebook and codebase. Revise earlier sections, reuse useful figures, and preserve your experiment history. One coordinator submits the shared work; each student submits their own assigned evidence with their team ID. Use the filenames and directory structure supplied in each CodeBench release. Unless a milestone says otherwise, tables and explanations belong inside the notebook, not in additional reports.
Sep 18 Week 3Data forensics Individual5 points
Learn to establish whether data are usable before trusting an analysis. Work independently on the small, separate forensics exercise, which includes disclosed staff-injected defects. This milestone uses Python and data-processing skills taught in the opening weeks; no model training is required.
What to do
Inspect the schema, row counts, missing values, duplicate records, and timestamp parsing. Distinguish an invalid value from a legitimate missing value.
Apply a justified repair or flag/exclude an unusable exercise row. Preserve the original input and show how your actions change the data.
Write and rerun assertions for the key rules you checked. Use the supplied worked examples to explain the seven-day boundary and maturity rule.
What to submit · Individual
Your completed forensics notebook with code, visible outputs, and a concise issue table: issue → evidence/affected rows → action and reason → verification check. Include a short before/after summary and note anything you cannot resolve safely.
What good work shows: another reader can trace each cleaning decision to evidence and rerun your checks. Do not invent missing outcomes, treat every missing closure as an error, or modify the frozen modeling files. Exercise-only outcome fields are not approved predictors.
Oct 2 Week 5Framing & honest EDA Team 3 + individual 25 points
Turn the main data release into a clearly framed investigation. Ask what the sample represents, which patterns matter for aggregate workload monitoring, and what the records cannot establish.
What to do
Read the data dictionary and manifest. Record the release version, time coverage, row counts, missingness, and important category imbalances. Join features and labels by row_id, not file position.
Create three decision-relevant figures. Possible questions include how request mix changes over time, how recorded non-closure rates vary across agency/problem groups, or where missing data limit comparisons. These are examples, not three additional required analyses.
State the intended user and decision, the sampled population, and key assumptions. Distinguish request counts from rates and administrative closure from actual resolution.
What to submit
Team · 3 points: the evolving notebook's framing and audit sections, plus the three figures and their generating code. Each figure needs readable labels, the relevant period and denominator/sample count, a takeaway, and a caveat.
Individual · 2 points: a concise critique of one team figure or associated claim. Identify a specific weakness, explain why it matters, and propose an improvement. A new fourth figure or separate report is not required.
What good work shows: the figures answer stated questions, and the conclusions stay within the sample's limits. A difference between groups is not, by itself, evidence of service quality or a causal effect.
Oct 16 Week 7Baselines & validation Team 3 + individual 36 points
Establish a trustworthy comparison before trying more flexible models. A simple probability estimate with an honest evaluation is more informative than an unexplained score.
What to do
Run and explain the two required baselines: the global training-label rate and a smoothed agency/problem rate. Explain how smoothing reduces unstable estimates for small groups and how the supplied baseline handles unseen categories.
Fit using training data and compare both methods on the same visible development rows using Brier score. Check that each input ID receives one valid probability; a constant probability is allowed.
Write your temporal validation plan before leaderboard feedback: date boundaries and counts, what is fitted on training, what development is used to choose, and how you prevent leakage. Record any earlier within-training temporal split and preserve later plan changes.
What to submit
Team · 3 points: runnable baseline code, a notebook comparison table with both development scores and evaluation counts, and the validation plan. Use the provided execution interface and explain the grouped baseline's fallback behavior.
Individual · 3 points: complete the short supervised written interpretation check during the announced session. Explain a supplied score comparison, split, or leakage example in your own words. There is no presentation or oral defense; exact prompts and collection instructions come through Canvas.
What good work shows: scores are comparable, the split reflects prediction on later requests, and no development labels were used to fit the models being evaluated. Preserve these results even though staff later refit the selected method on training plus development for official inference.
Weeks 8–9: time for the midterm. No graded challenge deadline.
Nov 6 Week 10Modeling & clean-run check Team 5 + individual 16 points
Test whether a taught model adds useful information, and make sure the analysis works outside your current notebook session. Apply logistic regression once it has been taught; additional model families are optional.
What to do
Compare logistic regression with both rate baselines under the same validation plan. Fit preprocessing on training data only and handle missing or unseen categories consistently.
Run one controlled feature comparison: add or remove a feature family while keeping the split and other settings fixed. State the hypothesis, record the result, and decide what to retain. Investigate a consequential error slice with its sample count.
Run the supplied command and preflight checks from a fresh process using visible data. Record the command, environment, result, and any unresolved problem. Begin the individually owned stress test you will interpret in Week 11.
What to submit
Team · 5 points: the runnable pipeline and supporting source, updated notebook with comparison and error-analysis results, and experiments.csv. Log the hypothesis, change, evaluation split, score/result, and resulting decision; put clean-run evidence in the notebook.
Individual · 1 point: identify one experiment you owned, point to its code/results, and explain what you tested, found, and decided. This may be an early version of your final stress test.
What good work shows: the comparison isolates a change and the submitted code reproduces the evidence. A model that does not improve can still support a strong conclusion. The formative clean-run and contribution check are included here; there is no separate rehearsal deadline or report.
Week 11 is for focused revisions, interpretation, and packaging. Use the preflight results and staff feedback to correct problems; no new model family is required.
Nov 13 Week 11Final pipeline & findings Team 6 + individual 28 points
Revise, select, and freeze the work you have already developed. Use Week-10 feedback to correct problems and explain your findings; no new model family or open-ended model search is required.
What to do
Select the final method using your documented evidence. Preserve the original train-only/development evaluation and rerun the final package's preflight checks. Staff will execute your code on unseen records; you do not download or analyze the hidden sets.
Include a development calibration view (predicted probabilities versus observed frequencies) and error slices with sample counts. Explain uncertainty and why a small or differently composed group may not support a strong comparison.
Make a supported recommendation about aggregate workload monitoring, including a concrete situation in which the model should not be used. A recommendation against use is acceptable when the evidence supports it.
What to submit
Team · 6 points: the frozen run_submission.py, supporting source and metadata, experiments.csv, and revised report.ipynb. The notebook should connect framing and data audit, validation, model comparisons, calibration/error analysis, limitations, and reproducibility.
Team summary (part of the same 6 points): at most two pages presenting the question, key evidence, recommendation, and limitations for a nontechnical reader. Summarize existing findings; this is not another analysis. The final package checklist in Requirements below summarizes the files.
Individual · 2 points: your own stress-test code/results and interpretation. Choose a focused test, such as missing-feature sensitivity, unseen-category handling, or performance in a later visible time slice. Explain the setup, comparison, sample size where relevant, result, limitation, and what you would change or monitor. These are alternatives, not three required tests; you may extend your Week-10 experiment.
What good work shows: the frozen package runs, claims are traceable to results, and limitations affect the recommendation. Leaderboard rank is not a grading criterion. Reuse and improve earlier work rather than rewriting it for this deadline.
Demonstrate your own understanding on a small, unfamiliar example. During your assigned recitation, students work independently at the same time on an approximately 50-minute supervised notebook assessment, with equivalent versions across sections. There is no oral defense, individual interview, or presentation.
What to do
Use the staff-provided dataset and starter pipeline to identify a leakage or validation issue, make one or two small code fixes, and compute or interpret an evaluation result.
Answer two short written questions explaining your reasoning and what the result does or does not establish. You do not need to demonstrate your team's full project.
Prepare using the advance practice example and previously taught skills. The session is open-notes and closed-AI; exact session and submission instructions will be announced through Canvas, and approved accommodations apply.
What to submit · Individual
Notebook check · 4 points: submit the completed assessment notebook, including code, results, and the two written answers, during the session. No separate assessment report is required.
Contribution/AI record · 1 point, due November 13: identify your contributions and point to specific notebook sections, experiments, tests, or source files. Consolidate your three short records of consequential AI suggestions across the project: what was proposed, how you verified it, and what you accepted, modified, or rejected. Include your team ID. Full chat transcripts and a fabricated story about an AI mistake are not required; if you did not use AI, state that honestly and document your own checks.
What good work shows: you can diagnose a familiar kind of problem and justify a small correction independently. Staff use common checks for code/numerical results and a shared rubric for explanations. You do not need an individual TA appointment to be assessed.
“Points” above are percentage points of the course grade: 17 shared team points + 18 individual points = 35.
Then switch to your individual project.
Weeks 12–15 are reserved for a separate, short individual research project. Topic selection and a brief proposal begin in Week 12; the project will have its own instructions and rubric. There is no additional scheduled NYC 311 challenge submission after Week 11.
04 / Requirements
Make every conclusion traceable.
Use the approved data. Follow the feature contract. Fit learned preprocessing using training data and keep outcome information out of predictors.
Validate across time. Fit on training and diagnose on visible development data. For official evaluation, refit the selected method on the combined training and development data using the course template. Preserve your earlier development results.
Compare against meaningful baselines. Use Brier score, calibration, and error slices with sample counts. Inspect performance within agency/problem groups.
Document experiments. Record the hypothesis, change, result, and resulting decision. Include a feature ablation and a robustness test.
Make the work reproducible. Use the course environment, fixed seeds, and provided input/output interface. The submitted pipeline must run offline, without API calls or LLM inference. Final resource limits will accompany the course release.
Verify AI assistance. If you use AI, keep three short records of consequential suggestions: what was proposed, how you checked it, and what you accepted, modified, or rejected. If you do not use AI, state that and document your own verification checks instead.
What goes in the final submission?
One team coordinator submits:report.ipynb, a summary of at most two pages, experiments.csv, run_submission.py, supporting source files, and the required team/environment metadata. The notebook contains the evidence; the summary communicates the main findings. You do not need separate model-card and stakeholder-memo documents.
Each student submits separately: the assigned individual response, contribution/AI record, and team ID. The in-session written/code notebook is collected separately during recitation. The release will provide the exact directory template and submission slots.
Before submitting: rerun the notebook from a clean kernel, check the executable pipeline with the supplied preflight tool, and make sure referenced source files are included. Staff generate official predictions by executing the frozen package. Keep shared raw data out of the submission and do not manually edit predictions. Use CodeBench and retain the accepted submission receipt for both shared and individual work.
A public guide, private coursework.
This instructor-published description is public. Keep course datasets, solution code, reports, and student submissions in approved course channels unless the instructor authorizes publication. AI use during project work does not permit retrieving withheld outcomes or publishing course materials.
05 / Starter toolkit
Try the ideas before the data arrives.
These browser exercises use made-up examples and run locally on your device. They do not upload data, record grades, or use the hidden evaluation sets.
Interactive 01
Read the seven-day label
Explore the boundary and the maturity rule.
Label 0
The request is mature and closure at exactly seven days counts as within the window.
Age and closure duration here are local calendar days. At least 14 days of maturity is required for a course label; the prediction horizon remains seven days.
Interactive 02
Build intuition for Brier score
Adjust three probabilities. Each example's loss is (probability − outcome)². Brier score is their mean.
Squared error: 0.0400
Squared error: 0.1600
Squared error: 0.0400
Brier score Lower is better · 0 to 10.0800
These labels are visible only to explain the metric. In a real evaluation, make predictions before inspecting test outcomes. Three examples cannot establish calibration or generalization.
Illustration 03 · Invented groups, not NYC results
Does a 70% prediction behave like 70%?
A calibration plot compares average predicted probabilities with observed label-1 rates for groups of held-out requests. The dashed diagonal is the agreement reference; each dot below summarizes 100 synthetic requests.
Read the highlighted dot: the group averages a prediction of 0.70, but only 50 of its 100 requests have label 1. The model overestimates the seven-day recorded non-closure rate in this illustrative group. Calibration describes groups, not whether an individual request must have outcome 0 or 1.
These invented values demonstrate how to read a plot, not how any course model performs. In your own analysis, include group counts, consider uncertainty and bin choices, and use held-out predictions. Five dots alone cannot establish reliable calibration or generalization.
See the synthetic numbers behind the plot
Five invented groups; each contains 100 requests.
Mean prediction
Label-1 count
Observed rate
0.10
12 / 100
0.12
0.30
25 / 100
0.25
0.50
40 / 100
0.40
0.70
50 / 100
0.50
0.90
70 / 100
0.70
Available now · Synthetic data
Your first reproducible baseline
Download a small Python example that demonstrates a temporal split, a global-rate predictor, Brier score, and the prediction-file format. It uses only Python's standard library.
Requires Python 3.8 or newer; no extra packages, account, or API key. This is a practice example, not the official graded submission template.
When the course release opens
Open the assignment announced in Canvas and fetch it through CodeBench.
Read the data dictionary and run the onboarding notebook.
Run the supplied baseline before changing features or models.
Use the preflight check, submit through CodeBench, and save the receipt.
06 / How you're assessed
Good analysis earns the credit.
17shared team points
+
18individual points
=
35%of the course grade
Credit emphasizes sound data reasoning, validation, controlled experiments, reproducibility, clear conclusions, and individual understanding. Leaderboard rank carries no grade points. At most one course percentage point depends on valid private predictions and comparisons with reference baselines.
A constant probability is a legitimate baseline. A complicated model that fails to improve can still produce a valuable experiment when you explain what happened. Report subgroup sample counts, uncertainty, and limitations alongside your results.
Use AI thoughtfully; know your work.
AI assistance is encouraged during project work. Verify its outputs, cite material help, and explain your decisions in your submissions. The supervised individual written/code check is closed-AI. Complete chat transcripts are not a routine deliverable.
07 / FAQs
Questions you might be asking.
Do I need prior machine-learning experience?
No prior data-science experience is assumed. Early milestones focus on Python, data quality, and summaries. Modeling requirements follow the course material. Begin with the supplied examples and ask for help early.
Where are the official dataset and starter notebooks?
They will be announced through Canvas and released in CodeBench. They are not available from this public page yet. The downloadable Python starter above uses synthetic data for practice only.
Can I use AI tools to write code or suggest analyses?
Yes, during project work. Check consequential suggestions and document what you tested and decided. You remain responsible for the code, figures, and claims. Use course-approved tools and follow the course data-sharing rules. The individual written/code check is closed-AI.
Is the individual check an oral defense?
No. Students complete a written/code notebook assessment independently during their assigned recitation, at the same time as classmates. It uses a small staff-provided example, not a presentation of your team's project. Staff use common checks and a shared rubric to grade the submitted notebooks; there are no scheduled one-to-one defenses or interviews.
Can I download extra NYC records or use external data?
Use only course-released data in the scored pipeline. Do not retrieve withheld outcomes, match evaluation records to the public portal, or embed lookup tables. Any approved extension will be made available to everyone.
Why can't I download the final inference set?
Staff run your executable pipeline on unseen records to evaluate generalization. The labeled training and development sets give you the material needed to build and diagnose your method. You will receive aggregate evaluation feedback.
Does “slow” mean the city failed to resolve a problem?
No. It describes the absence of a recorded closure timestamp within our course horizon. Administrative closure, actual resolution, and agency-specific service commitments are different measurements. Avoid claims about individual service quality that the data cannot support.
Do I need a GPU, paid API, or complex model?
No. The scored workflow uses the course CPU environment. Clear baselines and careful evaluation can earn strong marks. Final runtime and memory limits will be published with the course template after staff benchmarking.
What do we submit as a team, and what do I submit myself?
One coordinator submits the shared code and report package. Each student separately submits the assigned individual evidence and team ID. Follow the directory template in each CodeBench release and confirm that your submission was accepted.
Can we change our approach after receiving feedback?
Yes. The milestones develop one evolving project. Preserve the original validation plan, document substantial changes, and explain what evidence motivated them. Do not silently replace an evaluation plan after seeing a favorable score.
How does this fit with the individual end-of-semester project?
The NYC 311 challenge ends in Week 11, including the final package and individual written/code check. The separate individual project begins in Week 12. To keep the challenge manageable, the clean-run rehearsal is part of the Week-10 submission, extra model families are optional, and the final report revises work you have already developed.
What if a teammate contributes less?
Raise concerns early using private course channels. Staff consider specific contribution evidence and individual assessments. Planning, testing, reviewing, and explaining are meaningful contributions; commit counts alone do not measure them.
Can I publish my solution or put it in a public repository?
This project description is instructor-published and public. Your course submissions and teaching materials remain subject to the syllabus's sharing policy. Keep them in approved private course channels unless the instructor grants permission to publish.
What if I need an extension or CodeBench is unavailable?
Follow the syllabus and current Canvas instructions for late work and accommodations. If the platform fails, preserve the error and timestamp and use the announced emergency Canvas submission route. Contact your TA promptly; do not assume an unsuccessful submission was accepted.
