Getting Started
Your data is in a database. Use SQL or Python to explore it:
SQL: SELECT * FROM case_data LIMIT 10
Python: df = pd.read_csv('case_data.csv')
See data_dictionary.md for column descriptions.
03 — What we measure
We measure what other
We measure what other
interviews can't see.
Anyone can get the right answer with AI now. The signal is in how they got there
— what they questioned, what they trusted, where they pushed back.
What they bring beyond AI
Judgment, framing, and taste.
How they frame an ambiguous problem, what tradeoffs they prioritize, and whether they can tell when an answer is actually useful — not just technically correct.
How they work with AI
Verification, pushback, restraint.
Did they verify the AI's output? Did they catch its mistakes? Did they over-trust it? We watch the full session — every prompt, every accepted suggestion, every override.
A complete evaluation report.
We analyze every prompt, every cell, every decision — and we show you all the evidences behind every score.
Sample Report Template
Workspace / Reviews / Session
Review: Sample Candidate
Sample interview · Assessment submitted Apr 30, 2026, 6:08 PM
Final Report
Action Timeline
Case & Results
AI Chat Timeline
Follow-Up Questions
Overall score
9.0 / 10
Summary
Strong fit for the analyst role's core ask: candidate consistently led with a hypothesis before reaching for tooling, mirroring how the team scopes ambiguous business questions day-to-day. Worth probing in interview — recommendations stayed at the diagnostic level and didn't push into the operational tradeoffs this role owns once a quarter.
Your decision
Good Fit
Hold
No Fit
Notes
Auto-saved · just now
Scores & evidence
Edge Beyond AI
4.5 / 5
Candidate brought the right analytical lens to the problem before touching the data. Quiz answers already named the core issue; the IDE work mostly confirmed it.
Problem framing & hypothesis
5 / 5
Domain expertise
4 / 5
Strong fluency with the relevant industry context: cited domain-specific risk factors and operational tradeoffs that would not be visible from the data alone. Recommendation reflects awareness of stakeholder constraints, not just statistics.
-
+
Quiz Q1 invokes industry-relevant factors AI couldn't infer from the dataset Quiz Q1
-
+
Section 2 mentions operational load and downstream impact — stakeholder-aware framing Section 2 Q1
-
+
Suggested a practitioner-grade follow-up metric for sizing the next step Section 2 Q1
Insight depth
4 / 5
Recommendation quality
5 / 5
AI Collaboration
4.5 / 5
Candidate used AI as a focused executor after framing the problem on their own. AI was an accelerator for one specific task, not a thinking partner — exactly the pattern we want to see.
Delegation
5 / 5
Prompt description
5 / 5
Critical evaluation
4 / 5
Candidate verified the AI's output against their own prior analysis (both pointed the same direction), and read the result table critically — citing specific numbers as the basis for their conclusion rather than paraphrasing the assistant.
-
+
Section 2 cites specific output values as the basis for the conclusion — read the table, didn't just paraphrase AI Section 2 Q1
-
+
Cross-checked AI output against own prior cell — convergent evidence Cell #3 at min 4
-
−
Did not push back on or flag a suspicious value in the AI output — accepted the model as-is Cell #4 at min 6
Iteration & verification
4 / 5
Action Timeline
Every AI prompt, cell run, and edit — in order.
00:00
— · 0/30 min
Session started
Candidate opened the IDE and read the prompt.
02:14
Cell #1
Cell #1 run · explored data shape + lift
~3,000 rows × 7 cols. First-pass lift of the focal segment: ~1.04x — barely meaningful.
04:02
Cell #2
Cell #2 run · segmented metric by quartile
Looked at how the target rate varied across four buckets of the candidate variable.
04:47
msg #1 · 1 / ∞
Prompted AI · msg #1
Hypothesis-driven request for a regression with explicit controls.
"Fit a model of the target on the candidate variable plus three controls. Show coefficients with 95% CIs and p-values…"
05:18
[AI-generated]
Cell #3 added · AI-generated cell
Candidate kept the AI-suggested cell after reviewing the diff.
06:11
Cell #4
Cell #4 run · reviewed model output
Read the coefficient table; flagged the dominant variable and dismissed two non-significant ones.
06:42
phase 2 → 3
Moved to Section 2 · Findings
Investigate phase complete. Notebook locked for reference.
07:53
1 prompt used
Submitted answer
~1,500-character recommendation with confidence note and proposed next-step validation.
Case
[Case title goes here]
A short business framing for the case appears here — who the team is, what they're trying to decide, and the constraints they're operating under. Two or three sentences set the stage without prescribing the answer.
Pre-assessment quiz · 3 framing questions
Quiz Q1
[Framing question 1 goes here — sets up the business hypothesis the candidate needs to push back on.]
[Sample candidate response — typically 2–4 sentences naming the candidate's mental model, the variables they'd reach for, and the assumption they want to test. Long-form free text, no character cap.]
Quiz Q2
[Framing question 2 goes here — asks the candidate where they'd start the analysis and why.]
[Sample candidate response — describes the first cut of the data they'd run, what they'd compare it against, and which secondary check would either confirm or kill the hypothesis.]
Quiz Q3
[Framing question 3 goes here — drops a partial statistic on the candidate and asks what they'd interpret and check next.]
[Sample candidate response — names the missing baseline, describes the lift calculation they'd want to do, and adds a sanity-check on a likely confounder.]
Notebook · final state
Markdown
Cell 1
Getting Started
Your data is in a database. Use SQL or Python to explore it:SQL:
SELECT * FROM case_data LIMIT 10
Python:
df = pd.read_csv('case_data.csv')
See
data_dictionary.md for column descriptions.
Python
In [1]:
import pandas as pd df = pd.read_csv('case_data.csv') # Headline numbers outcome_rate = df['target'].mean() segment_share = (df['segment_flag'] == 1).mean() print(f'overall outcome rate: {outcome_rate:.4f}') print(f'segment share overall: {segment_share:.4f}') # Lift: P(segment | event) / P(segment) events = df[df['target'] == 1] seg_among = (events['segment_flag'] == 1).mean() print(f'lift: {seg_among / segment_share:.3f}x')
overall outcome rate: 0.0xxx segment share overall: 0.xxxx lift: 1.0xx (>1 means over-represented)
Python
In [2]:
# Outcome rate stratified by quartile of candidate variable df['var_q'] = pd.qcut(df['candidate_var'], 4, labels=['Q1 low','Q2','Q3','Q4 high']) print(df.groupby('var_q')['target'].mean().round(4))
var_q Q1 low 0.0xxx Q2 0.0xxx Q3 0.0xxx Q4 high 0.xxxx Name: target, dtype: float64
Python
In [3]:
[AI-GENERATED]
import statsmodels.api as sm features = ['segment_flag', 'candidate_var', 'control_a', 'control_b'] X = sm.add_constant(df[features]) y = df['target'] model = sm.Logit(y, X).fit(disp=False) print(model.summary())
coef OR ci_low ci_high p_value const -x.xxxx 0.0xx 0.0xx 0.0xx 0.000 segment_flag 0.0xxx 1.0xx 0.8xx 1.4xx 0.6xx candidate_var x.xxxx xx.xx xx.xx xxx.xx 0.000 control_a -0.0000 1.000 1.000 1.000 0.3xx control_b 0.0000 1.000 1.000 1.000 0.2xx
Section 2 · Share your findings
Q1
[Final-answer prompt goes here — asks the candidate to summarize findings and make a recommendation to a named stakeholder.]
[Sample candidate final answer — leads with the recommendation, then 2–3 supporting bullets, then a confidence note and a proposed validation step.] What I found: — First supporting bullet: the original hypothesis didn't survive the basic lift check. — Second bullet: the real signal sits on a different variable, with a clean monotonic gradient across quartiles. — Third bullet: a multivariate model confirms the direction — the focal variable is non-significant once controls are added; the alternative is strongly significant. Recommendation: ship the alternative routing rule; hold off on the original. Confidence: high on direction, medium on cutoff — next step is a holdout validation on a recent vintage.
AI Chat Timeline
Every prompt the candidate sent and the AI's response, in order.
04:47 · msg #1
You
[Sample user prompt — candidate names the hypothesis they're testing, the early signal they've already seen, and asks the assistant for a specific multivariate model with named variables. They specify what output they want — coefficient table, p-values, confidence intervals — and what they'll be looking for to confirm or reject the hypothesis.]
AI Assistant
[Sample assistant reply — confirms the cell was added, names the columns of the output table the candidate should expect, and points out what pattern in the output would support or contradict the candidate's hypothesis. Stays neutral; doesn't volunteer a conclusion.]
Resume-based questions
5 questions
Claim verification
[Resume-claim verification question — references a specific result the candidate cited and asks them to walk through how they validated it.]
Resume · prior role
Process probe
[Process-probe question — references a project on the resume and asks how the candidate handled the operational constraints around it.]
Resume · prior project
Case-based questions
5 questions
Probe critical eval
[Critical-eval question — surfaces a specific output the candidate accepted at face value and asks them to interpret it more carefully.]
Cell #4 at min 6
Depth check
[Depth-check question — acknowledges the candidate stopped at a coarse cut and asks how they'd land a specific operational threshold with more time.]
Section 2 Q1
Hold out concern
[Hold-out-concern question — surfaces a model-quality metric the candidate didn't flag and asks whether it would change their ship/no-ship call.]
Cell #4 at min 6
