NBME/CMS Analysis Dashboard
Summary
50Files discovered
50Files read
3551Question rows
3584Raw parsed rows
33Duplicate rows collapsed
3505Structurally usable rows
3491Pattern-ready rows
3929Parser warnings
99.0Average parse confidence
Parser Health
Parse Confidence
| Confidence band | Questions |
|---|---|
| high | 3522 |
| medium | 12 |
| low | 17 |
| unknown | 0 |
Top Warnings
| Warning | Count |
|---|---|
| HL_choice_choice_missing_from_ans_ocr | 153 |
| ref_stem_match:similarity=1.00,shared_terms=32 | 89 |
| ref_stem_match:similarity=1.00,shared_terms=25 | 81 |
| ref_stem_match:similarity=1.00,shared_terms=27 | 79 |
| ref_stem_match:similarity=1.00,shared_terms=41 | 74 |
| ref_stem_match:similarity=1.00,shared_terms=24 | 73 |
| ref_stem_match:similarity=1.00,shared_terms=30 | 72 |
| ref_stem_match:similarity=1.00,shared_terms=28 | 72 |
| ref_stem_match:similarity=1.00,shared_terms=43 | 70 |
| ref_stem_match:similarity=1.00,shared_terms=37 | 69 |
| ref_stem_match:similarity=1.00,shared_terms=31 | 69 |
| ref_stem_match:similarity=1.00,shared_terms=33 | 69 |
File Quality Triage
All NBME Files
All NBME files ranked by parse clean percentage. Use this to review structural usability first; pattern clean percentage is shown as secondary context.
| Source file | Total rows | Parse-clean rows | Parse excluded | Parse Clean % | Pattern Clean % | Recommendation | Top warnings |
|---|---|---|---|---|---|---|---|
| NBME/Q_txt_ref/NBME_16Q.txt | 169 | 162 | 7 | 95.9 | 94.7 | Structurally usable | missing_answer_key_csv (6); ref_stem_match:similarity=1.00,shared_terms=38 (6); ref_stem_match:similarity=1.00,shared_terms=43 (6); ref_stem_match:similarity=1.00,shared_terms=50 (6); ref_stem_match:similarity=1.00,shared_terms=45 (6); ref_stem_match:similarity=1.00,shared_terms=32 (5) |
| NBME/NBME_14A.txt | 200 | 198 | 2 | 99.0 | 100.0 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=31 (9); ref_stem_match:similarity=1.00,shared_terms=28 (9); ref_stem_match:similarity=1.00,shared_terms=24 (9); ref_stem_match:similarity=1.00,shared_terms=42 (9); ref_stem_match:similarity=1.00,shared_terms=40 (8); ref_stem_match:similarity=1.00,shared_terms=46 (8) |
| NBME/NBME_15A.txt | 200 | 198 | 2 | 99.0 | 98.0 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=41 (9); ref_stem_match:similarity=1.00,shared_terms=37 (9); ref_stem_match:similarity=1.00,shared_terms=20 (9); ref_stem_match:similarity=1.00,shared_terms=24 (9); ref_stem_match:similarity=1.00,shared_terms=28 (7); ref_stem_match:similarity=1.00,shared_terms=43 (7) |
| NBME/NBME_9A.txt | 199 | 197 | 2 | 99.0 | 98.5 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=38 (12); ref_stem_match:similarity=1.00,shared_terms=48 (10); ref_stem_match:similarity=1.00,shared_terms=45 (8); ref_stem_match:similarity=1.00,shared_terms=47 (8); ref_stem_match:similarity=1.00,shared_terms=44 (6); ref_stem_match:similarity=1.00,shared_terms=37 (5) |
| NBME/NBME_10A.txt | 197 | 196 | 1 | 99.5 | 100.0 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=37 (10); ref_stem_match:similarity=1.00,shared_terms=32 (9); ref_stem_match:similarity=1.00,shared_terms=36 (8); ref_stem_match:similarity=1.00,shared_terms=25 (8); ref_stem_match:similarity=1.00,shared_terms=30 (8); ref_stem_match:similarity=1.00,shared_terms=27 (8) |
| NBME/NBME_12A.txt | 200 | 199 | 1 | 99.5 | 99.0 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=32 (10); ref_stem_match:similarity=1.00,shared_terms=30 (9); ref_stem_match:similarity=1.00,shared_terms=43 (7); ref_stem_match:similarity=1.00,shared_terms=44 (7); ref_stem_match:similarity=1.00,shared_terms=39 (6); ref_stem_match:similarity=1.00,shared_terms=50 (6) |
| NBME/NBME_13A.txt | 198 | 197 | 1 | 99.5 | 99.0 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=40 (10); ref_stem_match:similarity=1.00,shared_terms=43 (9); ref_stem_match:similarity=1.00,shared_terms=35 (9); ref_stem_match:similarity=1.00,shared_terms=33 (8); ref_stem_match:similarity=1.00,shared_terms=32 (7); ref_stem_match:similarity=1.00,shared_terms=41 (7) |
| NBME/NBME_11A.txt | 199 | 199 | 0 | 100.0 | 99.0 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=41 (13); ref_stem_match:similarity=1.00,shared_terms=48 (8); ref_stem_match:similarity=1.00,shared_terms=46 (7); ref_stem_match:similarity=1.00,shared_terms=43 (7); ref_stem_match:similarity=1.00,shared_terms=25 (7); ref_stem_match:similarity=1.00,shared_terms=35 (6) |
Worst CMS Files
Worst 10 CMS files by parse clean percentage. Use this to prioritize source cleanup without duplicate or pattern-only penalties driving the ranking.
| Source file | Total rows | Parse-clean rows | Parse excluded | Parse Clean % | Pattern Clean % | Recommendation | Top warnings |
|---|---|---|---|---|---|---|---|
| CMS/EM_3A.txt | 50 | 44 | 6 | 88.0 | 100.0 | Usable with cleanup targets | missing_lead_in (6); ref_stem_match:similarity=1.00,shared_terms=35 (3); ref_stem_match:similarity=1.00,shared_terms=61 (3); ref_stem_match:similarity=1.00,shared_terms=41 (3); ref_stem_match:similarity=1.00,shared_terms=51 (2); ref_stem_match:similarity=1.00,shared_terms=34 (2) |
| CMS/FM_5A.txt | 36 | 33 | 3 | 91.7 | 91.7 | Structurally usable | trimmed_duple_ans_key_text (7); missing_lead_in (2); ref_stem_match:similarity=0.84,shared_terms=37 (1); ref_stem_match:similarity=0.76,shared_terms=22 (1); ref_stem_match:similarity=0.63,shared_terms=24 (1); ref_stem_match:similarity=0.16,shared_terms=5 (1) |
| CMS/Peds_4A.txt | 48 | 45 | 3 | 93.8 | 93.8 | Structurally usable | missing_answer_choices (3); missing_lead_in (3); ref_stem_match:similarity=0.89,shared_terms=34 (2); HL_choice_choice_missing_from_ans_ocr (2); ref_stem_match:similarity=1.00,shared_terms=19 (2); ref_stem_match:similarity=1.00,shared_terms=15 (2) |
| CMS/FM_4A.txt | 40 | 38 | 2 | 95.0 | 97.5 | Structurally usable | trimmed_duple_ans_key_text (7); ref_stem_match:similarity=0.80,shared_terms=32 (2); ref_stem_match:similarity=0.94,shared_terms=48 (1); ref_stem_match:similarity=0.94,shared_terms=32 (1); ref_stem_match:similarity=0.95,shared_terms=39 (1); ref_stem_match:similarity=0.71,shared_terms=39 (1) |
| CMS/Neuro_7A.txt | 41 | 39 | 2 | 95.1 | 95.1 | Structurally usable | HL_choice_choice_missing_from_ans_ocr (36); ref_stem_match:similarity=1.00,shared_terms=44 (2); ref_stem_match:similarity=1.00,shared_terms=16 (2); ref_stem_match:similarity=1.00,shared_terms=75 (2); ref_stem_match:similarity=1.00,shared_terms=34 (2); ref_stem_match:similarity=1.00,shared_terms=49 (2) |
| CMS/PSYCH_4A.txt | 48 | 46 | 2 | 95.8 | 93.8 | Structurally usable | trimmed_duple_ans_key_text (6); missing_answer_choices (2); choices_missing_correct_letter (2); missing_lead_in (2); ref_stem_match:similarity=0.99,shared_terms=67 (1); ref_stem_match:similarity=1.00,shared_terms=28 (1) |
| CMS/PSYCH_6A.txt | 48 | 46 | 2 | 95.8 | 93.8 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=45 (3); ref_stem_match:similarity=1.00,shared_terms=29 (2); ref_stem_match:similarity=1.00,shared_terms=19 (2); ref_stem_match:similarity=1.00,shared_terms=73 (2); ref_stem_match:similarity=0.04,shared_terms=4 (2); ref_answer_stem_mismatch:similarity=0.03,shared_terms=2 (2) |
| CMS/EM_2A.txt | 50 | 48 | 2 | 96.0 | 96.0 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=40 (3); ref_stem_match:similarity=1.00,shared_terms=44 (3); ref_stem_match:similarity=1.00,shared_terms=52 (3); ref_stem_match:similarity=1.00,shared_terms=21 (3); ref_stem_match:similarity=1.00,shared_terms=61 (2); ref_answer_stem_mismatch:answer_stem_too_short (2) |
| CMS/OBGYN_8A.txt | 41 | 40 | 1 | 97.6 | 87.8 | Structurally usable | HL_choice_choice_missing_from_ans_ocr (24); multiple_correct_ans_markers (4); ref_stem_match:similarity=1.00,shared_terms=38 (2); ref_stem_match:similarity=1.00,shared_terms=33 (2); ref_stem_match:similarity=1.00,shared_terms=11 (1); ref_stem_match:similarity=1.00,shared_terms=20 (1) |
| CMS/OBGYN_4A.txt | 44 | 43 | 1 | 97.7 | 97.7 | Structurally usable | ref_stem_match:similarity=1.00,shared_terms=32 (3); HL_choice_choice_missing_from_ans_ocr (3); ref_stem_match:similarity=1.00,shared_terms=43 (2); ref_stem_match:similarity=1.00,shared_terms=24 (2); ref_stem_match:similarity=1.00,shared_terms=22 (2); ref_stem_match:similarity=1.00,shared_terms=26 (2) |
Parser Quality by File
Parser-focused failures ranked by parse usability. Pattern clean percentage is kept here only as secondary context.
| Source file | Parsed rows | Parse-clean rows | Parse Clean % | Pattern Clean % | Skipped expl. | Synthetic rows | Trimmed duplicate keys | Top skip reasons |
|---|---|---|---|---|---|---|---|---|
| CMS/EM_3A.txt | 50 | 44 | 88.0 | 100.0 | 0 | 0 | 0 | missing_answer_key_and_choices (1) |
| CMS/Medicine_6A.txt | 49 | 44 | 89.8 | 89.8 | 0 | 0 | 1 | missing_answer_key_and_choices (4) |
| CMS/FM_5A.txt | 36 | 33 | 91.7 | 91.7 | 0 | 0 | 7 | |
| CMS/Peds_4A.txt | 49 | 45 | 91.8 | 91.8 | 0 | 0 | 1 | missing_answer_key_and_choices (1) |
| CMS/PSYCH_6A.txt | 50 | 46 | 92.0 | 90.0 | 0 | 0 | 0 | missing_answer_key_and_choices (1) |
| CMS/Neuro_7A.txt | 42 | 39 | 92.9 | 92.9 | 0 | 1 | 0 | missing_answer_key_and_choices (6) |
| CMS/FM_4A.txt | 40 | 38 | 95.0 | 97.5 | 0 | 0 | 7 | |
| CMS/OBGYN_8A.txt | 42 | 40 | 95.2 | 85.7 | 0 | 0 | 0 | missing_answer_key_and_choices (3) |
| CMS/PYSCH_3A.txt | 46 | 44 | 95.7 | 95.7 | 0 | 0 | 3 | missing_answer_key_and_choices (2) |
| CMS/PSYCH_4A.txt | 48 | 46 | 95.8 | 93.8 | 0 | 0 | 6 | |
| NBME/Q_txt_ref/NBME_16Q.txt | 169 | 162 | 95.9 | 94.7 | 0 | 0 | 0 | missing_real_question_stem_signal (1) |
| CMS/EM_2A.txt | 50 | 48 | 96.0 | 96.0 | 0 | 0 | 0 |
Remaining Cleanup Targets
Cleanup Buckets
| Cleanup bucket | Rows |
|---|---|
| other parser cleanup | 26 |
| low parse confidence | 12 |
| missing highlighted correct choice text | 12 |
| highlighted correct choice recovered | 6 |
| missing answer key | 2 |
| possible OCR damage | 1 |
| missing lead-in | 1 |
Lowest-confidence targets
| Source file | Q | Section | Section Q | Bucket | Recommended action | Confidence | Reasons |
|---|---|---|---|---|---|---|---|
| CMS/Peds_4A.txt | 43 | missing highlighted correct choice text | repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 0 | low parse confidence; missing correct answer text; correct letter not found in choices; reference-answer stem mismatch; ref_answer_stem_mismatch warning; reference stem does not match answer-file stem; incomplete answer choices; ambiguous extracted lead-in | ||
| CMS/OBGYN_8A.txt | 4 | missing highlighted correct choice text | repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 32 | low parse confidence; missing correct answer text; correct letter not found in choices; choices_missing_correct_letter warning; reference-answer stem mismatch; ref_answer_stem_mismatch warning; reference stem does not match answer-file stem | ||
| CMS/EM_2A.txt | 3 | low parse confidence | manual parser review recommended | 35 | low parse confidence; reference-answer stem mismatch; ref_answer_stem_mismatch warning; reference stem does not match answer-file stem | ||
| CMS/EM_2A.txt | 39 | low parse confidence | manual parser review recommended | 35 | low parse confidence; reference-answer stem mismatch; ref_answer_stem_mismatch warning; reference stem does not match answer-file stem | ||
| CMS/Medicine_4A.txt | 20 | low parse confidence | manual parser review recommended | 35 | low parse confidence; reference-answer stem mismatch; ref_answer_stem_mismatch warning; reference stem does not match answer-file stem | ||
| CMS/Medicine_3A.txt | 29 | missing highlighted correct choice text | repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 45 | low parse confidence; missing correct answer text; correct letter not found in choices; choices_missing_correct_letter warning; incomplete answer choices; ambiguous extracted lead-in | ||
| CMS/PSYCH_4A.txt | 17 | missing highlighted correct choice text | repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 45 | low parse confidence; missing correct answer text; correct letter not found in choices; choices_missing_correct_letter warning; incomplete answer choices; ambiguous extracted lead-in | ||
| CMS/PSYCH_4A.txt | 49 | missing highlighted correct choice text | repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 45 | low parse confidence; missing correct answer text; correct letter not found in choices; choices_missing_correct_letter warning; incomplete answer choices; ambiguous extracted lead-in | ||
| CMS/Peds_4A.txt | 22 | missing highlighted correct choice text | repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 45 | low parse confidence; missing correct answer text; correct letter not found in choices; choices_missing_correct_letter warning; incomplete answer choices; ambiguous extracted lead-in | ||
| CMS/Peds_4A.txt | 42 | missing highlighted correct choice text | repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 45 | low parse confidence; missing correct answer text; correct letter not found in choices; choices_missing_correct_letter warning; incomplete answer choices; ambiguous extracted lead-in | ||
| CMS/Neuro_7A.txt | 26 | missing answer key | needs answer key recovery | 50 | low parse confidence; missing correct answer; missing correct answer text; missing_answer_key warning | ||
| CMS/FM_5A.txt | 31 | low parse confidence | manual parser review recommended | 55 | low parse confidence; reference-answer stem mismatch; ref_answer_stem_mismatch warning; reference stem does not match answer-file stem | ||
| CMS/Neuro_4A.txt | 49 | missing answer key | needs answer key recovery | 55 | low parse confidence; missing correct answer; missing correct answer text; missing_answer_key warning | ||
| CMS/OBGYN_4A.txt | 12 | missing highlighted correct choice text | repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 55 | low parse confidence; missing correct answer text; correct letter not found in choices; choices_missing_correct_letter warning; multiple correct-answer markers; incomplete answer choices | ||
| CMS/PEDS_6A.txt | 43 | missing highlighted correct choice text | repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 55 | low parse confidence; missing correct answer text; correct letter not found in choices; choices_missing_correct_letter warning; incomplete answer choices | ||
| CMS/PSYCH_6A.txt | 39 | low parse confidence | manual parser review recommended | 55 | low parse confidence; reference-answer stem mismatch; ref_answer_stem_mismatch warning; reference stem does not match answer-file stem | ||
| CMS/PSYCH_6A.txt | 40 | low parse confidence | manual parser review recommended | 55 | low parse confidence; reference-answer stem mismatch; ref_answer_stem_mismatch warning; reference stem does not match answer-file stem; duplicate-looking file/question | ||
| NBME/Q_txt_ref/NBME_16Q.txt | 20 | low parse confidence | manual parser review recommended | 62 | low parse confidence; missing correct answer; missing correct answer text | ||
| NBME/Q_txt_ref/NBME_16Q.txt | 30 | low parse confidence | manual parser review recommended | 62 | low parse confidence; missing correct answer; missing correct answer text | ||
| NBME/Q_txt_ref/NBME_16Q.txt | 36 | low parse confidence | manual parser review recommended | 62 | low parse confidence; missing correct answer; missing correct answer text |
Reference Repair & File Review
Choice Source
| Choice source | Files/questions |
|---|---|
| reference | 3305 |
| ans_file_plus_reference | 153 |
| answer_file | 93 |
File Review Priority
| File review priority | Files |
|---|---|
| Structurally usable | 49 |
| Usable with cleanup targets | 1 |
Suggested Cleanup Action
| Suggested action | Files |
|---|---|
| manual review recommended | 26 |
| manual parser review recommended | 12 |
| repair answer choice OCR from Q_txt_ref or original PDF; correct answer letter may still be valid | 12 |
| no manual cleanup needed; correct answer choice text was repaired from Q_txt_ref | 6 |
| needs answer key recovery | 2 |
| review OCR text against source PDF | 1 |
| needs question stem cleanup | 1 |
Reference Repair Audit
Shows whether Q_txt_ref files are repairing choices/stems or failing to match.
| Source file | Total rows | Matched refs | Missing refs | Failed refs | Filled choices | Correct choice repairs | Still missing correct choice | Missing Q nums | Failed Q nums |
|---|---|---|---|---|---|---|---|---|---|
| CMS/FM_4A.txt | 40 | 29 | 11 | 0 | 0 | 153 | 11 | 15; 16; 17; 18; 20; 21; 22; 23; 25; 26; 27 | |
| CMS/OBGYN_7A.txt | 49 | 45 | 4 | 0 | 0 | 153 | 11 | 12; 28; 33; 50 | |
| CMS/FM_5A.txt | 36 | 32 | 3 | 0 | 0 | 153 | 11 | 1; 30; 41 | |
| CMS/FM_2A.txt | 50 | 49 | 1 | 0 | 0 | 153 | 11 | 31 | |
| CMS/Medicine_5A.txt | 49 | 48 | 1 | 0 | 0 | 153 | 11 | 47 | |
| CMS/Neuro_3A.txt | 48 | 47 | 1 | 0 | 0 | 153 | 11 | 37 | |
| CMS/Neuro_7A.txt | 41 | 40 | 1 | 0 | 0 | 153 | 11 | unknown_1 | |
| CMS/OBGYN_6A.txt | 45 | 44 | 1 | 0 | 0 | 153 | 11 | 47 | |
| CMS/PSYCH_4A.txt | 48 | 47 | 1 | 0 | 0 | 153 | 11 | 26 | |
| CMS/PSYCH_6A.txt | 48 | 45 | 1 | 0 | 0 | 153 | 11 | 4 | |
| NBME/Q_txt_ref/NBME_16Q.txt | 169 | 168 | 1 | 0 | 0 | 153 | 11 | 119 | |
| CMS/EM_1A.txt | 49 | 49 | 0 | 0 | 0 | 153 | 11 |
Duplicate / Collision Audit
After NBME section mapping, this should mostly show true parser collisions rather than routine Q1-Q50 section repeats.
| Source file | Q | Section | Section Q | Candidates | Collapsed | Retained confidence | Ref status | Candidate summary |
|---|---|---|---|---|---|---|---|---|
| CMS/Medicine_6A.txt | 2 | 4 | 3 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=B,has_correct_text=no,explanation_chars=1066; 2:dropped:confidence=100,warnings=2,choice_source=ans_file_plus_reference,ref=,correct=D,has_correct_text=no,explanation_chars=884; 3:dropped:confidence=32,warnings=2,choice_source=answer_file,ref=,correct=E,has_correct_text=no,explanation_chars=1109; 4:dropped:confidence=47,warnings=1,choice_source=answer_file,ref=,correct=B,has_correct_text=no,explanation_chars=492 | ||
| CMS/Surgery_6A.txt | 3 | 3 | 2 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=A,has_correct_text=no,explanation_chars=1414; 2:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=C,has_correct_text=no,explanation_chars=2256; 3:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=C,has_correct_text=no,explanation_chars=2339 | ||
| CMS/FM_3A.txt | 47 | 2 | 1 | 100 | matched_reference | 1:dropped:confidence=50,warnings=2,choice_source=reference,ref=,correct=,has_correct_text=no,explanation_chars=0; 2:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=D,has_correct_text=no,explanation_chars=2537 | ||
| CMS/Medicine_5A.txt | 6 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=2,choice_source=ans_file_plus_reference,ref=,correct=C,has_correct_text=no,explanation_chars=1488; 2:dropped:confidence=35,warnings=1,choice_source=answer_file,ref=,correct=D,has_correct_text=no,explanation_chars=1806 | ||
| CMS/Medicine_6A.txt | 10 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=D,has_correct_text=no,explanation_chars=612; 2:dropped:confidence=22,warnings=3,choice_source=answer_file,ref=,correct=E,has_correct_text=no,explanation_chars=1086 | ||
| CMS/Medicine_8A.txt | 3 | 2 | 1 | 92 | matched_reference | 1:kept:confidence=92,warnings=1,choice_source=reference,ref=,correct=A,has_correct_text=no,explanation_chars=2194; 2:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=E,has_correct_text=no,explanation_chars=2485 | ||
| CMS/Medicine_8A.txt | 20 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=D,has_correct_text=no,explanation_chars=3925; 2:dropped:confidence=100,warnings=1,choice_source=reference,ref=,correct=D,has_correct_text=no,explanation_chars=2377 | ||
| CMS/Neuro_5A.txt | 2 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=B,has_correct_text=no,explanation_chars=2471; 2:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=C,has_correct_text=no,explanation_chars=3651 | ||
| CMS/Neuro_7A.txt | 34 | 2 | 1 | 100 | matched_reference | 1:dropped:confidence=50,warnings=2,choice_source=reference,ref=,correct=,has_correct_text=no,explanation_chars=0; 2:kept:confidence=100,warnings=2,choice_source=ans_file_plus_reference,ref=,correct=D,has_correct_text=no,explanation_chars=2530 | ||
| CMS/OBGYN_5A.txt | 3 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=B,has_correct_text=no,explanation_chars=2608; 2:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=C,has_correct_text=no,explanation_chars=3019 | ||
| CMS/OBGYN_7A.txt | 2 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=B,has_correct_text=no,explanation_chars=3642; 2:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=C,has_correct_text=no,explanation_chars=2729 | ||
| CMS/OBGYN_7A.txt | 3 | 2 | 1 | 100 | matched_reference | 1:dropped:confidence=100,warnings=2,choice_source=reference,ref=,correct=A,has_correct_text=no,explanation_chars=999; 2:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=A,has_correct_text=no,explanation_chars=2625 | ||
| CMS/OBGYN_8A.txt | 3 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=2,choice_source=ans_file_plus_reference,ref=,correct=D,has_correct_text=no,explanation_chars=6800; 2:dropped:confidence=12,warnings=2,choice_source=answer_file,ref=,correct=C,has_correct_text=no,explanation_chars=3734 | ||
| CMS/PEDS_5A.txt | 2 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=D,has_correct_text=no,explanation_chars=2626; 2:dropped:confidence=35,warnings=1,choice_source=answer_file,ref=,correct=E,has_correct_text=no,explanation_chars=2424 | ||
| CMS/PEDS_5A.txt | 4 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=C,has_correct_text=no,explanation_chars=1466; 2:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=B,has_correct_text=no,explanation_chars=2223 | ||
| CMS/PEDS_8A.txt | 3 | 2 | 1 | 85 | matched_reference | 1:kept:confidence=85,warnings=2,choice_source=answer_file,ref=,correct=C,has_correct_text=no,explanation_chars=140; 2:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=B,has_correct_text=no,explanation_chars=2743 | ||
| CMS/PSYCH_6A.txt | 35 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=A,has_correct_text=no,explanation_chars=3467; 2:dropped:confidence=100,warnings=1,choice_source=reference,ref=,correct=B,has_correct_text=no,explanation_chars=2490 | ||
| CMS/PSYCH_6A.txt | 46 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=B,has_correct_text=no,explanation_chars=2150; 2:dropped:confidence=100,warnings=1,choice_source=answer_file,ref=,correct=H,has_correct_text=no,explanation_chars=2576 | ||
| CMS/PYSCH_3A.txt | 2 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=D,has_correct_text=no,explanation_chars=5965; 2:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=A,has_correct_text=no,explanation_chars=4821 | ||
| CMS/PYSCH_3A.txt | 4 | 2 | 1 | 100 | matched_reference | 1:kept:confidence=100,warnings=1,choice_source=reference,ref=,correct=C,has_correct_text=no,explanation_chars=2388; 2:dropped:confidence=55,warnings=1,choice_source=answer_file,ref=,correct=B,has_correct_text=no,explanation_chars=1938 |
Pattern Overview
Top Exclusion Reasons
| Reason | Questions |
|---|---|
| duplicate-looking file/question | 26 |
| low parse confidence | 24 |
| missing correct answer text | 20 |
| correct letter not found in choices | 12 |
| choices_missing_correct_letter warning | 11 |
| reference-answer stem mismatch | 8 |
| ref_answer_stem_mismatch warning | 8 |
| reference stem does not match answer-file stem | 8 |
| incomplete answer choices | 8 |
| missing correct answer | 8 |
| ambiguous extracted lead-in | 7 |
| multiple correct-answer markers | 7 |
Top Source Files
| Source file | Questions |
|---|---|
| NBME/NBME_12A.txt | 200 |
| NBME/NBME_14A.txt | 200 |
| NBME/NBME_15A.txt | 200 |
| NBME/NBME_11A.txt | 199 |
| NBME/NBME_9A.txt | 199 |
| NBME/NBME_13A.txt | 198 |
| NBME/NBME_10A.txt | 197 |
| NBME/Q_txt_ref/NBME_16Q.txt | 169 |
| CMS/EM_2A.txt | 50 |
| CMS/EM_3A.txt | 50 |