HackerRank Open-Source ATS Shows Unstable Resume Scoring
Original: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
Why This Matters
Highlights fundamental limitations of LLM-based hiring tools and raises concerns about fairness in automated resume screening systems.
HackerRank's open-source applicant tracking system produces inconsistent resume scores ranging from 66 to 99 for identical submissions, with scores varying by up to 33 points across multiple runs of the same resume.
HackerRank recently open-sourced its ATS tool, which has gained attention on LinkedIn and Reddit. A test by Dan Kinsky revealed significant scoring inconsistency: the same resume scored 90/100 on the first run, then 74/100 after removing debug statements, with 100 consecutive runs producing scores ranging from 66 to 99. For companies with an 85-point cutoff, identical resumes would fail 65% of the time. The system works by parsing PDF resumes into text, calling an LLM six times to extract structured information (basics, work history, education, skills, projects, awards), pulling GitHub profile data, then generating a final score out of 100 plus up to 20 bonus points. Scoring categories include: work experience (25 points), personal projects (30 points), technical skills (10 points), and open source contributions (35 points), with additional bonuses for startup experience and portfolios. Analysis revealed technical skills scoring showed 98/100 consistency (8/10 in nearly all runs) because it uses a simple checklist format. Projects showed high variation because LLMs struggle with subjective judgments like architectural complexity. Temperature settings as low as 0 did not resolve this non-determinism. Experience scoring remained consistently at 25/25 regardless of qualifications, stemming from minimal evaluation criteria with no detailed rubric or examples. The tool's design flaw stems from combining detailed but inconsistent rubrics (projects) with oversimplified anchors (experience), making hiring outcomes unpredictable.