HackerRank Hiring Agent open-source ATS

I ran the same resume through HackerRank's newly open-sourced hiring agent 100 times. The scores ranged between 66 and 99 out of 120. If a company sets a cutoff at 85, the same candidate gets rejected 65% of the time. That's not a hiring tool. That's a dice roll with extra steps.

HackerRank, the company behind one of the most widely used technical assessment platforms, just put their internal AI resume scorer on GitHub under an MIT license. The repo is called hiring-agent, and it does exactly what you'd expect: parses your PDF resume, checks your GitHub profile, and spits out a score. The internet lost its mind. LinkedIn posts with thousands of likes. Reddit threads in r/leetcode and r/internships. Everyone wanted to know how the machine judges them.

The problem is the machine can't make up its mind.

How the sausage gets made

The pipeline is straightforward enough. Your PDF resume gets converted to Markdown, then an LLM extracts structured data in six passes: basics, work history, education, skills, projects, and awards. The tool then hits the GitHub API, grabs your profile and repos, classifies your projects, and picks the top seven for evaluation. Finally, everything gets fed into an LLM for scoring.

The scoring breakdown is revealing in itself. Out of 120 base points: 35 for open source contributions, 30 for personal projects, 25 for work experience, and 10 for technical skills. There's a 20-point bonus bucket for things like startup experience, a portfolio site, or a blog. Notice what's happening here. Open source and personal projects account for 65 of 120 base points. That's 54% of your score before the LLM even looks at your actual job experience.

You can run it locally with Ollama (fully offline, any model) or point it at Google Gemini. The prompts live in Jinja templates. The scoring logic is in evaluator.py. Everything is transparent. That's the good news.

The luck filter

Here's what happens when you actually run it. A developer who tested the tool extensively found that running the same resume 100 times produced scores between 66 and 99. The variance isn't a bug in the traditional sense. It's an inherent property of using LLMs for subjective evaluation. Even at temperature zero, the model gives different answers on vague categories like "project quality" or "experience depth."

Some categories are stable. Technical skills scoring works like a checklist: do you list Python? Yes or no. That's deterministic. But projects and experience are pure vibes. The evaluator prompt for production experience reads: "Analyze the work and volunteer sections for real-world, internship, or production experience. Give extra points for founder roles." There's no rubric. No anchor points. No definition of what separates a junior intern from a principal engineer. The LLM guesses, and it guesses differently every time.

The danunparsed.com blog broke this down sharply: the tool effectively is a random filter rather than a quality filter. If you're a senior engineer who spent ten years building proprietary systems at a FAANG company, your GitHub profile might be empty. The tool docks you 35 points for open source and 30 for projects. A fresh grad with a bunch of side repos and Hacktoberfest contributions sails past you.

What this actually reveals

The interesting thing isn't that HackerRank's tool is flawed. It's that the flaw is the point. Every AI-powered resume screening tool on the market has the same fundamental problem. They all use LLMs (or older ML models with the same limitations) to make subjective judgments about experience quality. The only difference is that HackerRank open-sourced theirs so you can actually see the guts.

HackerRank's own blog post about AI hiring tools says it plainly: "Use AI screening to filter, not to evaluate. Treat it as a routing tool, not a scoring oracle." They know. They've been selling assessment platforms for years. They understand that LLMs are good at objective tasks (parsing data, checking for keywords) and bad at subjective ones (judging project complexity, evaluating career trajectory). But the market wants a score, so everyone builds a score.

The regulatory landscape is catching up. The EU AI Act, NYC Local Law 144, and EEOC guidance all push toward explainability and auditability. An AI tool that gives the same resume a 66 and a 99 within the same afternoon is not explainable. It's not auditable. It's a black box that happens to have its source code on GitHub.

What you should actually do with it

If you're a job seeker, run your resume through it. Not because the score matters, but because the output tells you what the AI sees. The tool produces category scores with evidence: it shows you exactly which parts of your resume it extracted, which GitHub repos it picked, and why it assigned the scores it did. That's useful debugging information for tailoring your resume to pass automated screening.

If you're a hiring manager, don't use this for scoring. Use it for parsing. The PDF-to-structured-data pipeline is genuinely good. The GitHub enrichment is clever. The extraction templates are well-designed. Rip out the evaluator, keep the rest, and make your own scoring rubric with actual anchor points. Or better yet, use a human.

If you're an engineer, the codebase is worth reading. It's a clean example of an LLM-powered pipeline: provider-agnostic models, Jinja templates for prompts, Pydantic schemas for structured output, and a fairness-constrained evaluator. Even if you think the scoring is broken, the architecture is solid.

The uncomfortable truth

The tool has been on GitHub since October 2025. It only went viral in June 2026. The commit history shows it was open-sourced months ago, but nobody cared until LinkedIn started buzzing about it. That tells you something about the state of AI hiring: people want to believe the machine can judge them fairly, and they want to believe that if they could just see the code, they'd understand the judgment. The code is right there. The judgment is still random.

HackerRank didn't make a bad tool. They made an honest one. Most ATS vendors hide their scoring behind API calls and proprietary models. HackerRank showed you exactly how the sausage gets made, and it turns out the sausage is 54% GitHub stars and vibes.

Sources