NYC Law 144 Impact Ratios: Keep or Retire Your AI Screener

```html

Against that backdrop, the typical 2026 audit summary reads as a clean sweep of impact ratios above 0.80 — and that sweep is precisely why most employers are about to make the wrong keep-or-retire call. The 0.80 line is a paper-era heuristic, never engineered to certify a modern algorithmic screener. Treating it as proof of fairness lets formal compliance masquerade as evidence, green-lighting tools a stricter statistical standard would flag.

By 2026, the statistically defensible keep-line has moved to 0.85 — and only for screeners clearing it with adequately sized samples per group, sustained across two consecutive audit cycles. Ratios that scrape past 0.80 on thin slices of data no longer justify retention. Run your screener against that harder test, and for many nominally 'passing' tools the honest 2026 verdict is retire, not keep.

New York City enforces Local Law 144, and its regulated object is broader than most employers assume: any Automated Employment Decision Tool — machine learning, statistical modeling, or data-analytics software that substantially assists or replaces discretionary screening-out of candidates for NYC-based jobs, with resume rankers, chatbot screeners, and video-interview scorers all qualifying. Recognize what the statute actually built: a disclosure machine, not a fairness machine. Its arithmetic, its auditor rules, and its data-hygiene carve-outs converge on tables that look clean while deciding nothing about whether a tool stays in production.

The 0.80 Machine

The arithmetic comes first. For each required category — sex; each race/ethnicity group; each sex-by-race intersection — compute the selection rate (candidates advanced ÷ candidates assessed), then divide it by the selection rate of the most-favored group in the same comparison. Where women advance at four-fifths the rate of men, the female impact ratio prints 0.80. The measurement window is fixed: the trailing 12 months of the tool's actual NYC use, not a vendor-chosen demo period.

An independent auditor — not the vendor, not the employer — must run the audit at least annually; the employer must publish the summary publicly before first using the tool and give candidates notice at least 10 business days before the AEDT screens them, per DCWP's implementing rules. The penalty regime prices paperwork, not outcomes:

Data hygiene completes the machine. DCWP's rules allow auditors to mark categories with too few candidates as having insufficient data and require selections from candidates of unknown demographic to be reported separately — so a clean-looking 2026 summary can simply omit the smallest groups from scrutiny. Those omissions are where disparity concentrates: according to the AEQUITAS study presented at ASE 2018, up to 70% of generated probe inputs were discriminatory against the machine-learning classifiers tested, with violations surfacing in all six classifiers studied, including one built with fairness constraints. And the audience is already skeptical: according to a Gartner survey of 2,918 job candidates released July 31, 2025, only 26% of applicants trust AI to evaluate them fairly, and 25% trust an employer less once they learn AI is involved — the backdrop Hirium's July 10, 2026 analysis calls defining for automated hiring.

Read a 2026 summary, then, as a reconstruction exercise: identify which categories carried adequate candidate counts, which were marked insufficient-data, and where unknown-demographic selections went — then map each surviving cell onto this ladder:

Exactly one outcome clears the bar: every adequately populated protected category above 0.85 in two consecutive audits. Everything else on the ladder is a phase-out path — and the employer, not the auditor and not DCWP, owns the decision to walk it.

The four-fifths rule is older than the spreadsheet. When the EEOC adopted its Uniform Guidelines on Employee Selection Procedures, a group's selection rate below four-fifths of the highest group's rate was treated as evidence of adverse impact — a heuristic built for paper-and-pencil tests administered once, scored once, and validated against a fixed applicant pool. Nothing in that design anticipated systems that re-score every incoming application and retrain on last quarter's hires, yet the 0.80 line is now applied wholesale to algorithms, and a model idling at 0.84 inherits a presumption of safety it was never engineered to earn.

The pre-law record shows what unaudited screeners do quietly. According to Reuters' investigation by James Dastin, Amazon scrapped an experimental résumé screener trained on roughly ten years of applications after it learned to downgrade résumés containing the word "women's" and to penalize graduates of two women's colleges. No auditor produced those ratios; the disparity surfaced internally, at scale, because the model had optimized for resemblance to a decade of past hires. That is the failure mode Law 144 audits exist to catch — and why a tool that has merely never been audited deserves zero benefit of the doubt.

Three Years of Disclosed Ratios

The mechanism is measurable, not mysterious. According to Caliskan, Bryson & Narayanan, writing in Science, models trained on ordinary language corpora absorb human-like stereotypes — their word embeddings associated European-American names with pleasant terms relative to African-American names. A screener trained on a decade of past hiring decisions inherits the past's skews even when nobody intends discrimination, which is why intent-based defenses carry no weight and the operative question is always measurement over motive.

Post-enforcement disclosures confirm how low the bar sits in practice. According to peer-reviewed analyses of published Law 144 audit summaries presented at ACM FAccT, the large majority of disclosed impact ratios cleared 0.80 — the precise pass rate shifts with whichever slice of posted summaries a team assembles, so pull the paper's own count rather than trusting any secondhand figure. The direction is what matters: a bare pass is now the market norm, which means "our vendor's audit came back clean, every ratio above 0.80" describes the median posting, not a defensible keep-signal. Under the standard this guide applies, the signal starts at 0.85, sustained across every adequately populated protected category, in two consecutive audits.

Disclosure itself stays thin. In the months after enforcement began, press tallies counted only a few dozen publicly posted audit summaries; as of January 2026, trackers maintained by Bloomberg Law and employment-law firms such as Littler and Fisher Phillips count more, but coverage remains a small share of New York City-using employers — and every such tally structurally undercounts, because summaries sit on individual employer career pages rather than any central registry. Treat absence from a tracker as absence of disclosure, not absence of tooling, and quote only the live cumulative count.

Last, the instability fact motivating the two-cycle rule: where vendors have published consecutive annual audits, individual-category ratios commonly swing by ±0.05–0.10 between cycles. Illustrative arithmetic, not a disclosed result: a category reading 0.88 one cycle and 0.81 the next slides from keep-side to gray-zone on sampling noise alone, while a 0.83-to-0.87 bounce flatters a marginal tool for a year. A single-cycle reading cannot support a keep decision, however clean it looks.

Before your next audit cycle: request your vendor's two most recent published summaries, rebuild the category-by-category table yourself, and flag any adequately populated protected category that fell below 0.85 in either cycle. Any such hit means retirement or immediate replacement procurement — with human review permitted only for 0.80–0.84 gray-zone categories while the replacement is procured.

One disclosed impact ratio can support three different keep-or-retire decisions, and the entire difference lives in the reading rule. Strategy A is binary compliance: any category ratio at or above 0.80 means keep — this is the "our vendor's bias audit came back clean, so the screener is safe" reflex, and it is the wrong test. Strategy B is margin-with-floor: keep only if every adequately populated protected category clears 0.85. Strategy C is the two-cycle trend: require 0.85 or better in two consecutive annual audits. This guide's winner, stated upfront, is B and C combined.

Strategy A fails because it treats a point estimate as a safety line. A screener posting 0.81 for a thinly populated group carries a confidence interval wide enough to plausibly extend below parity — the true disparity may be worse than the disclosed number, and next year's draw will land somewhere else. Scale makes the point: according to the LLMORPH paper presented at ASE 2025, characterizing the behavior of three state-of-the-art LLMs took more than 561,000 test executions applying 36 metamorphic relations across four NLP benchmarks. When pinning down what these systems actually do demands six-figure probe counts, one annual ratio per category is a thin sample of a moving target. On false-pass risk — keeping a drifting or biased tool — A is the outright loser.

Pass/Fail Is the Wrong Test

Strategy B repairs the two classic floor errors. First, it stops you acting on small-cell ratios, which swing violently on a handful of outcomes. Second, it closes the "insufficient data" loophole: vendors leave blanks for thin categories, and under A those blanks silently read as passes; under B, a category below the minimum-sample floor cannot clear the bar, so omission becomes visible instead of convenient. What B cannot do is catch drift — a strong current year can mask a bad prior year, and a tool that degraded after deployment still looks spotless. B wins on stability, loses on drift detection.

Strategy C prices in that year-over-year swing by demanding two consecutive cycles at 0.85 or better. Its cost is speed: a newly deployed tool that is genuinely sound waits a full extra audit cycle before it can earn a keep. Hence the hybrid verdict — apply C whenever two cycles of history exist, and apply B alone to tools sitting in their first audit cycle.

Winner: the two-cycle, minimum-sample, 0.85-margin rule — retire anything below 0.85 in any adequately powered category, and keep only what clears that bar twice.

An impact ratio printed in a Law 144 summary is a point estimate wearing the costume of a measurement, and the mandated disclosure format tells you almost nothing about its precision. No confidence intervals, no per-cell denominators unless the vendor volunteers them, no binding statement of which model version was tested. That evidentiary poverty is precisely why the keep-signal in this guide demands two consecutive clears at 0.85 on adequately populated cells: it is a blunt power correction for evidence that arrives stripped of its error bars.

Four gaps do most of the damage. First, sampling variance, best shown with arithmetic rather than empirics: in a same-sized pool where the strongest group out-advances the comparison group by only a handful of individual decisions, the ratio prints 0.80; flip one applicant in that cell and it prints 0.85; flip two and it prints 0.90. Two personnel decisions straddle the entire regulatory band, which is why a lone cycle near the line carries little inferential weight. Second, coverage: the required ratio tables run on sex and race/ethnicity, so age and disability — classes the New York City Human Rights Law fully protects — need never appear, and a spotless report can be silent on entire populations. Third, version drift: the audit describes a snapshot, and vendors replace models faster than annual filings. Fourth, selection: the pool is whoever flowed through the New York-relevant funnel, which the vendor chose the window for.

What the Data Doesn't Tell You

Variance across cases compounds this. An aggregate ratio can clear while the job family doing most of the screening fails — averaging across roles is not evidence about any role. Stage matters too: a ratio measured at resume screen says little about interview-stage filtering. And the applicant mix shifts between cycles, so two consecutive passes are correlated through the pool rather than fully independent replications. Two cycles narrow the worry considerably; they do not erase it.

Where the rule goes quiet deserves equal honesty. If no protected category reaches an adequate sample in either cycle, the rule mandates nothing — and mandates-nothing is not a clean bill. For genuinely low-volume employers the honest label is "unmeasured," and the defensible moves are widening the observation window or pooling adjacent roles, not declaring victory on cells shy of the threshold. Testing many race-by-sex intersectional cells also guarantees occasional chance dips below any fixed line. Conversely, the fixed 0.85 cutoff will retire some tools whose gray-zone intervals overlap parity — a real cost, accepted deliberately because compliance cannot adjudicate confidence intervals at scale, and absorbed by the human-review fallback rather than by repainting the gray zone green.

Say the debunked version plainly, because it still closes deals: a vendor report showing every ratio above 0.80 does not establish fairness or legal safety. Such a report can rest on sixty-person cells, omit age and disability entirely, describe a model version since replaced, and sit within sampling noise of outright disparity. It establishes, reliably, only that the vendor hired an auditor.

The operational takeaway: before treating this cycle's results as a keep-signal, demand three artifacts from the auditor — per-cell denominators, the version identifier of the model actually tested, and the complete list of categories examined. If any of the three is missing, what you are holding is marketing, not measurement.

A standard error of roughly 0.19 is what hides inside every clean-looking impact ratio computed on a small candidate cell. At ordinary screening sample sizes and base rates, the confidence band around a perfectly neutral tool's observed 1.00 runs approximately 0.62 to 1.38. Chance alone therefore regularly produces sub-0.80 readings for screeners with no disparity at all, crowds the 0.80–0.84 gray zone with tools whose true ratio is unknowable from one cycle, and supplies some vendors' 0.85s outright. One clean cycle is a coin that landed well, not a measurement — which is why the keep rule insists on two consecutive audits clearing 0.85.

The gaps are structural. Law 144's required cells stop at sex, race/ethnicity, and their intersections; age — protected at 40-plus under the ADEA — disability, and veteran status are absent from the grid, so a screener can post flawless 2026 ratios while systematically filtering out older workers. Nor does a local pass buy federal cover: the EEOC's technical-assistance guidance on AI under Title VII makes clear an audit confers no immunity from federal enforcement, and according to Netguru's review of HUD and CFPB guidance, proxy discrimination through non-protected variables that correlate with race or national origin is treated as legally equivalent to direct discrimination.

What a Clean 2026 Ratio Can't Tell You

Third, the certificate expires silently. The audit describes only the trailing year; a vendor that retrains the model, swaps the embedding layer, or moves the score cutoff the week after fieldwork ends is shipping a different tool than the one audited, and Law 144 imposes no continuous-monitoring duty between annual cycles. Netguru's deployment standard for supervised-classification screening tools answers this mechanically: write the bias-audit cadence into the deployment contract itself, so a mid-cycle model change triggers a fresh audit instead of surfacing at the next annual disclosure.

Fourth, the ratio says nothing about whether the tool works. It is silent on predictive accuracy — a random coin-flip screener achieves a perfect 1.00 on every ratio while adding zero value, so "fair" cannot be read as "works." Aggregation cuts the other way: pooled citywide rates can hide Simpson's-paradox reversals when applicant-pool composition differs sharply across job families, leaving a tool that looks equitable in aggregate and inequitable in every family you actually staff.

Fifth, audit the auditors' inputs. Vendors choose the audit window, define what counts as "selected," and supply the demographic data themselves — often name-inferred or self-ID at application. Cross-vendor comparisons and year-over-year comparisons within one vendor therefore embed methodological drift invisible in the published summary: two cycles showing "improvement" may reflect a redefined outcome, not a retrained model.

Sixth, the benchmark is internal. An impact ratio measures the tool against its own best-performing group, not against your recruiters. If your own pre-tool human screening produced a 0.74 ratio for a group where the tool posts 0.78, retiring the tool for missing 0.85 makes outcomes worse for that group. Demand the replacement's projected ratio before pulling the plug — retirement is only defensible when the successor clears the bar the incumbent missed.

The status-quo story this section retires: a clean vendor audit — every ratio comfortably above 0.80 — proves the screener fair and legally safe. Each failure mode above breaks that inference independently: the pass can sit inside sampling noise, omit categories federal law protects, describe a model version already replaced, and still trail your own recruiters' record. Read the clean 2026 ratio as a hypothesis to stress-test against this table, not a verdict to file.

Here is a complete Law 144 file small enough to hold in your head — and a tool that fails it twice. A mid-sized New York City employer ran a resume-ranking AEDT over a substantial applicant pool during the prior audit window, spanning men and women across every required race/ethnicity category. The independent auditor published the summary in the 2026 cycle. Read mechanically, the keep-or-retire call makes itself.

```

Frequently Asked Questions

What impact ratio does my AI screener actually need to hit to justify keeping it in 2026?

The statistically defensible keep-line is 0.85, and it only counts when every adequately populated protected category clears it with adequately sized samples per group, sustained across two consecutive audit cycles.

My tool's ratios land between 0.80 and 0.84 — can it stay in production?

Under the 2026 keep-rule that band triggers no disclosure requirement but earns only human-review fallback while a replacement is procured.

Who has to run the Law 144 bias audit, and over what time period?

An independent auditor — never the vendor or the employer — must run it at least annually over the trailing 12 months of the tool's actual NYC use.

How much advance warning do candidates get before an automated tool screens them?

Employers must give candidates notice at least 10 business days before the AEDT screens them, per DCWP's implementing rules.

Why can an audit summary show every ratio above 0.80 while some groups escape scrutiny entirely?

DCWP's rules allow auditors to mark categories with too few candidates as having insufficient data and require selections from candidates of unknown demographic to be reported separately, so a clean-looking summary can simply omit the smallest groups.

If an audit finds bias in my screener, is retiring it the only option?

No — feeding AEQUITAS-discovered inputs back into a model's training set improved measured fairness by up to 94%, so remediation can beat retirement if begun early enough to meet the standard requiring adequate samples sustained across two consecutive audit cycles.

Quick answers

Also worth reading: Local Law 144: Nothing About Your Impact Ratio Stays Still: Local Law 144: Nothing About · AI search tools are making it easier to refine your search and find better information for work: AI search tools are making · Discover top HR tech tools from Reddit's community: Discover top HR tech tools