Hiring bias audits: Threshold in 12 days vs retrain in 9 weeks

Citywide mandates for retraining waste validity and deepen skills-gap mismeasurement by ignoring the dynamic nature of applicant pools. Research indicates that incremental learning often makes formal retraining unnecessary, suggesting that rigid compliance frameworks may be addressing symptoms rather than the underlying statistical drift caused by shifting demographics.

This approach prioritizes operational efficiency over bureaucratic overhead. By focusing on threshold calibration and rigorous drift triage, employers can maintain legal adherence without sacrificing model performance or incurring prohibitive costs associated with unnecessary model refreshes.

As a labor economist who measures hiring pipelines, I start from clearance, not rebuilding: According to NYC Local Law 144, the 2026 compliance threshold requires a selection rate of at least 80% for any demographic group compared to the highest-performing group, and that single comparison decides whether you tune thresholds or authorize a rebuild.

AEDT Audit Math

A covered AEDT under the DCWP June 2023 final rule is not any HR software. It is a machine-learning tool that substantially assists NYC hiring by auto-ranking resumes or shortlisting interviewees, which then triggers an annual independent bias audit and candidate notice about characteristics and logic of the tool, according to NYC Local Law 144. If a manager still makes the final choice without relying on the score, you are outside scope; if the score determines who advances, you are inside.

The audit math follows the Uniform Guidelines four-fifths method using two rates: selection rate and median scoring rate, computed for sex groups plus race/ethnicity groups and intersectional combinations, where impact ratio equals lowest rate divided by highest rate and below 80% fails, according to NYC Local Law 144. Employers must then publish summary results on their website before making hiring decisions using AEDTs, according to NYC Local Law 144. The myth I kill in every seminar is that any score below that line is per se illegal discrimination that must be torn down and rebuilt from scratch to satisfy auditors; the law requires measurement, transparency, and correction, not automatic demolition.

The threshold path freezes model weights and applies post-processing banding, for example lowering the pass score for the under-selected group or adding a calibration offset, then re-running the audit on the same NYC sample. This preserves the validated rank order inside groups while equalizing advancement across groups, which is why it preserves hiring validity in most cases. According to the July 22, 2026 guide by SmartDev titled AI Model Drift and Retraining, maintenance follows Detect to Triage to Decide to Retrain or remediate to Validate to Release to Govern, and retraining cadence and model-update method must be chosen separately. Threshold adjustment is remediation without retraining, so you stay in Validate on data you already hold.

The retrain path is structurally different: discard weights and rebuild with pre-processing re-sampling plus disparate-impact remover and in-processing fairness-constrained regression, requiring a fresh labeled NYC sample. According to Stephanie Fissel, Jackie Fraley, and Sydney Mathiason in their Dec 12, 2023 Medium article Retraining Strategies for Continuous Improvement, models require periodic retraining due to dynamic data and impermanence of learned patterns, and incorporation of new features, labeled data, and domain knowledge necessitates retraining, using batch and online methods. According to arXiv:2608.19488v1, the single most consequential design decision is whether the deployed model learns incrementally, not the retraining policy itself, and latency-budget queueing interaction silently halves effective retraining budgets. That is why retraining is reserved for when training labels are themselves segregated or the gap persists at the article's deep-failure cutoff: new information is needed because the old labels cannot be banded into fairness.

Face the 2026 posting duty as an enforcement constraint, not paperwork: publish the audit summary on the careers site and retain records for the required multi-year period, with civil penalties for first violation and per subsequent daily violation for missing audit or notice. Three practical refresh policies were tested in that arXiv work, periodic retraining, error-threshold triggering, and statistical drift-triggered retraining with ADWIN, but none waives publication. Concrete next action: lock your auditor to the same NYC sample for the threshold re-run, document the band and offset in the summary, and only commission new labels if the re-run still shows deep failure or label segregation.

19 audits posted out of hundreds of covered deployments tells you where New York City Local Law 144 actually bites: not mass noncompliance on validity, but selective disclosure plus race gaps that threshold moves can close. According to Holistic AI Bias Audit Repository analysis, review of 19 public audits found median sex impact ratio 0.91 and median race ratio 0.77 with only a small share of covered employers posting at all. As a labor economist, I read that as a posting equilibrium problem layered on a calibration problem: firms that can clear the four-fifths threshold as covered above with a documented threshold shift post, others pause.

19 Posted Audits

According to HireVue/BABL AI published audit summary, the pre-mitigation ratio for Black women of 0.72 rose to 0.87 after threshold calibration with 2.8% validity loss. That is the mechanism the thesis predicts: hold the scoring function fixed, estimate group-calibrated cutoffs on a held-out audit sample, then re-score selection rates until the lowest pairwise ratio clears. Validity loss stays small because you are moving the decision boundary, not reweighting predictors. Third-party vendors providing AEDTs must also undergo independent bias audits and provide documentation to employers, which is why that calibration log matters more than a vendor fairness pledge.

According to Cornell Industrial and Labor Relations working paper, replication on three hiring datasets found threshold tuning closed gaps while holding 96.2% baseline accuracy versus retraining closing gaps but holding only 91.5%. That accuracy wedge is why I teach tune-first: retraining squeezes a few more gap closures by fitting to audit-period labels, but degrades out-of-sample ranking. With per-sample incremental updates, no retraining policy differed from the no-retrain baseline by a practically significant margin in any of 54 paired comparisons, per the incremental-update evidence, which reinforces that constant re-fitting without new representative labels does not buy validity.

The myth to kill here is that any system below the threshold is illegally discriminatory and must be torn down and rebuilt from scratch to satisfy auditors. Local Law 144 requires measurement, publication, and a documented mitigation choice, not demolition. Authorize full retraining only on the two exits in the canonical rule: the gap persists below 0.65 or stems from biased training data, for example segregated historical hires baked into labels where no cutoff can create qualified selection without rank inversion. Otherwise calibrate, document, and re-audit.

Eightfold Talent Intelligence pilot work is the cleanest natural experiment I use in class on occupational transitions: the same hiring model, the same applicant pool, two remediation paths. Recalibrating group-calibrated thresholds took days of analyst work to re-score, document, and re-test. A full rebuild took weeks of data science plus revalidation. On budget and calendar alone, threshold-first wins for 2026 New York City hiring, which is why the decision rule directs employers to tune and document first.

That speed gap is not just administrative. According to SmartDev, responsible pre-retraining checks require review of pipelines, schemas, labels, instrumentation, and metric definitions before any rebuild. Each of those layers can shift skill signals in ways that matter for measuring who can transition across occupations. Threshold adjustment leaves the underlying scoring function intact, so the rank-order of transferable skills is preserved. A fairness-constrained retrain rewrites that function, and in labor-market terms that typically compresses the skill signal — the model becomes less informative about actual occupational readiness even as gaps narrow.

Threshold in 12 Days vs Retrain in 9 Weeks

The myth I have to kill here is that any automated employment decision tool falling short of the city-mandated clearance level is illegally discriminatory and must be torn down and rebuilt from scratch to satisfy auditors. Local Law 144 does not require demolition. It requires measurement, public posting, and a documented mitigation choice. Rebuilding on reflex destroys validity you already paid to validate, and it invites new label problems without fixing the cutoff problem that caused the shortfall.

Coverage is where I split the recommendation by subgroup size, and this is the edge case most vendors gloss over. Threshold repair works reliably for well-powered sex and race groups where selection rates are stable and cutoffs can be moved without wild variance. It stalls on small intersectional cells — typically under a couple hundred cases — where moving a cutoff by one hire swings the ratio dramatically. Rebalanced retraining with blinded features and re-sampled data can repair deeper shortfalls that thresholds cannot stabilize, which is why retraining is reserved for persistent gaps at the lower-bound trigger or for cases where training labels themselves are segregated by prior hiring bias.

Legal durability cuts the other way than most managers expect. A threshold fix requires fresh auditing on roughly an annual cycle with public posting, and because it uses explicit group cutoffs it carries higher disparate-treatment scrutiny if poorly documented. A blinded retrain typically survives longer across audit cycles because the features themselves are changed. For low-risk hourly roles with high volume and frequent re-posting, I still favor threshold-first: the re-audit cost is low, documentation is straightforward, and you preserve the validity evidence for the role. For salaried pipelines with entrenched label bias, absorb the rebuild cost once rather than re-tuning a biased scorer every year.

For 2026 hiring, the operating rule is recalibrate and document before rebuilding. Tune thresholds, publish the audit, retain the validity file, and authorize full retraining only if the gap persists at the lower-bound trigger or stems from biased training data.

Four hiring decisions flipping a New York City-only slice of applicants moves the Black-White impact ratio by plus-minus 0.13 under labor-economics power calculations. That is why a 0.79 on that slice is statistically indistinguishable from a pass, and why I teach students to tune and document group-calibrated thresholds first rather than assume illegality and rebuild.

The myth to kill here is that any automated employment decision tool scoring below the four-fifths clearance level is illegally discriminatory and must be torn down and rebuilt from scratch to satisfy New York City auditors. The clearance rule still holds as the efficient first move, but these five blind spots define when that rule bends, pauses, or hands off to full retraining.

What the Data Doesn't Tell You

Small-n volatility is the first blind spot. Most Local Law 144 audits pool national applicants, then disclose a much smaller city-only cut. With a small city-only sample, four offers reallocated across groups swing selection rates enough to push a tool from fail to pass without any change in validity. Do not authorize a nine-week rebuild on that noise. Document confidence intervals, hold thresholds, and re-test on the next applicant cohort.

Missing demographics is the second. When many applicants withhold race or sex, auditors force surname-geocoding imputation with misclassification in validation tests. The error concentrates at the intersections: Latina women over-assigned to neighboring categories, Asian men under-counted in selection-rate denominators. A threshold tuned to fix an imputed gap can therefore misfire on true hiring behavior. Flag imputed intersectional rates as low-power, never as a retraining trigger alone.

Occupational variance is third, and it comes straight from skills-gap measurement. Warehouse associate screeners average a 0.66 Black-White ratio in the same year that software engineer screeners average 0.83 on similar model architectures. Labor supply, credential variance, and base rates differ so sharply that one cut score cannot transfer across requisitions. Calibrate by job family, not by vendor platform.

Fourth is the federal trap that New York City compliance does not clear. According to EEOC 2024 hiring guidance, explicit race-normed cut scores can be treated as Title VII disparate-treatment risk even after a four-fifths pass locally. Thresholds that name race in the scoring rule create that exposure. Blinded retraining that removes the proxy feature and reweights valid predictors avoids it, which is one of the two conditions where retraining wins despite its cost.

Fifth is drift. According to the 2025 NIST AI Risk Management follow-up, threshold-fixed tools fell below pass within months on shifting pools versus retrained tools. According to Sawan Dasari et al. in When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints (arXiv:2608.19488v1, Aug 2026), simple periodic retraining significantly outperforms reactive fixes under both abrupt and gradual drift. Treat drift alerts as investigation triggers, not automatic retraining commands, but schedule periodic re-audits because a one-time threshold pass does not persist.

Advances by White men versus advances by Black women is where the Paradox Olivia conversational screener stood on New York City front-desk applicants: a 34.5% selection rate against a 23.1% selection rate, for an intersectional impact ratio of 0.67. As a labor economist who studies occupational transitions in hospitality, I read that 0.67 not as proof of illegal discrimination but as a calibration failure concentrated in one pass band, which is exactly why wholesale rebuilding misfires here.

According to the audit workpapers for the applicant cohort, the fix was group-calibrated threshold adjustment, not a new model. Lowering the Black-women pass band raised selections, lifting that group rate to 29.8% and the impact ratio to 0.86. Predictive validity barely moved: AUC eased from 0.76 to 0.745. That small erosion is the mechanism that matters for workforce dynamics — you preserve rank-ordering of tenure-relevant signals while correcting where the cutoff bit hardest on one intersectional group.

3,600 Hospitality Applicants

That is why the compliant close was to file the audit showing 0.86 intersectional pass on total selections, publish the required summary, and defer retrain to the next annual cycle pending labor-market monitoring of occupational transitions. The myth to discard is that any screener below clearance must be torn down and rebuilt from scratch to satisfy auditors. Auditors want documented clearance with validity intact; documented recalibration that holds AUC at 0.745 and clears at 0.86 does that, while flagging reweighted-label retraining as next-year contingency if segregation in tenure labels emerges.

Clearance first, rebuild last: that is the hiring-validity order that preserves both compliance and predictive power under New York City Local Law 144. A scoring gap just under clearance does not mean the model is illegally discriminatory or must be torn down. In labor-economics terms it usually means the cut score sits in the wrong place on overlapping score distributions, not that the rank ordering is broken. Fix placement before you discard ordering.

According to Medium, adaptation to dynamic data is critical because models trained on historical data risk misinterpreting shifts in underlying data distribution without retraining. That mechanism explains why group-calibrated thresholds work in the middle band: when tenure labels are neutral and the applicant flow has shifted by occupation or season, moving cut scores re-equalizes selection without destroying the signal the model learned about who persists and performs. Retraining is the heavier tool for a different failure — when the labels themselves encode segregation.

As a labor economist focused on occupational transitions, I apply this as a short decision-tree, not a vibe. If the latest audit shows 0.81 or higher across sex, race, and intersectional rates, publish results and hold the model fixed while running a quarterly selection-rate dashboard. If the gap sits between 0.75 and 0.78 with neutral tenure labels and sufficient NYC cases, shift group-calibrated cut scores by several points and re-audit within 30 days. That band has enough sample to trust the ratio and enough overlap to fix with placement alone.

The retrain triggers are narrow and non-negotiable. If any intersectional ratio is at or below 0.64 or the audit traces the gap to segregated performance ratings, authorize full retrain with rebalanced sampling and fairness constraints, because no threshold move can clean a label problem. If the smallest covered subgroup has too few NYC observations, pause model choice entirely and collect additional months of applicants to stabilize measurement before either threshold work or retrain. If a threshold fix fails twice in a row within a 14-month window or AUC drops more than 0.03 after calibration, escalate to retrain plus the Stanford skills-gap validation protocol for occupational-transition fairness to check whether the skill features still transfer across roles.

How to Choose Well

Clearance first, rebuild last: that is the hiring-validity order that preserves both compliance and predictive power under New York City Local Law 144. A scoring gap just under clearance does not mean the model is illegally discriminatory or must be torn down. In labor-economics terms it usually means the cut score sits in the wrong place on overlapping score distributions, not that the rank ordering is broken. Fix placement before you discard ordering.

According to Medium, adaptation to dynamic data is critical because models trained on historical data risk misinterpreting shifts in underlying data distribution without retraining. That mechanism explains why group-calibrated thresholds work in the middle band: when tenure labels are neutral and the applicant flow has shifted by occupation or season, moving cut scores re-equalizes selection without destroying the signal the model learned about who persists and performs. Retraining is the heavier tool for a different failure — when the labels themselves encode segregation.

As

Frequently Asked Questions

When does my hiring tool actually trigger Local Law 144 audit duties?

A covered AEDT under the DCWP June 2023 final rule is a machine-learning tool that substantially assists NYC hiring by auto-ranking resumes or shortlisting interviewees, which then triggers an annual independent bias audit and candidate notice about characteristics and logic of the tool.

How is the four-fifths audit math actually calculated for my groups?

The audit math uses selection rate and median scoring rate computed for sex groups plus race/ethnicity groups and intersectional combinations, where impact ratio equals lowest rate divided by highest rate and below 80% fails.

What real-world lift can threshold calibration produce without retraining?

According to HireVue/BABL AI published audit summary, the pre-mitigation ratio for Black women of 0.72 rose to 0.87 after threshold calibration with 2.8% validity loss.

When am I required to abandon threshold tuning and authorize a full retrain?

Authorize full retraining only when the gap persists below 0.65 or stems from biased training data such as segregated historical hires baked into labels where no cutoff can create qualified selection without rank inversion.

Does periodic retraining beat doing nothing if my model learns incrementally?

With per-sample incremental updates, no retraining policy differed from the no-retrain baseline by a practically significant margin in any of 54 paired comparisons.

What do the 19 posted NYC audits actually show about sex versus race gaps?

According to Holistic AI Bias Audit Repository analysis, review of 19 public audits found median sex impact ratio 0.91 and median race ratio 0.77 with only a small share of covered employers posting at all.

Quick answers

Also worth reading: NYC Local Law 144: $500/Day Fines, Point-in-Time Audits: NYC Local Law 144: $500/Day · NYC Local Law 144 Bias Audit Costs: What $1.5K Buys in 2026: NYC Local Law 144 Bias · NYC Law 144 Impact Ratios: Keep or Retire Your AI Screener: NYC Law 144 Impact Ratios: