What is AI Bias Testing?

Definition

Bias testing is the technical work that measures whether outcomes differ across demographic or other sensitive slices. It supports formal audits where statutes require them and underpins discrimination reviews in high-risk AI programs. NYC Local Law 144 requires annual independent bias audits for covered hiring AEDTs (in effect since July 2023). SB24-205 historical Colorado AI Act controls paired impact assessments with discrimination reviews; after SB26-189, those artifacts remain useful governance evidence rather than the current Colorado minimum-law checklist. California FEHA Civil Rights Council rules treat anti-bias testing as central evidence in employment ADS discrimination claims. Illinois Human Rights Act AI amendments bar AI with discriminatory effects in employment (effective January 1, 2026). EEOC technical assistance applies Title VII disparate-impact concepts to algorithmic selection tools; the FTC enforces Section 5 against unfair or deceptive AI fairness claims.

See also our AI compliance glossary for short definitions of common terms.

Which regulations require this

• Colorado ADMT / AI Act (SB 26-189)

  Colorado SB 26-189 repeals and reenacts SB 24-205 into an automated decision-making technology (ADMT) framework for c...

  COAI-specificHighEnacted (pending)
• California FEHA regulations on automated decision systems (Civil Rights Council)

  California Civil Rights Council regulations apply FEHA's anti-discrimination framework to automated decision systems...

  CAAI-specificHighIn effect
• Illinois Human Rights Act (HB 3773 AI amendment)

  Illinois HB 3773 amends the Illinois Human Rights Act to prohibit employers from using AI that has the effect of subj...

  ILAI-specificMediumIn effect
• EEOC Guidance on AI in Employment Selection

  EEOC technical assistance documents explain how existing Title VII and ADA obligations apply to AI and algorithmic em...

  FEDERALFederalMediumIn effect
• NYC Local Law 144 (Automated Employment Decision Tools)

  NYC Local Law 144 requires employers and employment agencies using automated employment decision tools for hiring or...

  NYCAI-specificMediumIn effect
• FTC Enforcement Policy on AI and Algorithmic Fairness

  FTC enforces Section 5 of the FTC Act against deceptive and unfair AI practices. Key areas: unsubstantiated AI market...

  FEDERALFederalMediumIn effect
• HUD AI Guidance in Housing

  Fair Housing Act disparate impact standard applies to AI-driven tenant screening, lending algorithms, and property va...

  FEDERALFederal guidanceLowIn effect
• NIST AI Risk Management Framework (AI RMF 1.0)

  NIST AI RMF is a voluntary framework used as a practical benchmark by regulators and lawmakers. NIST released AI RMF...

  FEDERALFrameworkLowIn effect
• DOL AI in Workplace Guidance

  Non-binding principles for AI in the workplace covering transparency, human oversight, informed consent, data protect...

  FEDERALFederal guidanceLowIn effect

Which states reference this obligation

What you should do next

• Define the outcomes and populations to compare under each statute that may apply.
• Gather statistically sound samples with counsel on data use.
• Run disparity tests with agreed thresholds for escalation.
• Document mitigations when results exceed your internal or regulatory triggers.
• Record who ran the test, when, and what changed afterward.