Rajan is a 45-year-old entrepreneur from a small town in Bihar. He has built a profitable distribution business, has a strong local reputation, and is actively applying for senior roles through an AI-driven job platform.

He never gets shortlisted. Not once.

After months of rejections, a friend in tech takes a look at his profile. The friend tells him quietly: “The algorithm probably doesn’t recognise you. You don’t look like the people it was trained to select.”

Rajan did not know an algorithm was deciding his future. He assumed a recruiter had read his profile and found it wanting. The recruiter never saw it.

When the algorithm learns from biased history

AI systems learn patterns from historical data. The assumption is that historical data reflects good decisions. But what if those decisions were themselves biased — shaped by decades of geographic, socioeconomic, and institutional inequality?

IS Audit Module 6 of the ICAI IS Audit 3.0 Course is explicit: a big problem with AI systems is that their level of goodness or badness depends on how much data they are trained on. Bad data is often associated with ethnic, communal, gender or racial biases. Proprietary algorithms are used to find out information like who gets bail, whose loan is sanctioned. If the bias hidden in the algorithms — which take crucial decisions — goes unrecognised, it could lead to unethical and unfair results.

Rajan’s algorithm had learned that successful candidates, historically, came from certain geographies, certain institutions, and certain career trajectories. It had never been trained to question whether that pattern reflected genuine merit or merely reinforced historical exclusion. The algorithm was confident. The algorithm was wrong. And Rajan had no way of knowing — or challenging — either.

The Puttaswamy dimension — data creates new knowledge about peopleThe Supreme Court’s judgment in Justice K.S. Puttaswamy (Retd.) vs Union of India (2018), included in the project knowledge base, addresses exactly this: the creation of new knowledge complicates data privacy law as it involves information the individual did not possess and could not disclose, knowingly or otherwise.

The job platform’s algorithm created a new piece of knowledge about Rajan — a risk score, a fit score, a ranking — that he did not produce, did not verify, and did not consent to. That score is personal data under Section 2(t) of the DPDP Act: any data about an individual who is identifiable by or in relation to such data. Rajan is identifiable. The score is about him. It is personal data — and its creation and use must have a valid basis.

The DPDP and algorithmic bias — this is a legal obligation, not just an ethical one

Section 10(2)(c)(i) of the DPDP Act requires every Significant Data Fiduciary to conduct a Data Protection Impact Assessment that includes assessment and management of risk to the rights of Data Principals. Rajan’s right to non-discriminatory treatment flows from his fundamental rights under the Constitution and is directly implicated when a biased algorithm systematically excludes him from opportunity based on geographic origin.

Rule 13(3) of the DPDP Rules, 2025 goes further: a Significant Data Fiduciary must observe due diligence to verify that algorithmic software adopted for processing personal data is not likely to pose a risk to the rights of Data Principals. The word “pose a risk” is important. The organisation does not need to wait for Rajan to be demonstrably harmed. The obligation is preventive — to verify, before and during deployment, that the algorithm does not create this risk.

A hiring algorithm trained on historically biased data, operating without bias auditing, and producing systematically skewed outcomes for certain demographic groups poses a risk to Data Principal rights as a matter of its design. The DPDP Act requires that risk to be identified, documented, and mitigated.

The CERT-In AI governance framework — human oversight is mandatory

The CERT-In Guidelines on Secure Adoption and Governance of Artificial Intelligence Systems (Version 1.0, 25 May 2026) identify Human Oversight and Decision Governance as a core control area requiring organisations to: validate AI-generated outputs, restrict fully autonomous critical actions, and maintain auditability and approval mechanisms.

A hiring shortlist generated entirely by an AI model — with no human review of borderline cases, no audit trail of the factors that drove the ranking, and no mechanism to flag demographically anomalous patterns — fails the human oversight standard. AI-assisted decisions are permissible. Fully autonomous, unreviewed decisions that affect an individual’s livelihood are a governance failure.

What Rajan is entitled to — and what the platform owes him

Under Section 11 of the DPDP Act, Rajan has the right to access a summary of what personal data the platform is processing about him and what processing activities are being undertaken. He has the right to know that a risk score or fit score has been generated, even if he does not know to ask for it.

Under Section 13, he has the right to raise a grievance about how his personal data has been processed. If the platform cannot explain, at the individual level, why its algorithm ranked him as it did — and what data drove that ranking — it cannot respond to that grievance in any meaningful way.

The algorithm decided. DPDP says that decision must be auditable, explainable, and subject to human governance. Rajan deserves at least that much.

Disclaimer

The contents of this post are intended for general awareness and informational purposes only. They do not constitute legal opinion, professional advice, consultancy, statutory interpretation, or a recommendation to act in any particular manner.

The Digital Personal Data Protection Act, 2023, related rules, notifications, regulatory guidance and judicial interpretations may evolve from time to time. The applicability of the law may also vary depending on the facts, sector, nature of data processing, organisational role, contractual terms and compliance framework.

Readers should not rely solely on this post for making legal, business, HR, technology, data-processing or compliance decisions. Specific advice from a qualified legal, privacy, cybersecurity, governance or compliance professional should be obtained before acting on any matter discussed.

The author / publisher shall not be responsible for any loss, liability, claim, penalty or consequence arising from reliance on the contents of this post without independent professional advice.

Authors:
This article has been co-authored by CA. Sunil Elayadath and CA. Karthik Narayanan S, Partners of Karthik & Sunil, together with Mr. Dhanesh P. K., Designated Partner, DSK Sustainability Tech.

Note: The images used are AI Generated Images