INTRODUCTION
Generative AI and DPDP compliance represent the most consequential frontier in India’s data protection landscape — and the one that is evolving fastest.
Generative AI refers to systems — large language models (LLMs), image generators, voice synthesisers, code generators, and multimodal models — that are trained on vast quantities of data and can produce new content: text, images, audio, video, and code that did not previously exist. These systems are already embedded in enterprise workflows across India: drafting documents, summarising records, answering customer queries, analysing contracts, generating reports, and supporting HR processes.
IS Audit 3.0 by ICAI identifies data privacy and security as a primary AI risk, noting that most AI applications depend on massive volumes of data — often sensitive and personal in nature — to learn and make intelligent decisions. As machine learning systems become more capable, they also become more capable of learning, retaining, and inadvertently reproducing the personal data on which they were trained or with which they were prompted.
IS Audit Standard 420 (ICAI ISAS) goes further, acknowledging that generative AI models present non-deterministic outputs — meaning the same input may produce different outputs across sessions. For AI models with these characteristics, it notes, reproducibility requires that parameters, prompts, and methodology be documented sufficiently to allow independent verification. This observation is not merely an audit standard. It is a compliance architecture requirement: if a generative AI system cannot reproduce how a personal data output was generated, the Data Fiduciary deploying it cannot adequately defend its processing decisions under the DPDP Act.
CERT-In’s Blueprint on AI-Assisted Cyber Threats (CIGU20260002, May 2026) adds a threat dimension that no organisation can afford to ignore: generative AI is being actively leveraged by threat actors to generate highly convincing phishing content, deepfake voice and video fraud, synthetic identities, and spear phishing campaigns that bypass traditional awareness-based detection — all of which directly target personal data of Indian citizens. Consequently, generative AI is not merely a compliance risk as a tool of enterprise productivity. It is simultaneously the instrument of some of the most sophisticated personal data threats India’s digital ecosystem has ever faced.
Against this backdrop, the DPDP Act, 2023 applies with full force — to every organisation that deploys generative AI on personal data, to every generative AI vendor that processes personal data on behalf of Indian organisations, and to every system that generates outputs touching identifiable individuals. The compliance deadline is 13 May 2027.
Three stories ground this in the reality of Indian organisations today.
Story 1 — Vikram’s HR Query and the LLM That Remembered Everyone
The Scenario
Vikram is an HR manager at a large IT services firm in Bengaluru. His company had deployed a generative AI chatbot — integrated with its internal HR system — to help managers answer HR policy questions, draft performance review language, and retrieve employee records. The system was built using a commercially available LLM, configured with access to the company’s HR data warehouse containing salary records, performance ratings, disciplinary histories, medical leave records, and confidential appraisal notes for approximately twelve thousand employees.
Over several months, Vikram and other managers used the system extensively. Gradually, however, a pattern emerged. When managers queried the chatbot about HR policies unrelated to specific employees, the model’s responses occasionally incorporated specific employee details — names, salary ranges, performance descriptors — drawn from the training context it had been given, even when those details were not requested and not relevant to the query.
In one instance, a senior manager querying about leave policy received a response that referenced a specific employee’s medical leave pattern as an illustrative example — without Vikram having requested that information. That employee had not been informed that her medical leave data was part of the AI system’s accessible context. She had not consented to its use as illustrative material in chatbot responses to other managers.
The DPDP Act Position
This scenario reveals a compliance failure that is specific to generative AI deployments: the gap between data that is technically accessible to the model and data that has lawful authorisation to appear in the model’s outputs.
First — Processing without lawful basis under Section 4. Section 4(1) of the DPDP Act permits processing only for a lawful purpose for which the Data Principal has given consent, or for certain legitimate uses under Section 7. Medical leave data, performance ratings, and salary information were collected for HR administration — a specific, documented purpose. Incorporating this data as illustrative context in chatbot responses to third-party managers is a materially different processing activity. It has neither a consent basis nor a legitimate use basis. The data appeared in the output not because it was requested but because the model’s context window encompassed it and its generation logic surfaced it as relevant. That is still processing — and it is processing without a lawful basis.
Second — Notice failure under Section 5 and Rule 3. The employee whose medical leave data appeared in the chatbot response had never been informed that her personal data would be processed by an AI system for the purpose of generating responses to queries by other managers. Section 5(1) requires that every Data Principal be given notice of the personal data to be processed and the purpose for which it will be processed. Rule 3 of the DPDP Rules requires that this notice be in clear and plain language, with an itemised description of personal data and a specific description of its processing purposes. “Your data will be used for HR administration” does not cover “your data may appear in AI-generated responses to your colleagues’ policy queries.”
Third — Rule 13(3) algorithmic software due diligence obligation. Rule 13(3) of the DPDP Rules places a specific obligation on Significant Data Fiduciaries to observe due diligence to verify that technical measures — including algorithmic software — adopted for storing, updating, or sharing personal data are not likely to pose a risk to the rights of Data Principals. A generative AI system configured with access to twelve thousand employee records, with no output filtering controls, no data minimisation at the inference layer, and no audit trail linking specific outputs to specific data inputs, is an algorithmic architecture that poses a risk to Data Principal rights. Due diligence requires its assessment before deployment — not after an incident surfaces.
Fourth — IS Audit Standard 420 (ICAI ISAS) governance failure. IS Audit Standard 420 requires that organisations deploying AI tools establish governance controls covering approval, validation, reproducibility, accreditation, and oversight. For AI models with non-deterministic outputs — which all current LLMs are — this requires documenting parameters, prompts, and methodology sufficiently to allow independent verification. Vikram’s company had no mechanism to trace which data inputs contributed to any specific chatbot output. When the problematic response appeared, there was no audit trail to investigate it, no log linking the output to the employee’s data, and no governance framework to determine whether a personal data breach had occurred.
IS Audit Standard 430 (ICAI ISAS, Section 3.3) requires that auditors understand the entire lifecycle of personal data — from collection through storage, processing, sharing, and eventual erasure. An LLM deployed on an HR data warehouse has a data lifecycle that extends into every inference it generates. Without lifecycle mapping specific to the AI layer, ISAS 430 compliance cannot be demonstrated.
The CERT-In Dimension
CERT-In CIGU20260002 (May 2026, Section 12) specifically identifies sensitive data leakage as a primary adversarial risk to AI systems. It notes that AI-assisted offensive tooling can target AI inference systems and AI-integrated workflows — meaning that an LLM with broad access to employee personal data is not merely a compliance liability. It is an attack surface. An adversary who manipulates the model’s input — through prompt injection — can potentially extract personal data that the model has in context, without any system alert being generated.
CERT-In’s framework requires organisations to restrict upload of sensitive information to public AI platforms, define acceptable usage policies, and implement access control and authentication on AI system workflows. Applied to enterprise LLM deployments, these requirements mean that personal data context must be scoped and governed — not simply made available to the model in bulk.
The Compliance Fix
Organisations deploying LLMs or generative AI systems on employee or customer personal data must:
→ Scope the model’s data access to the minimum necessary for each specific use case. An LLM answering HR policy questions does not need access to twelve thousand employees’ disciplinary histories and medical leave records simultaneously.
→ Issue a specific, itemised notice under Rule 3 to every Data Principal whose personal data forms part of the model’s accessible context — identifying the AI system as a processing activity and the inference generation as a purpose.
→ Implement output filtering controls at the inference layer: if the model’s response includes personal data not requested by the querying user, the response must be intercepted, logged, and reviewed before delivery.
→ Maintain an AI processing register that links every LLM deployment to the personal data categories it can access, the purposes for which it was deployed, and the consent records that authorise that processing.
→ Conduct a DPIA under Rule 13 before deploying any generative AI system on personal data at scale, with specific attention to the risk of unintended data surfacing in model outputs.
→ Implement audit logging at the inference level: every query and every response must be logged in a form that allows post-hoc investigation if a personal data concern is raised.
Vikram’s company gave their AI system a broad memory and no output accountability. Under the DPDP Act, the outputs that memory generated were processing events — each one requiring a lawful basis.
The compliance deadline is 13 May 2027. An AI system with a broad memory and no output accountability is not a productivity tool. It is an uncontrolled personal data processor.
(to be continued)
Disclaimer
The contents of this post are intended for general awareness and informational purposes only. They do not constitute legal opinion, professional advice, consultancy, statutory interpretation, or a recommendation to act in any particular manner.
The Digital Personal Data Protection Act, 2023, related rules, notifications, regulatory guidance and judicial interpretations may evolve from time to time. The applicability of the law may also vary depending on the facts, sector, nature of data processing, organisational role, contractual terms and compliance framework.
Readers should not rely solely on this post for making legal, business, HR, technology, data-processing or compliance decisions. Specific advice from a qualified legal, privacy, cybersecurity, governance or compliance professional should be obtained before acting on any matter discussed.
The author / publisher shall not be responsible for any loss, liability, claim, penalty or consequence arising from reliance on the contents of this post without independent professional advice.
Authors: This article has been co-authored by CA. Sunil Elayadath and CA. Karthik Narayanan S, Partners of Karthik & Sunil, together with Mr. Dhanesh P. K., Designated Partner, DSK Sustainability Tech.