Table of Contents
Most research on LLM privacy has focused on the wrong problem, according to a new paper by researchers from Carnegie Mellon University and Northeastern University. The authors argue that while most technical studies target data memorization, the biggest risks come from how LLMs collect, process, and infer information during regular use.
A narrow view of privacy research
The study reviewed 1,322 AI and machine learning privacy papers published between 2016 and 2025. It found that 92 percent of them focused on just two areas: training data leakage and protection against direct chat exposure. The remaining 8 percent dealt with other risks such as inference attacks, context leakage through LLM agents, and large-scale data aggregation.
The authors say this imbalance leaves organizations unprepared for more subtle privacy violations that are harder to detect or control. They argue that the privacy landscape extends well beyond model training and includes every stage of the LLM lifecycle, from data collection to deployment.
Co-author Niloofar Mireshghallah, Assistant Professor at Carnegie Mellon University, told Help Net Security the lack of research attention on these areas is rooted in deeper systemic barriers. “There’s a persistent lag between security and technology research and policy development, with policy typically trailing behind technological advances. This creates a vacuum where these risks remain unaddressed,” she explained.
Mireshghallah added that there is also a cultural problem within the technical research community. “Privacy work involving human factors is often dismissed as non-technical or unworthy of serious attention. Many technologists view these concerns as someone else’s problem, which leads to a negative bias where technologists blame users rather than acknowledging systemic design issues,” she said.
She pointed out that privacy research often happens in silos, with limited cross-pollination between AI, policy, and human-computer interaction fields. “LLM researchers rarely engage with publications from other relevant venues, creating knowledge silos. Combined with a lack of institutional incentives to pursue this interdisciplinary work, these factors create a perfect storm of neglect for these critical privacy risks,” Mireshghallah said.
Beyond memorization: the new privacy map
The researchers propose a taxonomy of five categories of privacy incidents. The first, and most studied, is training data leakage through regurgitation of text the model was trained on. The second is direct chat leakage that occurs when stored conversations are exposed through weak policies or compromised infrastructure.
The remaining three categories are less explored but growing in importance. These include indirect context leakage through tool or agent integrations, indirect attribute inference where models deduce sensitive traits from ordinary data, and direct aggregation of public information into detailed personal profiles.
These categories show how privacy incidents can emerge even when no explicit data breach has occurred. For example, models can infer a person’s location or background from an image or short text exchange. Aggregation risks increase when systems combine public data from many sources to answer detailed personal questions.
Data collection practices under scrutiny
The paper highlights how current LLM ecosystems collect and retain more user data than many users realize. Opt-out controls are often buried or ineffective, and feedback features can trigger long-term data storage even for users who believe they have opted out.
According to the researchers, some LLM services now store user data for several years, with feedback or safety systems creating exceptions that allow retention beyond stated limits. They point out that legal requirements or security flags can override deletion requests, leaving many users without practical control over their data.
The authors describe this as a pattern of “privacy erosion disguised as choice,” where design and policy decisions systematically favor data collection. For CISOs, this highlights the importance of verifying vendor retention practices and understanding how user interactions are processed, flagged, and stored.
Indirect privacy risks from agents and retrieval systems
The paper also warns that as LLMs evolve into connected systems with retrieval and agent capabilities, new privacy attack surfaces are emerging. Retrieval-augmented generation systems pull information from databases, APIs, and other sources that may contain sensitive or proprietary data.
Autonomous agents can amplify these risks by combining permissions, accessing external systems, or misinterpreting user intent. The study notes that even without malicious actors, users may unknowingly expose private data because they cannot see or control how an agent gathers or shares information.
The authors caution that expecting users to monitor these systems themselves is unrealistic. Human oversight often fails to catch privacy violations, especially when agents act quickly or process large volumes of data.
Bridging research, policy, and practice
Mireshghallah said the way forward requires a structural shift in how privacy research and policy are designed. “At the research funding level, grant proposals should explicitly require interdisciplinary collaboration across technical, social, and policy domains, with representation from diverse scientific fields as a prerequisite for funding,” she said.
She also argued that regulators and companies must be pushed to adopt incentive-based frameworks that prioritize privacy. “We need regulatory frameworks that create friction in data gathering practices, forcing companies to justify collection and retention on a strict need-to-know basis. This could be coupled with monetary incentives for privacy-preserving practices and penalties for violations,” she said.
Academic and industry incentives, she added, must evolve as well. “We need to restructure academic and industry incentives to reward cross-disciplinary work that addresses these sociotechnical challenges, rather than treating them as peripheral concerns.”
“Our existing privacy frameworks were built for institutional accountability, not for managing the human-to-human risks that intelligent agents are starting to amplify. That’s the paradigm shift we’re starting to see unfold,” said Tianshi Li, Assistant Professor at Northeastern University and co-author of the research.
A call for broader accountability
The paper argues that privacy protection should not rely on individual user choices alone. Instead, LLM providers and policymakers should adopt mechanisms that make privacy expectations explicit and enforceable across technical and organizational layers.
The findings suggest that evaluating LLM privacy should go beyond standard data retention and encryption reviews. The researchers encourage organizations to consider where and how data flows through connected systems, how user consent is gathered, and what happens when that consent fails.
