Prompt injection is inevitable. Enterprise AI security is about what happens next. 

Artificial intelligence has rapidly evolved from an internal productivity tool into a customer-facing technology that shapes how organizations engage with employees, partners, and buyers. AI assistants now answer support questions, guide purchasing decisions, surface enterprise knowledge, and increasingly act as digital representatives of a company’s brand. As these systems become more capable and more visible, they also become more attractive targets for abuse. 

One of the most widely discussed threats is prompt injection, an attack in which someone attempts to manipulate an AI assistant into ignoring its intended instructions. High-profile chatbot failures have demonstrated how a single manipulated interaction can generate embarrassing or misleading responses that quickly spread across social media and damage customer trust. While these incidents often make headlines, they also raise a broader question for enterprise leaders: how should organizations evaluate the security of AI systems that interact directly with customers? 

Understanding prompt injection 

Prompt injection is a technique designed to influence the behavior of an AI assistant by introducing instructions that conflict with its original purpose. In a direct attack, the malicious instruction is entered explicitly into the conversation, such as asking the assistant to ignore its previous instructions or reveal confidential information. Indirect prompt injection is more subtle, embedding hidden instructions inside documents, web pages, support tickets, or other content that the AI retrieves while answering a legitimate question. 

Unlike traditional software, which clearly separates executable code from user data, large language models process everything as language. System instructions, user questions, retrieved documents, and hidden commands all become part of the same context window that the model interprets. Security researchers and standards bodies such as NIST have highlighted this as one of the defining security challenges of generative AI. Because the model reasons over all of this text together, distinguishing between trusted instructions and malicious ones is fundamentally difficult. 

That is why no AI vendor can honestly claim complete immunity to prompt injection. Researchers continue to improve model robustness, but there will always be new ways for attackers to phrase requests or conceal instructions. The challenge is not unique to one model or one provider… It’s an inherent characteristic of today’s large language models. 

For enterprises, this shifts the focus away from asking whether prompt injection can be eliminated. A far more practical question is what happens if an attacker succeeds in influencing the model’s reasoning. 

What are attackers actually trying to achieve? 

An attacker may try to convince the assistant to reveal its hidden system prompt or internal instructions, hoping to better understand how it operates. Others attempt to extract confidential information, such as customer records or internal documents, by persuading the AI to ignore its security instructions. As AI assistants become increasingly connected to enterprise applications, attackers may also attempt to manipulate them into making unauthorized API calls, interacting with external systems, or retrieving sensitive data from connected knowledge bases. 

Indirect prompt injection introduces another layer of complexity. A malicious instruction hidden inside an uploaded document or web page may tell the AI to ignore the user’s question, disclose confidential information, or perform some other unauthorized action. To the language model, those hidden instructions can appear just as legitimate as the user’s request. 

The growing use of AI agents that can access tools, enterprise systems, and external services makes these attacks even more significant. Once an AI can do more than simply generate text, the consequences of manipulation extend beyond incorrect answers to potentially affecting real business processes. That is why enterprise AI security must extend far beyond the language model itself. 

Security should not depend on perfect AI 

Traditional cybersecurity has never relied on software behaving flawlessly. Organizations assume that employees will occasionally click phishing emails, applications may contain vulnerabilities, and attackers will continually search for new techniques. Rather than attempting to eliminate every possible failure, security architectures are designed to limit the consequences when failures inevitably occur. 

The same principle applies to enterprise AI. Instead of assuming that a language model will always make the correct decision, modern AI platforms increasingly treat the model itself as an untrusted component. Sensitive capabilities are protected by deterministic security controls that operate independently of the model’s reasoning. Even if the AI produces an unexpected response or follows a malicious instruction, those controls continue to govern what the system can access and what actions it can perform. 

This philosophy reflects a long-established cybersecurity concept known as defense in depth. Organizations do not rely solely on a firewall to protect their infrastructure. They combine identity management, access controls, encryption, monitoring, network segmentation, logging, and endpoint protection because any individual layer can eventually fail. Security comes from multiple independent controls working together. 

The same thinking is increasingly recommended for enterprise AI. Rather than relying on the model alone to recognize malicious prompts, organizations should combine grounded retrieval, least-privilege access, sandboxed execution, deterministic guardrails, monitoring, governance, and comprehensive auditing. If one layer fails, the others continue limiting what the attacker can accomplish. 

Grounding reduces both hallucinations and attack surface 

One of the most effective ways to improve both accuracy and security is to narrow the role of the AI assistant. Rather than generating answers from everything it learned during training, enterprise AI should answer questions using only approved organizational knowledge. 

This retrieval-based approach significantly reduces hallucinations because the assistant is expected to cite trusted information rather than improvise. It also narrows the attack surface.  

When no relevant information exists in approved sources, the assistant simply declines to answer instead of attempting to invent one. 

Grounding does not eliminate prompt injection, but it does reduce opportunities for attackers to steer conversations into unsupported topics or persuade the model to fabricate information. 

Permissions should always follow the user 

A second foundational principle is least-privilege access. An AI assistant should never possess broader permissions than the individual interacting with it. Every search, retrieval operation, and tool invocation should execute within the requesting user’s existing authorization scope. 

This means that even if an attacker manages to manipulate the AI’s reasoning, the assistant still cannot retrieve documents, records, or systems that the user could not already access. Those resources remain entirely outside the AI’s visibility. Instead of relying on the model to decide what should remain confidential, the underlying platform enforces access controls before information is ever presented to the model. 

Kaltura’s Genie follows exactly this model. Every retrieval and tool call executes on behalf of the requesting user and within that user’s permission scope. The AI never receives elevated credentials or broader access than the individual using it. As a result, even a successful prompt injection cannot expose information that the user was never authorized to see. 

Sandboxing and controlled access limit the blast radius 

Many organizations extend AI assistants with custom workflows, APIs, and enterprise integrations. While these capabilities unlock significant business value, they also create additional opportunities for abuse if left unrestricted. 

Kaltura addresses this through sandboxed execution and tightly controlled external connectivity. Custom logic executes inside isolated environments without unrestricted access to internal infrastructure, file systems, or enterprise networks. Likewise, external tools, APIs, and web search are disabled by default and can be enabled only under controlled conditions. Administrators can further restrict which domains and services Genie is permitted to access, ensuring that external integrations remain tightly governed. 

These controls recognize an important reality: even if an attacker influences what the model wants to do, that should not automatically determine what the system is allowed to do. 

Slowing attackers down and making every action visible 

Not every defense aims to stop an attack outright. Some are designed to reduce the speed and scale at which attacks can occur. 

Public-facing AI assistants are particularly attractive targets for automated probing, where attackers submit thousands of prompt variations in search of weaknesses. Rate limiting and web application firewalls help slow these attempts, making large-scale experimentation significantly more difficult while preserving a smooth experience for legitimate users. 

Equally important is visibility. Every interaction with Genie is logged, including the user’s question, retrieved sources, generated response, and supporting citations. These audit trails allow organizations to investigate unusual behavior, improve answer quality, demonstrate compliance, and understand exactly what occurred if an incident is suspected. Without comprehensive logging, organizations may never know that an attempted prompt injection occurred in the first place. 

Governance is becoming as important as technology 

As AI becomes part of critical business processes, organizations are evaluating vendors on governance as much as technical capability. 

Security certifications such as ISO/IEC 42001 and SOC 2 demonstrate that AI systems are supported by formal processes for risk assessment, validation, monitoring, incident response, and continuous improvement. These frameworks reinforce that enterprise AI security extends well beyond the language model itself to encompass the people, policies, and operational controls surrounding it. 

Combined with infrastructure isolation, zero-retention policies, and independent audits, governance provides another layer of confidence that AI systems behave consistently, securely, and responsibly. 

The future of enterprise AI depends on architecture 

Prompt injection will not disappear as language models improve. If anything, it will become more important as AI evolves from conversational assistants into autonomous agents capable of interacting with business systems, executing workflows, and taking actions on behalf of users. 

As those capabilities expand, organizations will increasingly evaluate AI platforms not only on how intelligent they appear, but on how securely they operate. Questions about permission enforcement, sandboxing, auditability, governance, and infrastructure isolation will become just as important as response quality or reasoning ability. 

Ultimately, enterprise AI security is not about building a model that can never be manipulated. It is about designing an architecture that assumes manipulation is possible and ensures that, even when an attacker gets creative with a prompt, the consequences remain tightly contained. That is the philosophy behind Kaltura’s approach: trust neither the input nor the model, enforce least privilege at every layer, and ensure that a compromised conversation never becomes a compromised system.