Known AI inventory
Controlled data access
Limited model permissions
Tested AI workflows
Accountable monitoring
AI IS PART OF A LARGER APPLICATION
The model is one component of the security boundary
An AI feature may combine a user interface, application code, prompts, retrieval systems, external models and connected business tools. Security decisions require a view of the full system and the trust relationships between these components.
"Model behaviour matters. System permissions determine the consequence."
User interaction
Prompts, uploaded files, conversation history and identity context.
Application and orchestration
System instructions, workflow logic, agents, memory and output handling.
Data and retrieval
Documents, embeddings, vector databases, search services and access filters.
Model and infrastructure
Models, APIs, hosting environments, fine-tuning services and model configurations.
Connected tools
Email, databases, ticketing, code execution, payments and business applications.
TRACE DATA AND AUTHORITY
Understand what enters the system and what can happen next
01
User
Identity, prompt and uploaded content
Can the user be trusted to access the requested feature and data?
02
Application
Session, instructions and business context
Are user and system instructions kept separate and protected?
03
Retrieval
Queries, documents and metadata
Does retrieval enforce the user's actual permissions?
04
Model
Prompt context and model output
Which provider receives the data, and how is that data handled?
05
Validation
Generated content and proposed actions
Is the output checked before it reaches another system?
06
Tool
API call, command or business action
Does the tool independently enforce identity, scope and approval?
07
Record
Logs, decisions and response evidence
Can the organisation investigate what happened?
KNOW WHERE AI IS USED
Create an inventory that supports real risk decisions
A software inventory may show that an AI service exists. An AI register should also explain its purpose, users, data, provider, connected tools and the consequence of an incorrect or manipulated output.
| AI system | Business owner | Purpose | Data used | Provider or model | Action level |
|---|---|---|---|---|---|
| Internal assistant | IT Operations | Staff knowledge queries | Internal documents | Third-party LLM | Read and summarise |
| Customer-facing AI feature | Product team | Customer support responses | Customer records | Hosted model API | Respond and route |
| Automated workflow agent | Operations | Process automation | Business systems | Internal model | Execute actions |
Classification prompts
Does the system handle sensitive information?
Does it make or influence a decision?
Can it initiate an action?
Is it available to external users?
Does it rely on a third-party model?
Can users upload untrusted content?
Is human approval required?
PRIORITY EXPOSURES
Risks created by language, data access and delegated authority
Prompt injection
Instructions within user input, documents or retrieved content may alter the intended behaviour of an AI system.
Excessive agency
An AI system with broad permissions may perform damaging actions after a manipulated, ambiguous or incorrect output.
Sensitive information disclosure
Prompts, retrieval results, memory or outputs may expose information beyond the intended audience.
Improper output handling
Generated content may be passed into code, databases or other systems without suitable validation.
Insecure retrieval
Search or vector systems may return documents that the user is not authorised to access.
Model and data supply chain
External models, datasets, libraries and providers introduce dependencies that require review.
Data and model poisoning
Manipulated training, fine-tuning or retrieval content may influence system behaviour.
Unbounded consumption
Automated or malicious use may create excessive resource use, cost or service degradation.
The OWASP Top 10 for LLM and GenAI documents risks including prompt injection, sensitive-information disclosure, improper output handling and excessive agency. Use it as technical reference material without presenting it as a complete control framework.
TEST THE INSTRUCTION BOUNDARY
Examine direct and indirect prompt manipulation
Prompt injection testing should assess how the application separates system instructions, user content, retrieved documents and tool responses. The impact depends on the data and authority available to the system.
Discuss AI Application TestingDirect input
Can a user prompt override restrictions, extract protected instructions or alter the expected workflow?
Uploaded content
Can instructions inside a document, image or other file influence the AI system?
Retrieved content
Can malicious content in a connected knowledge source manipulate the model or its tools?
Tool response
Can untrusted output from an external service change subsequent AI decisions?
SECURE THE KNOWLEDGE LAYER
Retrieval must preserve the organisation's access rules
A user's ability to ask a question should not provide wider access to source information. Permissions must be enforced during retrieval rather than relying on the model to decide what the user may see.
Source approval
Document permissions
Query-time filtering
Retrieval layer
Search, embeddings, document indexes, vector databases and access metadata.
Content integrity
Retrieval logging
Assessment questions
Which sources enter the index?
Are source permissions retained?
How are public and restricted documents separated?
Can one tenant retrieve another tenant's information?
How are documents updated or removed?
Can retrieved content contain malicious instructions?
Which retrieval events are logged?
MATCH AUTHORITY TO CONSEQUENCE
Give AI agents the minimum ability required for the task
Read public information
Search and summarise information already available to the intended user.
Read internal information
Access approved internal content according to user and system permissions.
Prepare an action
Draft an email, transaction, record update or command without executing it.
Execute with approval
Complete a sensitive action only after verified human approval.
Execute independently
Perform actions without case-by-case approval within a tightly defined scope.
For every level, define:
Permitted tools
Accessible data
User identity
Transaction or action limits
Required approval
Logging
Emergency disablement
OUTPUT VALIDATION AND DOWNSTREAM SAFETY
Treat model output as untrusted input
Validation layer: Treat model output as untrusted input
Generated output
Text
Structured data
Code
Queries
URLs
Tool instructions
Classification
Recommendations
Downstream consequence
Displayed to a user
Stored in a database
Executed as code
Sent to another system
Used for access decisions
Used for customer communication
Used to trigger a workflow
Validation should reflect the destination. A response displayed as text requires different controls from output that will execute code, change a record or approve a transaction.
CONTROL DATA THROUGHOUT AI USE
Know what the system receives, retains and reveals
Input
Question
What sensitive information can users place into prompts or uploads?
Control direction
User guidance, classification, input restrictions and approved use cases
Processing
Question
Which model, provider and region process the information?
Control direction
Provider assessment, configuration, contracts and data-flow mapping
Memory and storage
Question
Are prompts, outputs, embeddings or conversation history retained?
Control direction
Access control, retention, encryption and deletion
Output
Question
Can responses reveal personal, confidential or proprietary information?
Control direction
Retrieval permissions, output validation, testing and monitoring
SHARED TECHNOLOGY, RETAINED RESPONSIBILITY
Assess the provider and the way your organisation uses it
Provider evidence
Service architecture
Data use and retention
Model and service changes
Security controls
Incident notification
Subprocessors
Availability and recovery
Exit and deletion support
Your implementation
Data sent to the service
User and administrator access
Prompt and application design
Retrieval configuration
Tool permissions
Output handling
Monitoring
Customer communication
Provider assurance does not validate the complete AI application. The organisation still needs to assess its application, data flows, permissions, business use and downstream actions.
SECURITY THROUGHOUT DEVELOPMENT
Build controls into the AI system lifecycle
01
Define
Document the use case, users, data, decisions and unacceptable outcomes.
02
Design
Map trust boundaries, model access, retrieval, tools and approval points.
03
Source
Assess models, datasets, libraries and external providers.
04
Build
Protect repositories, prompts, configurations, secrets and development environments.
05
Test
Evaluate application security, AI-specific abuse, permissions and failure conditions.
06
Release
Control approvals, configurations, access and production changes.
07
Monitor
Review usage, incidents, model changes, unusual behaviour and control effectiveness.
NIST SP 800-218A adds AI model development considerations to the Secure Software Development Framework. Use it as one of the foundations for this lifecycle.
TEST THE CONNECTIONS BETWEEN COMPONENTS
AI security testing requires more than conventional scanning
The scope should reflect the system's architecture, data and authority. A chatbot that answers public questions has a different risk profile from an agent that can access internal records and execute business actions.
| Component | Access | Manipulation | Disclosure | Action | Investigation |
|---|---|---|---|---|---|
| User input | Test | Test | Test | Validate | Review |
| Uploaded content | Test | Test | Test | Validate | Review |
| Retrieval | Test | Test | Test | Out of scope | Review |
| Model behaviour | Validate | Test | Test | Validate | Review |
| Tool use | Test | Test | Validate | Test | Review |
| Output | Validate | Test | Test | Test | Review |
| Logging | Review | Out of scope | Review | Out of scope | Test |
VISIBILITY AFTER RELEASE
Monitor events that help teams understand AI-related incidents
Usage
Users and applications
Request volume
High-risk features
Unusual automation
Resource consumption
Data
Sensitive input
Restricted retrieval
Cross-tenant access
Output exposure
Retention events
Behaviour
Repeated injection attempts
Policy bypass
Unexpected tool requests
Validation failures
Model or prompt changes
Actions
Tool execution
Approval decisions
Access changes
Transaction outcomes
Emergency shutdown
Monitoring should support a defined response. Collecting every prompt without a clear purpose may create privacy, security and operational problems.
PREPARE FOR AI-SPECIFIC FAILURE
Respond across the model, application, data and business process
Establish facts
Identify affected users
Confirm the model and version
Preserve prompts and outputs
Determine connected data and tools
Limit authority
Disable affected tools
Restrict data sources
Revoke credentials
Suspend high-risk features
Correct the system
Remove malicious content
Update retrieval or application controls
Review prompts and configurations
Validate the proposed change
Restore and monitor
Test expected behaviour
Restore in controlled stages
Monitor for recurrence
Record lessons and ownership
GOVERNANCE WITH NAMED OWNERS
Every AI system needs accountable decisions
Business owner
Product or technology owner
Cybersecurity
AI use-case decision
Privacy and legal
Risk and compliance
For each use case, document:
Purpose
Accountable owner
Intended users
Approved data
Model or provider
Connected tools
Human oversight
Testing requirements
Monitoring
Review date
Retirement process
NIST's AI RMF organises AI risk-management activity through Govern, Map, Measure and Manage. Use these functions as guidance without presenting the framework as a certification.
ALIGNMENT WITH CONTEXT
Connect AI security with existing risk and development practices
AI requirements depend on the organisation, use case, data, decisions and jurisdictions involved. Digisecuritas helps teams map applicable guidance and obligations to working controls and evidence.
Framework alignment does not confirm that an AI system is safe, secure, compliant or free from harmful outcomes. Requirements must be assessed according to the use case and jurisdiction.
INDEPENDENT AI SECURITY SUPPORT
Assess the system, test the controls and improve accountability
AI security assessment
Develop a clear view of exposure across the application, model, data, retrieval, connected tools, infrastructure and external providers.
Expected outputs:
AI system and data-flow map
AI threat model
Prioritised risk register
Technical testing findings
Permission and access review
Supplier observations
Monitoring requirements
Practical remediation roadmap
AI application threat modelling
Identify trust boundaries, abuse paths, sensitive assets and failure conditions.
LLM penetration testing
Test approved AI features for manipulation, disclosure and unsafe actions.
RAG security assessment
Review retrieval permissions, source integrity, tenant separation and malicious content.
AI agent security review
Assess tools, credentials, approvals, limits and emergency controls.
AI data protection assessment
Map data use, provider handling, retention and output exposure.
AI provider risk assessment
Review third-party model and service dependencies.
Secure AI development review
Assess lifecycle controls across sourcing, development, release and monitoring.
AI governance assessment
Review inventory, ownership, approval, evidence and periodic oversight.
AI incident readiness
Develop response scenarios for AI-related security events.
A CLEAR WORKING METHOD
Move from use-case context to tested controls
01
Understand
Discuss the use case, users, business process, model, data and connected tools.
02
Map
Document architecture, trust boundaries, data flows and system authority.
03
Test
Perform approved application, AI-specific and infrastructure validation.
04
Prioritise
Rank findings by exposure, consequence and exploitability.
05
Improve
Support remediation, governance updates and validation of completed work.
Common reasons to assess an AI system
Launching a customer-facing AI feature
Connecting an AI agent to business tools
Introducing retrieval over internal documents
Allowing users to upload untrusted content
Sending sensitive data to an external model
Deploying AI across a regulated process
Preparing for an enterprise customer review
Investigating unexpected AI behaviour
AI security requires evidence from the complete system
Digisecuritas assesses the model within its application, data, identity and business context. Recommendations remain independent of model providers and security-product vendors.
System-level perspective
The model, application, retrieval, tools and infrastructure are examined together.
Independent validation
Findings are based on approved testing and available evidence.
Business-aware prioritisation
Risk is assessed according to data, authority and operational consequence.
Practical remediation
Recommendations identify ownership and validation conditions.
FREQUENTLY ASKED QUESTIONS
AI security questions
Common questions about AI security assessment, testing and governance.
START WITH THE COMPLETE AI SYSTEM
Build AI capabilities with clearer security boundaries
Tell us what the AI system can access, which decisions it influences and what actions it can perform. Digisecuritas will help you define a focused assessment and a practical route forward.
Independent security guidance for generative AI, LLM applications, RAG systems and AI agents.