Intermediate
When is an AI system placed on the market?
Placing on the market means the first making available of an AI system or general-purpose AI model on the EU market. The term is important because certain regulatory obligations attach at or before this point.
What is the intended purpose of an AI system?
Intended purpose is the use for which an AI system is designed, marketed, or deployed by its provider. It defines the expected context, users, functions, outputs, and limitations, and is central to risk classification, compliance, and accountability.
What is reasonably foreseeable misuse?
Reasonably foreseeable misuse is the use of an AI system outside its intended purpose in a way that can still be anticipated from human behavior or interaction with other systems.
What is CE marking for AI systems?
CE marking is a conformity mark indicating that a product or AI system meets applicable requirements under relevant European Union law. For AI systems, it signals that the provider has assessed compliance with the legal obligations that apply to the system.
What is general-purpose AI?
General-Purpose AI (GPAI) refers to AI models or systems that can be adapted to a wide range of tasks and applications. Foundation models are a common example, in contrast to narrow AI systems designed for a specific task.
What is systemic risk in GPAI?
Systemic risk is a risk associated with high-impact capabilities of general-purpose AI models that can produce significant effects across markets, society, public health, safety, security, or fundamental rights.
Why can RAG create privacy and security risk?
Retrieval Augmented Generation (RAG) is a technique that combines generative AI with retrieval from external data sources. The model uses retrieved documents, passages, or records as grounding context to produce responses that are more current, specific, and verifiable.
What are guardrails in an LLM system?
Guardrails are technical, procedural, or policy controls designed to keep AI systems within acceptable boundaries. They may reduce risks such as harmful outputs, data leakage, unauthorized access, policy violations, or unsafe behavior.
What is grounding in generative AI?
Grounding is the process of connecting an AI model’s output to trusted, verifiable sources of information at inference time. In generative AI, it helps improve factual reliability by supplying relevant context without changing the model’s underlying weights.
What is hallucination in an AI system?
A hallucination is an AI-generated output that appears plausible or confident but is false, unsupported, misleading, or fabricated. In generative AI, hallucinations may include incorrect facts, invented sources, false citations, or reasoning that is not grounded in reliable evidence.
How is a system prompt different from a user prompt?
Use system prompt for durable application instructions and constraints; use user prompt for the user’s immediate request, context, and desired output.
What is red teaming for LLMs?
Red teaming is a structured adversarial testing process used to identify weaknesses, unsafe behavior, or harmful outputs in an AI system. It involves deliberately probing the system with challenging, malicious, or edge-case inputs so developers can evaluate and improve safeguards.
How is AI risk assessment different from AI audit?
Use AI Risk Assessment to identify and manage risks; use AI Audit to evaluate evidence, controls, or claims against defined criteria.
What is AI assurance?
AI Assurance is the set of practices that provide justified confidence that an AI system works as intended in its operational context.
What is shadow AI?
Shadow AI is the use of AI tools or systems without formal approval, oversight, or governance by an organization. It creates risks around data leakage, compliance, security, and inconsistent decision-making, and is typically managed through policy, approved alternatives, training, and monitoring.
What is an AI use policy?
An AI Use Policy is an internal governance document that defines how AI tools and systems may be used within an organization. It typically covers acceptable and prohibited uses, roles and responsibilities, data handling, approval requirements, monitoring, and incident escalation.
What is vendor due diligence for AI?
AI vendor due diligence is the assessment of third-party AI products or providers for security, privacy, reliability, governance, and compliance risks. It helps organizations decide whether and how a vendor can be safely approved, procured, monitored, or restricted.
What is algorithmic accountability?
Algorithmic Accountability is responsibility for the design, operation, decisions, and impacts of algorithmic systems. It involves transparency, traceability, oversight, and mechanisms for explaining, contesting, or remedying harmful outcomes.
How is pseudonymisation different from anonymisation?
Use Anonymisation / De-identification when discussing reduced or removed identifiability; use Pseudonymisation when personal data can still be attributed with protected additional information.
What is differential privacy?
Use Data Anonymisation when modifying data to reduce identifiability; use Differential Privacy when limiting individual inference from outputs through a privacy framework.
Can synthetic data still create privacy risk?
Synthetic data is artificially generated data designed to resemble selected patterns, structures, or statistical properties of real data.
What is data provenance?
Data provenance is the recorded history of data, including its origin, creation, transformations, movements, and changes over time. In AI governance, provenance helps verify dataset lineage, assess trustworthiness, support audits, and understand how digital content or model inputs were produced.
What is data residency in AI?
Data residency and sovereignty concern where data is stored, processed, and governed, and which legal regimes apply to it. For regulated AI use, these concepts affect cloud-region selection, cross-border transfers, contractual controls, and compliance with sectoral or national requirements.
How do GDPR and the EU AI Act overlap?
Use GDPR when the issue is processing personal data; use the EU AI Act when the issue is AI system risk category, safety, transparency, governance, or conformity obligations.
How is data drift different from concept drift?
Use Data Drift when input data distribution changes; use Concept Drift when the relationship between inputs and the target outcome changes.
What is alignment in AI safety?
Alignment is the practice of designing, training, and governing an AI system so that its goals, outputs, and behavior remain consistent with human intentions, values, and acceptable use constraints.
What is a serious AI incident?
An AI Incident is an event or series of events in which the development, deployment, use, or malfunction of an AI system causes or could cause harm. Such harm may involve safety, discrimination, privacy, legal compliance, critical infrastructure, property, communities, or the environment.
What is human-in-the-loop control?
HITL, or human-in-the-loop, is an approach in which people remain involved in shaping, reviewing, approving, or improving AI system outputs or decisions. It is used to combine machine efficiency with human judgment, especially in higher-risk or context-sensitive tasks.
What is AI toxicity?
AI Toxicity is harmful, offensive, false, biased, or otherwise inappropriate output generated or amplified by an AI system. It may arise from training data, system design, prompt context, misalignment, or inadequate safeguards.
Why does post-market monitoring matter?
A post-market monitoring system consists of activities by AI system providers to collect and review experience from systems placed on the market or put into service. Its purpose is to identify whether corrective or preventive action is needed after deployment.
What is an F1 score?
The F1 score is the harmonic mean of precision and recall, used to evaluate classification and information extraction performance. It ranges from 0 to 1, with higher values indicating a better balance between correctly identified positives and missed positives.
How are false positives and false negatives different?
Use False Positive for an incorrect positive flag; use False Negative for an incorrect failure to detect a real positive condition.
What is a classification threshold?
A classification threshold is a chosen cutoff value used to convert a model's score or probability into a class prediction. Changing the threshold can alter the balance between false positives and false negatives, making threshold selection an important governance and performance decision.
What is validation data?
Validation data is data used to evaluate a model during development and tune choices such as hyperparameters, thresholds, or learning processes. It should be distinct from training and test data to help detect underfitting, overfitting, and generalization problems.
What is a test set?
A test set is a reserved dataset used to provide an independent evaluation of a trained model or AI system. It should be separated from training and validation data so that performance estimates better reflect behavior on unseen data before deployment or release.
What is model monitoring?
Model monitoring is the continuous observation of a deployed model's performance, inputs, outputs, drift, errors, latency, and operational health. It helps detect degradation, misuse, bias, security issues, and conditions that require retraining or intervention.
What is AI orchestration?
AI Orchestration is the coordination and routing of AI tasks, workflows, model calls, tools, and agents within a system. It helps break complex processes into manageable steps and assign them to the appropriate models, sub-agents, or automation components.
What is function calling in AI agents?
Function calling is a capability that allows a language model to invoke external tools, APIs, or predefined functions. It extends model behavior beyond text generation by enabling structured actions, retrieval, computation, or system integration.
What is the Model Context Protocol?
Model Context Protocol (MCP) is a protocol for connecting AI models or assistants to external tools, data sources, and services through a standardized interface. It helps separate model behavior from tool integration and can improve portability, governance, and context management.
What is action space in agentic AI?
Action Space is the set of tools, operations, permissions, or moves available to an agent when performing a task. A well-scoped action space should be sufficient for the task while limiting unnecessary capabilities that may increase errors or governance risk.
Why do permissions matter for AI agents?
IAM, or Identity and Access Management permissions, are granular access controls that define which principals may perform which actions on which resources. They are commonly assigned through roles to users, groups, service accounts, or applications to enforce least-privilege access.
What is a router agent?
A router agent is an agent that classifies a user request or task and directs it to the most appropriate specialized agent, tool, or workflow. It is used in multi-agent systems to coordinate responsibilities and improve task handling.
What is a remote biometric identification system?
A remote biometric identification system is an AI system used to identify natural persons without their active involvement, typically at a distance. It compares a person's biometric data with biometric data stored in a reference database.
What is real-time remote biometric identification?
A real-time remote biometric identification system is a system in which biometric data capture, comparison, and identification occur without significant delay. The category includes instant identification as well as short delays intended to prevent circumvention.
What is an emotion recognition system?
An emotion recognition system is an AI system intended to identify or infer the emotions or intentions of natural persons, often from biometric data. Because such systems can affect privacy, autonomy, and fairness, they are subject to heightened scrutiny in governance and regulatory contexts.
What is biometric categorisation?
A biometric categorisation system is an AI system that assigns natural persons to specific categories on the basis of biometric data.
Why is facial recognition high risk?
Facial Recognition Technology (FRT) refers to automated systems that identify, verify, or categorize people by analyzing facial features from digital images or video. It is a biometric technology and may raise privacy, accuracy, and discrimination concerns depending on its use.
What are deepfakes in biometric AI?
Deepfakes are AI-generated or AI-manipulated images, audio, or video that make people, objects, places, entities, or events appear authentic when they are not.
What is self-supervised learning?
Self-supervised learning is a machine learning approach that creates training signals from unlabeled data itself. It converts aspects of an unsupervised problem into supervised learning by generating surrogate labels, making it useful when labeled data is scarce but unlabeled data is abundant.
What is semi-supervised learning?
Semi-supervised learning is a machine learning approach that uses both labeled and unlabeled data during training. It is useful when labeled examples are costly or limited but larger volumes of unlabeled data can help improve representation, classification, or prediction.
What is training data?
Training data is the dataset used to teach or fit a machine learning model by adjusting its learnable parameters. Its quality, quantity, diversity, labeling, and provenance strongly influence the performance, reliability, and compliance posture of the resulting model.
What is overfitting?
Overfitting occurs when a model learns the training data too closely, including noise or idiosyncratic details, and performs poorly on new data. It is commonly indicated by high training performance but weaker validation or test performance.
What is underfitting?
Underfitting occurs when a model is too simple or insufficiently trained to capture meaningful patterns in the data. An underfit model performs poorly on both training data and new data because it has not learned the underlying relationships needed for the task.
What is a decision tree?
A decision tree is an interpretable supervised learning model that represents decision rules in a tree-like structure. Inference follows a path from a root node through feature-based conditions to a leaf node that provides the predicted class or value.
What is a generative adversarial network?
Generative Adversarial Network (GAN) is a machine learning architecture in which a generator and discriminator are trained in opposition. The generator learns to create realistic samples, while the discriminator learns to distinguish generated samples from real data.
What is a deepfake?
Deepfakes are AI-generated or AI-manipulated images, audio, or video that make people, objects, places, entities, or events appear authentic when they are not.
What is synthetic media?
Synthetic media is media content that is wholly or partly generated or manipulated by AI. It can include text, images, audio, video, avatars, voices, and other digital artifacts.
What is prompt engineering?
Prompt engineering is the practice of designing, testing, and refining inputs to generative AI models so they produce desired outputs more reliably. It can involve specifying roles, constraints, examples, context, format, and evaluation criteria without changing the model's underlying weights.
What is temperature in generative AI?
Temperature is a model-generation parameter that controls the randomness of an AI system’s output. Lower values tend to produce more focused and deterministic responses, while higher values increase variety, creativity, and the risk of less coherent or less reliable output.
What is AI watermarking?
AI Watermarking is the embedding of visible or hidden identifiers into AI-generated or AI-modified content to indicate origin, provenance, or generation status. Watermarks may be added during generation or after generation and may require specialized tools to detect.
Advanced
Who supervises AI systems in the EU?
The AI Office is the European Commission function established to support implementation, monitoring, and supervision of AI systems, general-purpose AI models, and AI governance under the EU AI framework.
What is a serious incident under AI governance?
A serious incident is an AI-system incident or malfunction that directly or indirectly causes severe harm, such as death, serious health harm, critical infrastructure disruption, fundamental-rights infringement, or serious property or environmental damage.
What is data poisoning in AI systems?
Data poisoning is an adversarial attack in which a malicious actor interferes with the data or training process used to build a model. By inserting, modifying, or corrupting training examples, the attacker seeks to manipulate model behavior, degrade performance, or create targeted failures.
What is content moderation for AI outputs?
Content moderation and safety filters are controls designed to detect, block, reduce, or escalate harmful, illegal, or policy-disallowed content. In AI systems, they are often combined with policy rules, model-based classifiers, logging, and human review for higher-risk cases.
What is an AI model card?
An AI Model Card is a concise documentation artifact that accompanies a trained model or AI service. It typically describes intended use, limitations, evaluation results, relevant risks, data or training context, and other information that supports transparency and responsible deployment.
What is transparency in AI governance?
Transparency is the degree to which an AI system’s purpose, data use, operation, limitations, and outputs can be understood or examined by relevant stakeholders.
What is training data privacy risk?
Training data is the dataset used to teach or fit a machine learning model by adjusting its learnable parameters. Its quality, quantity, diversity, labeling, and provenance strongly influence the performance, reliability, and compliance posture of the resulting model.
What is encryption in AI data governance?
Data encryption is the transformation of readable data into an encoded form that is unintelligible without an authorized key or decryption process. It is used to protect sensitive information from unauthorized access during storage, transmission, or processing.
What is anomaly detection in AI monitoring?
Anomaly detection is the process of identifying observations, events, or patterns that differ significantly from expected behavior. In AI and data analysis, it is used to flag outliers that may indicate errors, fraud, security incidents, system faults, or other unusual conditions.
What is security by design for AI systems?
Security by design is an approach in which security requirements and safeguards are integrated from the start of system development. For AI systems, it includes anticipating misuse, protecting data and models, and embedding access controls, monitoring, and resilience throughout the lifecycle.
What is groundedness in LLM evaluation?
Groundedness is the property of a model output being supported by specific source material or provided context. In generative AI, groundedness helps assess whether an answer is traceable to evidence rather than unsupported model generation.
What is unsupported-claim rate?
UCR (unsupported-claim rate) is the percentage of claims in a model response that are not grounded in supporting evidence. A high UCR indicates that the system may be producing too many unsupported or hallucinated assertions.
What is a multi-agent system?
Multi-agent systems are systems composed of multiple interacting agents that can perceive, decide, communicate, and act within an environment. They are used to model distributed problem solving, coordination, negotiation, competition, and collective behavior.
How should agentic AI be governed?
AI Governance is the framework of policies, processes, roles, controls, and oversight used to steer AI development and use across its lifecycle. It aims to ensure responsible, lawful, secure, transparent, and accountable AI practice within an organization or jurisdiction.
What is synthetic media?
Synthetic media is media content that is wholly or partly generated or manipulated by AI. It can include text, images, audio, video, avatars, voices, and other digital artifacts.
How can biometric AI harm fairness?
Fairness in AI refers to the principle that AI systems should treat individuals and groups equitably and avoid unjustified discrimination or harmful bias. It involves identifying and mitigating bias in data, models, deployment contexts, and decision processes.
What is deep learning?
Deep learning is a subfield of machine learning that uses neural networks with multiple layers to learn representations and patterns from data.
How are classification and regression different?
Use Classification when the target is a category; use Regression when the target is a continuous or numerical value.
What is content provenance?
Provenance and watermarking refer to signals that help identify how content was created, where it came from, or whether it was AI-generated or modified. These mechanisms support authenticity, trust, moderation, auditability, and responsible content governance.
Why does generated content need human review?
Human evaluation is the process of having people judge the quality, safety, usefulness, or correctness of model outputs. It is especially important for tasks where there is no single correct answer, such as translation quality, summarization, dialogue, or creative generation.