Beginner
What is LLM security?
LLM, or large language model, is a language model with a large number of parameters trained on broad text or multimodal data to understand and generate language.
What is prompt injection?
Prompt injection is an adversarial technique that manipulates a language model by placing malicious or unintended instructions in user input, documents, web pages, or other content the model processes.
What is a jailbreak in an LLM system?
Prompt injection or jailbreak refers to adversarial input intended to bypass a model's instructions, safety policies, or operational controls. Mitigations include input filtering, retrieval isolation, least-privilege tool access, policy enforcement, and layered guardrails.
How is RAG different from fine-tuning?
Use RAG when the model needs current, source-grounded information; use fine-tuning when the model needs specialized behavior, task performance, domain adaptation, or style learned from data.
Intermediate
Why can RAG create privacy and security risk?
Retrieval Augmented Generation (RAG) is a technique that combines generative AI with retrieval from external data sources. The model uses retrieved documents, passages, or records as grounding context to produce responses that are more current, specific, and verifiable.
What are guardrails in an LLM system?
Guardrails are technical, procedural, or policy controls designed to keep AI systems within acceptable boundaries. They may reduce risks such as harmful outputs, data leakage, unauthorized access, policy violations, or unsafe behavior.
What is grounding in generative AI?
Grounding is the process of connecting an AI model’s output to trusted, verifiable sources of information at inference time. In generative AI, it helps improve factual reliability by supplying relevant context without changing the model’s underlying weights.
What is hallucination in an AI system?
A hallucination is an AI-generated output that appears plausible or confident but is false, unsupported, misleading, or fabricated. In generative AI, hallucinations may include incorrect facts, invented sources, false citations, or reasoning that is not grounded in reliable evidence.
How is a system prompt different from a user prompt?
Use system prompt for durable application instructions and constraints; use user prompt for the user’s immediate request, context, and desired output.
What is red teaming for LLMs?
Red teaming is a structured adversarial testing process used to identify weaknesses, unsafe behavior, or harmful outputs in an AI system. It involves deliberately probing the system with challenging, malicious, or edge-case inputs so developers can evaluate and improve safeguards.
Advanced
What is data poisoning in AI systems?
Data poisoning is an adversarial attack in which a malicious actor interferes with the data or training process used to build a model. By inserting, modifying, or corrupting training examples, the attacker seeks to manipulate model behavior, degrade performance, or create targeted failures.
What is content moderation for AI outputs?
Content moderation and safety filters are controls designed to detect, block, reduce, or escalate harmful, illegal, or policy-disallowed content. In AI systems, they are often combined with policy rules, model-based classifiers, logging, and human review for higher-risk cases.