What Is Legal AI? A Practical Glossary for Law Firm Technology Leaders
RAG, LLM, fine-tuning, vector databases, embeddings, AI agents — every term explained in legal industry context, with examples from real legal AI applications.
Why this glossary exists
When we work with law firm CTOs, managing partners, and legal publishers, the same terminology questions come up repeatedly. AI terminology is evolving fast, and most explanations are written for engineers, not legal professionals. This glossary covers the terms that matter most for legal AI, explained in the context of how they actually apply to legal work.
The fundamentals
An LLM — large language model — is the foundation of modern AI tools. Models like GPT-4, Claude, and Llama are trained on vast amounts of text and learn to generate coherent, contextually appropriate responses. Think of an LLM as a very capable associate who has read everything but remembers things imperfectly and sometimes makes things up with great confidence. That last part is why you can’t just point an LLM at legal questions and trust the output.
A prompt is the instruction you give to an LLM. Prompt engineering is the practice of crafting these instructions to get reliable, useful outputs. In legal AI, prompt engineering is how you tell the model to answer only from retrieved sources, cite everything, and say “I don’t know” when appropriate. The quality of the prompt directly determines the quality of the output — a well-engineered prompt can eliminate entire categories of errors.
Hallucination is the term for when an AI model generates false information presented as fact. In legal AI, this typically means fabricated case citations, invented holdings, or mischaracterised legal principles. The Stanford “Hallucination-Free?” study found even leading legal AI tools (Lexis+ AI, Westlaw AI-Assisted Research) hallucinate on roughly 1 in 6 queries. It’s the central challenge in legal AI development.
Retrieval and search
RAG — retrieval-augmented generation — is the architecture behind most legal AI tools. Instead of relying on the LLM’s training data (which may be wrong or outdated), RAG retrieves relevant documents from a knowledge base and provides them to the LLM as context for generating its response. In a legal research tool, when you ask “what are the rules for arbitrator disclosure?”, the system first finds relevant passages from its source material, then synthesises an answer from those passages. This is how the AAA ChatBook tools work — each answer is grounded in specific, curated publications.
Embeddings are numerical representations of text that capture meaning. When you embed a legal document, you convert its text into a list of numbers that represents its semantic content. Similar concepts get similar numbers, which enables semantic search — finding documents that are conceptually related to your query even if they don’t share the same keywords. When a lawyer searches for “unfair dismissal compensation” and finds a document about “wrongful termination damages,” embeddings are what make that connection.
A vector database stores these embeddings and enables fast similarity search across millions of documents. It’s where your embedded legal content lives and what powers the retrieval step in RAG. Common options include Qdrant, Pinecone, and pgvector (a PostgreSQL extension). Choosing the right one depends on corpus size, filtering needs, and operational preferences.
Chunking is how documents are divided into smaller pieces for embedding and retrieval. This matters enormously for legal content. Standard chunking — splitting text every 500 tokens — destroys the logical structure of legal arguments. Section-aware chunking, which we use for legal projects, respects document boundaries so that a statutory provision stays whole and a court’s reasoning doesn’t get split mid-argument.
Hybrid search combines semantic search (finding conceptually similar content) with keyword search (finding exact terms). Legal queries often need both — a query about “discovery in patent litigation” needs semantic understanding, while a query citing “35 U.S.C. § 271” needs exact keyword matching. Hybrid search handles both.
Model customisation
Fine-tuning is the process of additional training on an existing model using your specific data. In legal AI, you might fine-tune a model on arbitration decisions, patent filings, or construction contracts. The result is a model that understands domain-specific language better than a general-purpose model. Fine-tuning is expensive and data-intensive, so it’s reserved for situations where the improvement justifies the investment — typically large corpora in specialised practice areas.
The difference between RAG and fine-tuning is important. RAG keeps the base model unchanged and retrieves relevant information at query time. Fine-tuning changes the model itself to make it more knowledgeable about a specific domain. Most legal AI applications use RAG (cheaper, more flexible, easier to update) rather than fine-tuning. Some use both — a fine-tuned model for better domain understanding, combined with RAG for access to specific current content.
A knowledge base is the corpus of content that a RAG system draws from. For a law firm’s internal assistant, this might be the firm’s document management system. For the AAA ChatBook tools, it’s specific published guides and rules. For PlanYourSunset, it’s the platform’s expert-written Learning Hub and New York state law. The quality of the knowledge base determines the quality of the AI’s answers.
AI agents and automation
An AI agent is an AI system that can execute multi-step tasks using tools. Unlike a chatbot (which answers one question at a time), an agent can receive a task like “review this contract against our playbook” and autonomously break it into sub-tasks: extract clauses, compare against standards, score risks, generate a report. Agents use tools — document parsers, search engines, calculators — to accomplish each step. Legal AI agents are emerging rapidly in 2026 for workflows like contract review, compliance monitoring, and research.
n8n and similar no-code automation platforms connect AI capabilities with existing business tools — CRM, email, cloud storage, spreadsheets. We use n8n to build AI-powered workflows that automate repetitive tasks like document triage, intake processing, and report generation. It’s useful when the AI doesn’t need to be deeply customised — it’s connecting existing capabilities rather than building new ones.
Security and compliance
On-premise deployment means running AI systems on the firm’s own hardware rather than in the cloud. Some firms require this for data security reasons — particularly when handling sensitive client matters. On-premise deployment gives complete control over data but requires more technical infrastructure to maintain.
Data residency refers to where data is physically stored and processed. UK GDPR and the EU AI Act have specific requirements about data transfers across jurisdictions. If your AI tool sends client data to US-based servers, you need appropriate transfer mechanisms.
The EU AI Act classifies AI systems by risk level. For legal AI, the relevant classifications are limited risk (chatbots requiring transparency about AI interaction) and high risk (AI used in administration of justice, including legal research and document analysis for court proceedings). High-risk obligations — documented risk management, technical documentation, human oversight, accuracy requirements — take effect in August 2026.
How these pieces fit together
A typical legal AI research tool combines several of these concepts. Documents from the knowledge base are chunked with section-aware splitting, then embedded and stored in a vector database. When a user asks a question, hybrid search retrieves relevant passages. The LLM, guided by carefully engineered prompts, synthesises an answer from those passages. A citation verification layer checks that every cited source exists and supports the claims made. The whole system runs on infrastructure that meets the firm’s security and data residency requirements.
Understanding these pieces helps technology leaders make better decisions about what to build, what to buy, and what questions to ask their vendors.
Have questions about legal AI technology? Contact us — we’re happy to explain how any of these concepts apply to your specific situation.