Augmented Generation

How to add knowledge to LLMs and make them more useful for specific tasks.
NoteBonus material

We won’t cover this unit during the workshop. It’s included for anyone who wants to go further on their own. Expect to spend about 40 minutes on the activities. Start with the setup steps below.

Before you start

Install the extra packages for this unit:

pak::local_install_dev_deps(dependencies = "Config/Needs/bonus")
uv sync --group bonus

The activities also need an embedding model to compute the vectors used for retrieval. You can use any embedding model you like, local or hosted. Both exercises use nomic-embed-text-v2-moe, which you can run locally with LM Studio or ollama:

The LM Studio copy is half the size because it’s a quantized build of the same model. Quantization stores each model weight in fewer bits, which shrinks the file with a small cost to accuracy. For this exercise the difference doesn’t matter. If you already use one of these tools, stick with it.

The exercise as written talks to LM Studio on localhost:1234. To use ollama instead, switch the embedding call. In R, use ragnar::embed_ollama() in place of embed_lm_studio():

store <- ragnar_store_create(
  store_location,
  title = "R for Data Science",
  embed = \(x) {
    embed_ollama(x, model = "nomic-embed-text-v2-moe")
  }
)

The exercise as written talks to LM Studio on localhost:1234. To use ollama instead, switch the embedding call. In Python, point EmbeddingOpenAI() at base_url="http://localhost:11434/v1", since ollama also exposes an OpenAI-compatible API:

embed = EmbeddingOpenAI(
    model="nomic-embed-text-v2-moe",
    base_url="http://localhost:11434/v1",
)

Slides

View slides in full screen

Outline

  • (10m) Manual RAG
    • Activity 50_coding-assistant: Use an LLM as a coding assistant
      • Write a function that uses {weathR} (R) or NWS (Python) to get the weather for a location.
      • Then, give the LLM the {weathR} (R) or NWS (Python) README and see how much better the response is.
  • (30m) RAG
    • High-level overview of how RAG works
    • Activity 51_rag: Build a dynamic RAG system
      • Make a chatbot that can draw from the knowledge of:
      • Preprocess and compute embeddings for each chunk using ragnar (R) or raghilda (Python)
        • https://posit-dev.github.io/raghilda/user-guide/chatlas-integration.html
        • https://ragnar.tidyverse.org/articles/ragnar.html#setting-up-rag
      • Add a tool that searches the embeddings and returns the top few chunks
        • ellmer: Use ragnar
        • chatlas: this means writing a function