LIKE is the tool most of us reach for first when we search free text. You end up chaining OR and NOT LIKE clauses to patch each miss, and you are still reading every record by hand.

The talk works through what to reach for instead, using 448 free-text notes from a fictional dog spa.

Take it home

• Code and notebooks on GitHub – start with search_toolkit.ipynb. Each method is a few lines you can lift straight into your own DuckDB project. go_hybrid.ipynb is the full walkthrough that follows the talk.
• Download the slides (PDF)

Run the code

git clone https://github.com/jamalhansen/go-hybrid
cd go-hybrid
uv sync
uv run jupyter lab search_toolkit.ipynb

The first run downloads the embedding model (all-MiniLM-L6-v2, about 90 MB). After that it is fully offline.

The dataset is synthetic and curated so each method fails visibly. Bring a note of your own, drop it in, and see which tools find it.

If you saw the talk, thank you for coming. If you have questions, reach out.

DuckDB’s file-native design is a first-class feature, not a limitation. File-native is good design. This article makes that clearer by pulling back the curtain on Postgres – a great database, but ultimately a complex program that stores data in a filesystem.

Note: the article contains a long quote used here for context. Trim to under 15 words before publishing.

At its core, postgres is simply a program that turns SQL queries into filesystem operations. A CREATE TABLE becomes a mkdir. An UPDATE eventually becomes: open a file, write to it, close it. It’s a complex and powerful system—but fundamentally, it’s just an executable that manipulates files.

In our last post, we created a customer database and generated 500 rows of fake data. Our in-memory database has the same problem: when Python exits, all that data vanishes:

import duckdb

con = duckdb.connect(':memory:')
con.execute("CREATE TABLE customers (id INT, name VARCHAR)")
con.execute("INSERT INTO customers VALUES (1, 'Alice')")
print(con.execute("SELECT * FROM customers").fetchdf())
# Script ends... and the data is gone forever

A database is supposed to provide persistent storage, isn’t it? Let’s fix that with one small change.

The One-Line Fix

So far, we’ve been creating our connection like this:

import duckdb
con = duckdb.connect(':memory:')

If we connect using a file path rather than the :memory: keyword, DuckDB stores everything to that file:

import duckdb
con = duckdb.connect('practice.duckdb')

That’s it. One small change.

Persistent Data Storage

Now, let’s upgrade our customer generation script to use persistent storage. This version uses a handy CREATE TABLE IF NOT EXISTS, which will only create the table when it isn’t there. This avoids errors that happen when you run the script multiple times.

from datetime import date

from faker import Faker
import duckdb
import random

# Connect to a FILE instead of memory
con = duckdb.connect('practice.duckdb')

# CREATE TABLE IF NOT EXISTS lets us run this script repeatedly
con.execute("""
    CREATE TABLE IF NOT EXISTS customers (
        id INTEGER,
        name VARCHAR,
        email VARCHAR,
        city VARCHAR,
        signup_date DATE,
        is_premium BOOLEAN
    )
""")

# Only generate data if the table is empty
row_count = con.execute("SELECT COUNT(*) FROM customers").fetchone()[0]

if row_count == 0:
    fake = Faker()
    Faker.seed(42)
    random.seed(42)
    
    customers = []
    for i in range(500):
        customers.append((
            i + 1,
            fake.name(),
            fake.email(),
            fake.city(),
            fake.date_between(start_date=date(2024, 1, 1), end_date=date(2025, 12, 31)),
            random.choice([True, False, False, False])
        ))
    
    con.executemany(
        "INSERT INTO customers VALUES (?, ?, ?, ?, ?, ?)",
        customers
    )
    print("Created 500 customers!")
else:
    print(f"Data already exists! ({row_count} customers)")

# Either way, we can query it
print(con.execute("SELECT * FROM customers").fetchdf().head())

Proving It Persists

In a fresh Python session:

import duckdb

# The file persists between Python sessions
con = duckdb.connect('practice.duckdb')

# Data is still there!
print(con.execute("SELECT * FROM customers").fetchdf().head())

The data survived. Now you have a persistent database with 500 customers ready for querying.

What’s Next

So far, we’ve only written SQL that brings back all the data in a table. To move forward quickly, we need to make a mental model for how SQL works. SQL is more like a list comprehension than a for loop, and next week we’ll explore this mental shift.

Python has the perfect tool: faker. It’s a library that generates realistic fake data: names, emails, addresses, dates, and anything else you’d find in a real database. Let’s use it to build a dataset we can explore for the rest of this series.

The Dataset: Customers

We’ll create a customer table, the kind you’ll find in almost any database:

customers
├── id (integer)
├── name (varchar)  
├── email (varchar)
├── city (varchar)
├── signup_date (date)
└── is_premium (boolean)

Building Our Dataset

Now let’s write the Python code that will:

• Generate 500 rows of data with faker
• Create the customer table
• Insert the generated data into the table
• Query that data to ensure it’s there

To get started, we will install the faker module:

pip install faker

Now let’s generate 500 customers:

from datetime import date

from faker import Faker
import random

fake = Faker()
Faker.seed(42)  # Same data every time
random.seed(42)

customers = []

for i in range(500):
    customers.append(
        {
            "id": i + 1,
            "name": fake.name(),
            "email": fake.email(),
            "city": fake.city(),
            "signup_date": fake.date_between(start_date=date(2024, 1, 1), end_date=date(2025, 12, 31)),
            "is_premium": random.choice([True, False, False, False]),
        }
    )

# Peek at what we generated
for c in customers[:3]:
    print(c)

{'id': 1, 'name': 'Allison Hill', 'email': 'donaldgarcia@example.net', 'city': 'New Roberttown', 'signup_date': datetime.date(2024, 3, 9), 'is_premium': True}
{'id': 2, 'name': 'Lance Hoffman', 'email': 'lrobinson@example.com', 'city': 'Port Lindachester', 'signup_date': datetime.date(2024, 9, 5), 'is_premium': True}
{'id': 3, 'name': 'Tyler Rogers', 'email': 'jamesmichael@example.com', 'city': 'Lindsaymouth', 'signup_date': datetime.date(2025, 1, 26), 'is_premium': False}

A few things to notice. First, faker can generate data for a variety of purposes and data types. We tell faker that we want a name or an email address, and it will generate not only a string, but an actual name or email address.

It will also create randomized values such as dates within a range or a choice of values from a list.

The is_premium line is a small trick: by putting False in the list three times, roughly 25% of customers end up as premium members. More realistic than a coin flip.

We are setting the random number seed at the beginning of the script. This makes the “random” data reproducible, so you’ll get the same values every time you run it. This is useful in a number of situations, including testing.

Now we have our generated data in the customers variable. Let’s create a table for it to live in and then insert the data into the table. You’ll notice VARCHAR in the CREATE TABLE statement. That’s SQL’s name for text data (like Python’s str).

import duckdb

con = duckdb.connect(':memory:')

con.execute("""
    CREATE TABLE customers (
        id INTEGER,
        name VARCHAR,
        email VARCHAR,
        city VARCHAR,
        signup_date DATE,
        is_premium BOOLEAN
    )
""")

con.executemany(
    "INSERT INTO customers VALUES (?, ?, ?, ?, ?, ?)",
    [
        (
            c['id'], 
            c['name'], 
            c['email'], 
            c['city'], 
            c['signup_date'], 
            c['is_premium']
        ) 
        for c in customers
    ]
)

Once the data has been inserted into the table, we can query it. Like last time, we just want to bring back all the records to see that they are there.

# See the first few rows
result = con.execute("SELECT * FROM customers").fetchdf()
print(result.head(10))

# Or just specific columns
result = con.execute("SELECT name, city FROM customers").fetchdf()
print(result.head(10))

The * in SELECT * is shorthand for “all columns.” Or you can list the columns that you want.

You should see ten unique customers: real-looking names, varied cities, a mix of premium and regular accounts. That’s 500 rows of realistic data we can explore for the rest of this series.

The Full Script

Here’s everything in one runnable file:

from datetime import date

from faker import Faker
import random
import duckdb

# Generate fake data
fake = Faker()
Faker.seed(42)
random.seed(42)

customers = []
for i in range(500):
    customers.append(
        {
            "id": i + 1,
            "name": fake.name(),
            "email": fake.email(),
            "city": fake.city(),
            "signup_date": fake.date_between(start_date=date(2024, 1, 1), end_date=date(2025, 12, 31)),
            "is_premium": random.choice([True, False, False, False]),
        }
    )

# Load into DuckDB
con = duckdb.connect(':memory:')

con.execute("""
    CREATE TABLE customers (
        id INTEGER,
        name VARCHAR,
        email VARCHAR,
        city VARCHAR,
        signup_date DATE,
        is_premium BOOLEAN
    )
""")

con.executemany(
    "INSERT INTO customers VALUES (?, ?, ?, ?, ?, ?)",
    [
        (c['id'], c['name'], c['email'], c['city'], c['signup_date'], c['is_premium']) 
        for c in customers
    ]
)

# Query the data
result = con.execute("SELECT * FROM customers").fetchdf()
print(result.head(10))

Try It Yourself

• Scale it up: Change range(500) to range(10000). See how fast it handles larger data.
• Change the seed: Try Faker.seed(123). Completely different people.
• Bonus challenge: Faker has fake.user_name(). Can you add a username column? You’ll need to update the customer dict, the CREATE TABLE statement, and the INSERT query.

Coming Up

We now have 500 customers to work with. In future posts, we’ll filter them, sort them, and answer real questions about this data.

But there’s one problem: every time you restart Python, this data disappears. Next week, we’ll save the database to a file so you can keep building on it.

Why are we using DuckDB to learn SQL?

• No setup - Just import and start using it
• Real SQL - PostgreSQL compatible syntax so what you learn transfers
• Fast enough - Handles millions of rows on your laptop
• Python-native - Works well with lists, DataFrames, CSV files

It is perfect for learning without infrastructure headaches.

Installation

pip install duckdb

That’s it. You’re done. No, really.

Your First Query

Now that we are ready to query the database, let’s write our first query and run it.

import duckdb

# Connect to DuckDB
with duckdb.connect(':memory:') as con:
	# Create table
	con.execute("""
	    CREATE TABLE data(name VARCHAR, score INTEGER)
	""")
	
	# Insert data
	con.executemany(
	    "INSERT INTO data VALUES (?, ?)",
	    [('Alice', 95), ('Bob', 87), ('Carol', 92)]
	)
	
	# Query the data
	result = con.execute("""
	    SELECT name, score 
	    FROM data 
	""").fetchall()

print(result)

When you run this, you should see the result:

[('Alice', 95), ('Bob', 87), ('Carol', 92)]

What happened?

So what exactly is DuckDB, and what did we do in these 22 lines of code?

DuckDB is an embedded SQL database optimized for analytics. Think “SQLite for data analysis”. It runs inside your Python process with zero setup.

Our 22-line script did four things:

1. Connect - Created an in-memory database (:memory:)
2. Create - Defined a table structure (name + score)
3. Insert - Added three rows of data
4. Query - Retrieved scores from the database

This is the basic SQL workflow you’ll use everywhere. Often, steps 2 and 3 aren’t necessary because someone else already did them for you.

Try It Yourself

Want to experiment? Modify the query to:

• Get only scores above 90: Add WHERE score > 90
• See the highest score first: Add ORDER BY score DESC
• Get just the top scorer: Add LIMIT 1

The best way to learn SQL is by experimenting. Try these modifications and see what happens!

What You Just Learned

In 5 minutes, you:

• Installed a production-grade SQL database
• Created your first table
• Inserted data
• Wrote your first SELECT query

No servers. No complexity. Just SQL.

Next week, we will use Python’s faker library to generate realistic datasets and practice SQL on hundreds of rows. You’ll see why data folks love SQL for exploration.

I learned SQL very early in my career. At the time, I didn’t understand why, and for the first month or so, it didn’t make sense to me. The syntax didn’t resemble any language I had seen before, and it employed concepts with which I was unfamiliar. This all made SQL seem scary, and oddly enough, SQL hasn’t changed much in the past 40 years, which makes it even more of an oddity.

The problem wasn’t the SQL; it was how I was thinking. I had some experience with coding, but only in a procedural style, telling the computer to perform a series of steps in order. My aha moment was when I realized that SQL doesn’t do this. Instead, a SQL query describes the data that you would like, and it returns that dataset. Making this mind shift turned SQL into a beautiful and functional language that is relatively simple to learn.

SQL is the language of data, and over the past 40 years, data has become more prevalent and more important. If you work with data, learning SQL will help you. This is true even if you already know and use another programming language like Python with good data management libraries. Python has great data support with SQL Alchemy, Pandas, and other great libraries. Learning SQL won’t replace them, but it will help you use them more effectively. When you understand how SQL thinks, you’ll write better pandas queries, ask smarter questions of your ORM, and know when to push work to the database instead of pulling everything into memory.

In this series, I’ll walk you through the steps to level up your SQL skills. We will use Python along the way to keep things practical. Our first step will be to set up a database. This is the biggest hurdle for beginners: you need a database to practice, but setting one up feels intimidating before you know what you’re doing. For this series, we will use a modern, local, and user-friendly database called DuckDB, which is simple to get started with.

Next, we will dig into querying data. We will learn about keywords such as SELECT, FROM, WHERE, and GROUP BY, and how they can be used to shape your results.

Finally, we will tackle some more complex topics, like when it’s useful to use SQL over an ORM or Pandas, and why you might consider it.

If you are a Python developer, you work with data. Follow along, and I’ll see you next week to set up DuckDB.