Parquet + Polars Workshop#

A hands-on workshop teaching SAS-background analysts to inspect Parquet files in VS Code and read/manipulate Parquet data with Polars in Python. By the end of this workshop, you will be able to replace common SAS data-step and PROC SQL patterns with concise Polars equivalents, and understand why Parquet is the modern columnar format of choice for analytics.

This repository serves as both the synchronous workshop materials and a standalone post-workshop reference. Every exercise is self-contained with built-in assertions that verify your solution.

Target audience: SAS analysts who are new to Python and want to work with Parquet files using Polars — a fast, type-safe DataFrame library that replaces pandas for analytics work.

Prerequisites#

• Python 3.11+ — verify with python3 --version. On Windows with Git Bash, if python3 is not found, use python or the full path to your installation. Make sure Python is on your PATH during installation (check "Add to PATH").
• VS Code — install from https://code.visualstudio.com/
• Git Bash (Windows) — installed with Git for Windows. macOS and Linux users already have a terminal with bash.

Data Description#

This workshop uses synthetic data modeled on the Sentinel Common Data Model (SCDM) v8.2.2 — the same data standard used by the FDA Sentinel system for distributed query analytics. The data is entirely fabricated; no real patient records are included.

The three tables are:

PatID uniquely identifies a patient across all three tables. Enr_Start and Enr_End define the coverage window for each enrollment span. RxDate is the date a prescription was filled. DrugCov indicates whether drug (pharmacy) coverage was active during the enrollment span — a key field for coverage QC checks.

The SCDM specification is publicly available at the Sentinel CDM repository.

Exercise 0 — Environment Setup (~30 min)#

Step 1: Clone the repo#

git clone https://github.com/your-org/parquet-polars-training.git
cd parquet-polars-training

Step 2: Verify Python version#

python3 --version

You should see Python 3.11.x or higher.

Step 3: Create a virtual environment#

python3 -m venv .venv

Step 4: Activate the virtual environment#

On macOS and Linux:

source .venv/bin/activate

On Windows (Git Bash), the venv puts its scripts in Scripts/ instead of bin/:

source .venv/Scripts/activate

You should now see (.venv) at the start of your terminal prompt.

Step 5: Install dependencies#

pip install .

This reads pyproject.toml and installs a single package: polars==1.42.1.

Step 6: Generate the data#

python data/generate_data.py

You should see output like:

Data generation complete.
  demographic.parquet :   10,000 rows
  enrollment.parquet  :   14,221 rows
  dispensing.parquet  :  199,613 rows

Step 7: Verify the Parquet files exist#

ls -la data/*.parquet

You should see three files: demographic.parquet, enrollment.parquet, and dispensing.parquet.

Troubleshooting#

python3 not on PATH (Windows Git Bash)#

On Windows, if python3 is not found in Git Bash, try python instead. If neither works, Python may not be on your PATH. Reinstall from python.org and check "Add to PATH" during installation, then restart Git Bash.

pip behind a proxy#

If you are behind a corporate proxy, configure pip:

pip config set global.proxy <your-org-proxy-settings>

pip SSL certificate verification error#

If you see SSL: CERTIFICATE_VERIFY_FAILED due to a corporate firewall that intercepts HTTPS traffic:

pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org .

Wrong interpreter selected in VS Code#

If VS Code shows a different Python interpreter (e.g., a system Python instead of your venv):

1. Press Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (macOS)
2. Type and select Python: Select Interpreter
3. Choose the interpreter that shows .venv in its path

SAS-to-Polars Mapping#

The Polars Expression Model#

The biggest conceptual jump from SAS is the expression model. In SAS, you write conditions as strings inside a WHERE clause:

where rxdate >= '01JAN2021'd and rxamt > 30;

In Polars, conditions are first-class objects built with pl.col():

df.filter(
    (pl.col("RxDate").dt.year() == 2021)
    & (pl.col("RxAmt") > 30)
)

What is pl.col()?#

pl.col("RxDate") returns an expression — a recipe for a computation that has not yet been executed. Expressions can be:

• Chained: pl.col("RxDate").dt.year() — extract the year from a date
• Combined: (pl.col("RxAmt") > 30) & (pl.col("RxSup") >= 30) — logical AND
• Aliased: pl.col("RxDate").dt.year().alias("RxYear") — rename the result
• Aggregated: pl.col("RxAmt").sum(), pl.col("RxAmt").mean()

Why expressions instead of strings?#

In SAS, the WHERE clause is a string parsed at runtime. If you misspell a column name, you get a runtime error. In Polars, expressions are validated against the DataFrame schema when filter() or select() is called. If you type pl.col("RxAmt") but the column is named rxamt, you get a clear error immediately.

Expressions also compose: you can store an expression in a variable and reuse it across multiple operations, which is impossible with string-based WHERE clauses:

is_2021 = pl.col("RxDate").dt.year() == 2021
high_amount = pl.col("RxAmt") > 30

df.filter(is_2021 & high_amount)    # combined
df.filter(is_2021).filter(high_amount)  # sequential (same result)

This is the pattern you will use throughout Exercises 3-6.

Exercises#

Exercise 1: Parquet in VS Code (~15 min)#

Before writing any Python, learn to visually inspect Parquet files in VS Code. This is the replacement for double-clicking a .sas7bdat file in SAS Enterprise Guide — look first, code second. Install the Data Wrangler extension, open each of the three Parquet files, and answer the inspection questions in exercises/ex01_vscode_explorer.md.

Exercise 2: Read and Inspect (~20 min)#

Read a Parquet file into a Polars DataFrame and inspect it. This is the Polars equivalent of PROC CONTENTS (schema), PROC PRINT (OBS=10) (head), and PROC MEANS (describe) — all in a few lines of Python. See exercises/ex02_read_and_inspect.py.

Exercise 3: Filter and Select (~25 min)#

Filter rows and select columns using the Polars expression model. This is where you learn pl.col(), expression chaining, and logical operators — the Polars equivalents of SAS WHERE, KEEP, DROP, and PROC SORT. See exercises/ex03_filter_select.py.

Exercise 4: Group By and Aggregate (~25 min)#

Group data and compute summary statistics. This covers df.group_by().agg() — the Polars equivalent of PROC SUMMARY / PROC MEANS with a CLASS statement, or PROC SQL GROUP BY. See exercises/ex04_groupby_agg.py.

Exercise 5: Joins (~25 min)#

Join multiple tables and check data coverage. Part (a) is a straightforward inner join (like a MERGE with IN= flags). Part (b) is a miniature Sentinel enrollment-coverage QC check: which dispensings have no matching drug-covered enrollment span? Part (c) uses an anti-join to find those uncovered dispensings. This mirrors a real Sentinel analytic pattern. See exercises/ex05_joins.py.

Exercise 6: Write and Partition (~20 min)#

Write filtered data to new Parquet files and partition datasets by a column value (e.g., year). This covers write_parquet with partition_by and introduces lazy evaluation via scan_parquet. See exercises/ex06_write_and_partition.py.

Partitioning#

Partitioning a Parquet dataset by a column (e.g., RxYear) creates a directory structure like RxYear=2021/, RxYear=2022/, etc. This matters for large tables because it enables predicate pushdown — when you filter for RxYear == 2022, Polars only reads the RxYear=2022/ directory and skips the rest. It also enables parallel reads across partitions and column pruning within each file, making queries on multi-gigabyte datasets fast.

A Note on Pandas#

You may have heard of pandas — the most popular Python DataFrame library. This workshop uses Polars instead. Polars is faster (written in Rust, multi-threaded), has a stricter type system (catches bugs earlier), and uses a lazy evaluation model that pandas lacks. The expression-based API is also more consistent and composable than pandas' mixed string/method approach.

If you are following along with external tutorials, do not mix pandas idioms with Polars code — the APIs are different. Everything in this workshop is self-contained in Polars.

Lazy Evaluation#

Polars supports both eager and lazy evaluation:

• Eager (pl.read_parquet, df.filter()): operations execute immediately. Use this for interactive exploration and small-to-medium datasets.
• Lazy (pl.scan_parquet, lf.filter().collect()): operations are queued into a query plan, optimized as a whole, and executed only when .collect() is called. Use this for large datasets or complex multi-step queries.

The key benefit of lazy evaluation is query optimization. When you chain multiple operations (filter → group_by → sort), the lazy engine reorders and fuses them to minimize memory usage and avoid scanning unnecessary data. For example, a filter on RxYear applied before a group_by will be pushed down to the Parquet reader, so only the relevant partitions are read from disk.

Exercise 6 includes an optional stretch goal that uses scan_parquet to perform a lazy aggregation across the partitioned dataset.

Repository Structure#

parquet-polars-training/
├── README.md                   ← you are here
├── pyproject.toml              ← project metadata + dependencies (polars==1.42.1)
├── .gitignore
├── data/
│   ├── generate_data.py        ← run this to create the Parquet files
│   ├── demographic.parquet    ← generated (gitignored)
│   ├── enrollment.parquet      ← generated (gitignored)
│   └── dispensing.parquet      ← generated (gitignored)
├── exercises/
│   ├── ex01_vscode_explorer.md
│   ├── ex02_read_and_inspect.py
│   ├── ex03_filter_select.py
│   ├── ex04_groupby_agg.py
│   ├── ex05_joins.py
│   └── ex06_write_and_partition.py
├── solutions/
│   ├── ex02_read_and_inspect.py
│   ├── ex03_filter_select.py
│   ├── ex04_groupby_agg.py
│   ├── ex05_joins.py
│   └── ex06_write_and_partition.py
└── scripts/
    └── verify_scdm.py          ← validates generated data against SCDM spec

Exercises contain TODO blocks with hints. Solutions are complete and pass all assertions. To check your work, run any exercise script:

python exercises/ex02_read_and_inspect.py

If you see All checks passed., your solution is correct. If you see a NotImplementedError, start filling in the TODO blocks.