scoiattolo.mountainherder.xyz
A training to familiarize TIDE analysts with parquet datasets and python
Fix Data Wrangler capabilities, Windows venv path, and switch to pyproject.toml
Exercise 1 taught a SQL query step that Data Wrangler does not support; it
is a low/no-code tool. Rework the exercise around the data grid, column
statistics panel, and filter/sort UI instead. Add the Git Bash Scripts/
activation path for Windows venvs, and replace requirements.txt with
pyproject.toml (pip install .).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
author
D Scarnecchia
co-author
Claude Fable 5
committer
D. Scarnecchia
date
(Jul 20, 2026, 10:26 AM -0400)
commit
b1b55635
b1b55635b5e88c959081df84a1da4f11142ff347
parent
47878751
47878751b7796d193a1fe733a43810a117bbb999
+70
-47
+1
.gitignore
+1
.gitignore
+13
-4
README.md
+13
-4
README.md
···
79
79
80
80
### Step 4: Activate the virtual environment
81
81
82
+
On macOS and Linux:
83
+
82
84
```bash
83
85
source .venv/bin/activate
84
86
```
85
87
88
+
On Windows (Git Bash), the venv puts its scripts in `Scripts/` instead of
89
+
`bin/`:
90
+
91
+
```bash
92
+
source .venv/Scripts/activate
93
+
```
94
+
86
95
You should now see `(.venv)` at the start of your terminal prompt.
87
96
88
97
### Step 5: Install dependencies
89
98
90
99
```bash
91
-
pip install -r requirements.txt
100
+
pip install .
92
101
```
93
102
94
-
This installs a single package: `polars==1.42.1`.
103
+
This reads `pyproject.toml` and installs a single package: `polars==1.42.1`.
95
104
96
105
### Step 6: Generate the data
97
106
···
142
151
intercepts HTTPS traffic:
143
152
144
153
```bash
145
-
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org -r requirements.txt
154
+
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org .
146
155
```
147
156
148
157
#### Wrong interpreter selected in VS Code
···
323
332
```
324
333
parquet-polars-training/
325
334
├── README.md ← you are here
326
-
├── requirements.txt ← polars==1.42.1
335
+
├── pyproject.toml ← project metadata + dependencies (polars==1.42.1)
327
336
├── .gitignore
328
337
├── data/
329
338
│ ├── generate_data.py ← run this to create the Parquet files
+41
-42
exercises/ex01_vscode_explorer.md
+41
-42
exercises/ex01_vscode_explorer.md
···
2
2
3
3
> **SAS framing:** This is the replacement for double-clicking a `.sas7bdat` file
4
4
> in SAS Enterprise Guide — look first, code second. Before writing any Python,
5
-
> you should be able to open a data file and visually inspect its contents. You
6
-
> can also run SQL queries against Parquet files directly in VS Code, just like
7
-
> `PROC SQL` against a SAS dataset.
5
+
> you should be able to open a data file and visually inspect its contents.
6
+
> Data Wrangler gives you a point-and-click data grid with column statistics,
7
+
> much like the SAS viewtable window combined with `PROC FREQ` output.
8
8
9
9
---
10
10
···
25
25
26
26
After generating data (Exercise 0), you should have three files in `data/`:
27
27
28
-
1. Open `data/demographic.parquet` — click it in the Explorer sidebar.
28
+
1. Open `data/demographic.parquet` — right-click it in the Explorer sidebar
29
+
and choose **Open in Data Wrangler**.
29
30
2. Open `data/enrollment.parquet`
30
31
3. Open `data/dispensing.parquet`
31
32
32
33
If the extension is working, you will see a data grid (not raw bytes). Take a
33
34
moment to scroll through each file and get a feel for the columns and values.
34
35
35
-
## Step 3: Query Parquet files with SQL
36
+
## Step 3: Explore with column statistics
36
37
37
-
Both Data Wrangler and Data Preview let you run SQL queries against opened
38
-
Parquet files. This is the same concept as `PROC SQL` against a SAS dataset —
39
-
the table name is the file name (without the `.parquet` extension).
38
+
Data Wrangler is a **low-code/no-code tool** — there is no SQL query panel.
39
+
Instead, everything you would ask with a small `PROC SQL` or `PROC FREQ` step
40
+
is answered by clicking around the interface:
40
41
41
-
Here is a worked example. Open `demographic.parquet`, then run this query:
42
+
- **Row count** — shown in the summary bar (e.g., "10,000 rows"). This is your
43
+
`SELECT COUNT(*)`.
44
+
- **Column dtype** — shown in each column header next to the column name. This
45
+
is your `PROC CONTENTS`.
46
+
- **Column statistics panel** — click a column header to open a summary of that
47
+
column: value distribution, unique values, and missing-value count. This is
48
+
your `PROC FREQ` / `PROC MEANS`.
49
+
- **Filter and sort** — use the column header menus to filter rows or sort,
50
+
the way you would with a `WHERE` clause or `PROC SORT`.
42
51
43
-
```sql
44
-
-- PROC SQL equivalent: SELECT COUNT(*) AS n FROM demographic;
45
-
SELECT COUNT(*) AS row_count
46
-
FROM demographic;
47
-
```
52
+
Here is a worked example. Open `demographic.parquet` in Data Wrangler and look
53
+
at the summary bar — you should see **10,000 rows**, one row per patient. Now
54
+
click the `PostalCode` column header: the statistics panel shows the count of
55
+
missing values, which answers "how many patients have no postal code?" without
56
+
writing a line of code.
48
57
49
-
You should get a result like `10000` — one row per patient.
50
-
51
-
Try another one. This query counts how many patients are missing a
52
-
`PostalCode` value (the `IS NULL` check works just like in PROC SQL):
53
-
54
-
```sql
55
-
-- PROC SQL equivalent: SELECT COUNT(*) FROM demographic WHERE PostalCode IS NULL;
56
-
SELECT COUNT(*) AS missing_postal
57
-
FROM demographic
58
-
WHERE PostalCode IS NULL;
59
-
```
60
-
61
-
> **Tip:** If you are not sure where to type the query, look for a "Query" or
62
-
> "SQL" button in the Data Wrangler panel, or open a new SQL file
63
-
> (`File → New File`) and save it with a `.sql` extension, then select the
64
-
> Parquet file as the connection target.
58
+
> **Note:** Data Wrangler can also generate Python (pandas) code for the
59
+
> operations you perform in the grid. Feel free to peek at it, but do not copy
60
+
> it into the exercises — this workshop uses Polars, and the two APIs do not
61
+
> mix. Treat Data Wrangler as a viewer here, not a code generator.
65
62
66
63
## Step 4: Answer the following questions
67
64
68
-
Use SQL queries (and the data grid view) to answer each question. Write your
69
-
answers below each question or in a separate document.
65
+
Use the data grid and column statistics panel to answer each question. Write
66
+
your answers below each question or in a separate document.
70
67
71
68
### Q1: How many rows are in `dispensing.parquet`?
72
69
73
-
Hint: `SELECT COUNT(*) FROM dispensing;`
70
+
Hint: Open the file in Data Wrangler and read the row count from the summary
71
+
bar.
74
72
75
73
Your answer:
76
74
77
75
### Q2: What is the dtype of the `RxDate` column?
78
76
79
77
Hint: Look at the column header in the data grid — the dtype is shown next to
80
-
the column name. You can also query the schema if the extension supports it.
78
+
the column name.
81
79
82
80
Your answer:
83
81
84
82
### Q3: Which columns are in `enrollment.parquet`?
85
83
86
-
Hint: `SELECT * FROM enrollment LIMIT 5;` — then look at the column headers
87
-
in the result.
84
+
Hint: Open the file and read the column headers across the top of the grid.
88
85
89
86
Your answer:
90
87
91
88
### Q4: What are the unique values of `Sex` in `demographic.parquet`?
92
89
93
-
Hint: `SELECT DISTINCT Sex FROM demographic;`
90
+
Hint: Click the `Sex` column header — the statistics panel lists the unique
91
+
values and their counts (like `PROC FREQ`).
94
92
95
93
Your answer:
96
94
97
95
### Q5: What is the null count for `PostalCode` in `demographic.parquet`?
98
96
99
-
Hint: `SELECT COUNT(*) FROM demographic WHERE PostalCode IS NULL;`
97
+
Hint: Click the `PostalCode` column header and read the missing-value count
98
+
from the statistics panel.
100
99
101
100
Your answer:
102
101
···
105
104
## Why this matters
106
105
107
106
In SAS, you would run `PROC CONTENTS` to see column metadata, `PROC PRINT` to
108
-
eyeball the data, and `PROC SQL` to query specific values. The VS Code
109
-
Parquet viewer gives you all three capabilities without writing a DATA step or
110
-
PROC. Use it to confirm your expectations before you start writing Polars
111
-
queries in the exercises that follow.
107
+
eyeball the data, and `PROC FREQ` to check value distributions. Data Wrangler
108
+
gives you all three capabilities without writing a DATA step or PROC. Use it to
109
+
confirm your expectations before you start writing Polars queries in the
110
+
exercises that follow.
+15
pyproject.toml
+15
pyproject.toml
···
1
+
[build-system]
2
+
requires = ["setuptools>=68"]
3
+
build-backend = "setuptools.build_meta"
4
+
5
+
[project]
6
+
name = "parquet-polars-training"
7
+
version = "0.1.0"
8
+
description = "Parquet + Polars workshop for SAS analysts"
9
+
requires-python = ">=3.11"
10
+
dependencies = [
11
+
"polars==1.42.1",
12
+
]
13
+
14
+
[tool.setuptools]
15
+
packages = []
-1
requirements.txt
-1
requirements.txt
···
1
-
polars==1.42.1