Polars Notes

Data Manipulation¶

Polars¶

Polars is eager by default (pl.DataFrame) but shines with the lazy API (pl.LazyFrame + collect()), which optimises your query pipeline. Core idea: everything is an expression combined with the .select() / .filter() / .with_columns() / .group_by() methods.

import polars as pl

Casting¶

Use .cast() (or pl.col().cast()). String → numeric needs .str.strip_chars()/replace_all first.

df.select(
    pl.col("price").cast(pl.Float64),
    pl.col("symbol").cast(pl.Categorical),
    pl.col("size").cast(pl.Int64).round().cast(pl.Int64),
    pl.col("ts").str.to_datetime(),                    # parse string → datetime
    pl.col("ts").str.to_datetime("%Y-%m-%d"),          # explicit format: faster/safer
)
# errors → null:  pl.col("price").str.to_numeric(strict=False)
# downcast:       pl.col("price").cast(pl.Float32)

Gotchas: - Casting float → int truncates — .round() first if you need rounding. - .cast(pl.Utf8) = to string; .cast(pl.Categorical) = memory-efficient category. - .str.to_datetime(..., strict=False) turns bad values into null instead of erroring. - .dt accessor for components: pl.col("ts").dt.year(), .dt.month(), .dt.weekday() (Mon=0), .dt.day().

Filtering¶

.filter() takes one or more expressions (comma-separated = AND).

# filter rows, then select columns
df.filter(pl.col("strategy") == "Strategy_A")
df.filter(pl.col("strategy") == "Strategy_A").select(["pnl", "currency"])

# AND / OR
df.filter((pl.col("strategy") == "Strategy_A") & (pl.col("pnl") > 0))
df.filter((pl.col("strategy") == "Strategy_A") | (pl.col("strategy") == "Strategy_B"))

# membership
df.filter(pl.col("strategy").is_in(["Strategy_A", "Strategy_B"]))
df.filter(~pl.col("strategy").is_in(["Strategy_A"]))

# strings
df.filter(pl.col("currency").str.contains("USD"))
df.filter(pl.col("strategy").str.starts_with("Strategy"))

# nulls
df.filter(pl.col("pnl").is_null())
df.filter(pl.col("pnl").is_not_null())

# numeric range
df.filter(pl.col("pnl").is_between(-100, 100))

Selecting / adding columns¶

# .select() keeps only listed columns; .with_columns() adds/overwrites in place
df.select("symbol", "size", (pl.col("price") * pl.col("size")).alias("notional"))
df.with_columns((pl.col("price") * pl.col("size")).alias("notional"))

# multiple new columns in one call
df.with_columns(
    notional=pl.col("price") * pl.col("size"),
    px2=pl.col("price") ** 2,
)

Indexing¶

There is no index — rows are identified by position; sort/group keys are just columns.

df.head(n)          # first n rows
df.tail(n)          # last n rows
df.row(0)           # first row as a tuple
df.rows()           # all rows as tuples
df[0]               # first row (DataFrame)
df[:, 0]            # first column (Series)
df.filter(pl.col("strategy") == "Strategy_A").select("pnl")
df.nth(2)           # column by position

GroupBy¶

group_by() returns a context; aggregations go inside .agg(). Returns rows directly — no separate apply step.

df.group_by("strategy").agg(pl.col("pnl").sum())
df.group_by("strategy").agg([
    pl.col("pnl").sum().alias("total_pnl"),
    pl.col("pnl").mean().alias("avg_pnl"),
    pl.col("pnl").count().alias("count"),
    pl.col("pnl").std(),
])
df.group_by("strategy").agg(pl.col("pnl").quantile(0.5))
df.group_by(["strategy", "side"]).agg([pl.col("size").min(), pl.col("size").max()])

# VWAP convention (weighted mean of price by size) — per symbol
df.group_by("symbol").agg(
    (pl.col("price") * pl.col("size")).sum() / pl.col("size").sum()
)

# nested aggregations → struct
df.group_by("strategy").agg(pl.col("size").quantiles([0.1, 0.5, 0.9]))

Notes: - Chain .sort() after .agg() — group keys become columns by default. - .agg() accepts a list → multiple aggregations; expressions with .alias() name the results. - Polars is lazy: nothing computes until .collect() — write the whole pipeline, then collect once.

Sort / unique¶

df.sort("timestamp", descending=False)
df.sort(["symbol", "size"], descending=[False, True])   # per-column direction
df.unique(subset=["symbol"])                 # drop duplicate symbols
df.n_unique()                                # count unique rows (DataFrame-level)

Time series¶

Works on a regular datetime column — no index needed.

# resample to bars → group_by_dynamic with every="1min"
df.sort("timestamp").group_by_dynamic(
    "timestamp", # column or list of columns to group by
    every="1min",
    closed="left",  # which boundary the bucket closed (inclusive). left means [start, end), right means (start, end]
    start_by="window", # determine the start of the first window by. other values like 'datapoint' or 'monday' are also possible
).agg(
    open=pl.col("price").first(),
    high=pl.col("price").max(),
    low=pl.col("price").min(),
    close=pl.col("price").last(),
)

# volume bars
df.sort("timestamp").group_by_dynamic("timestamp", every="5min").agg(pl.col("size").sum())

# window boundaries: every vs period vs offset
#   every="5min"  → start a window every 5 minutes
#   period="10min" → window length (defaults to every)
#   offset="-2m"   → shift window start
# closed="left"/"right" controls which boundary the bucket is labelled by

# rolling / window functions via .rolling_* or .rolling().over()
df.sort("timestamp").with_columns(
    ma=pl.col("price").rolling_mean(window_size=20),        # SMA
    std=pl.col("price").rolling_std(window_size=20),
    ewma=pl.col("price").ewm_mean(span=10),                 # EWMA
    # rolling z-score
    z=((pl.col("price") - pl.col("price").rolling_mean(20))
       / pl.col("price").rolling_std(20)),
)
# grouped / per-symbol rolling: add .over("symbol")
df.sort("timestamp").with_columns(
    ma=pl.col("price").rolling_mean(20).over("symbol"),
)

# cumulative
df.with_columns(
    cum_size=pl.col("size").cum_sum(),
    # cumulative VWAP
    cum_vwap=pl.col("price").cum_sum() * pl.col("size").cum_sum() / pl.col("size").cum_sum(),
)
# fix for cum_vwap — proper weighted cumulative mean:
# (pl.col("price") * pl.col("size")).cum_sum() / pl.col("size").cum_sum()

Notes: - Sort the data before rolling/resample operations (they assume order). - Prefer built-ins (.rolling_mean, .rolling_std) over .rolling_apply (slow Python callback). - .over(...) is Polars' way of doing window/grouped operations without group_by (keeps row count).

Merging / Joins¶

df1.join(df2, on=..., how=...) — SQL-style.

df1.join(df2, on="symbol", how="inner")              # same key name
df1.join(df2, left_on="sym", right_on="symbol", how="left")   # different names
df1.join(df2, on=["symbol", "date"], how="outer")   # composite key
df1.join(df2, on="symbol", how="left", suffix="_right")        # avoid col-name clash

Notes: - how options: inner (default), outer, left, right, semi, anti, cross. - semi = keep rows in left that have a match (no fan-out); anti = keep rows with no match. - Duplicated keys can fan out — validate with .join(..., validate="one_to_one").

join_asof (time-aware) — attach most recent preceding row, e.g. mid-price at each trade.

# both frames must be sorted by the time key; by= groups per symbol.
trades_df = trades.sort("timestamp")
book_df   = book.sort("timestamp")

merged = trades_df.join_asof(
    book_df.select("timestamp", "mid"),
    on="timestamp",
    by="symbol",           # group the merge per symbol
    strategy="backward",   # most recent ≤ t (default)
    tolerance="5s",        # max time diff; unmatched → null
)

Key options: - strategy="backward" (default) — most recent ≤ t; "forward" — next ≥ t; "nearest" — closest. - by= groups the merge (e.g. per symbol). Both frames sorted by time key and group keys. - tolerance= caps the time difference — unmatched rows become null. - Classic uses: arrival-price benchmark, VWAP benchmark, spread at trade time.

Vectorization / Expressions¶

Everything is already vectorised, expressed via the expression API — no Python apply loops needed.

# vectorised column ops (lazy, runs in parallel)
notional = (df["price"] * df["size"]).collect()
# in a DataFrame:
df.with_columns((pl.col("price") * pl.col("size")).alias("notional"))

# when you must apply a Python function row-wise
df.with_columns(
    pl.col("size").map_elements(lambda v: v * 2, return_dtype=pl.Int64)
)

Rule of thumb: prefer expression API → .map_elements → apply on groups (last resort) — never loop rows.

Lazy API & query planning¶

Build a lazy plan and collect once — Polars optimises (predicate pushdown, projection pushdown, join reordering).

q = (
    df.lazy()
    .filter(pl.col("pnl") > 0)
    .with_columns((pl.col("price") * pl.col("size")).alias("notional"))
    .group_by("strategy")
    .agg(pl.col("notional").sum().alias("total"))
    .sort("total", descending=True)
)
result = q.collect()          # execute the whole plan once
q.show_graph()                # visualise the optimised plan
q.explain()                   # show the logical plan

Notes: - pl.scan_csv(...) / pl.scan_parquet(...) read lazily — the plan is only evaluated on .collect(). - Use .collect(streaming=True) for large datasets that don't fit in memory.

Memory optimisation¶

# categories for low-cardinality strings
df.with_columns(pl.col("symbol").cast(pl.Categorical))

# downcast numerics (only if it preserves values)
df.with_columns(
    pl.col("price").cast(pl.Float32),
    pl.col("size").cast(pl.Int32),
)

# per-column memory audit
df.estimated_size()          # total bytes
df.select(pl.all().suffix("_mem")).estimated_size()

Notes: - df.estimated_size() returns total bytes in RAM — useful to quantify savings. - Categorical shines when unique values << row count (saves memory + speeds group_by).