PySpark Transformations

df.select("col") or df.select(col("col"))

Selects specific columns. Accepts strings, col() objects, or expressions. Equivalent to SQL SELECT.

df.filter(condition) / df.where(condition)

Filters rows. where() is an alias for filter(). df.filter(col("age") > 18). Equivalent to SQL WHERE.

df.withColumn("name", expr)

Adds a new column or replaces an existing one. df.withColumn("total", col("price") * col("qty"))

df.withColumnRenamed("old", "new")

Renames a column. Does not modify other columns.

df.drop("col1", "col2")

Removes one or more columns from the DataFrame.

df.groupBy("col").agg(…)

Groups rows and applies aggregate functions. df.groupBy("dept").agg(F.sum("sal").alias("total_sal"))

df.orderBy(col("col").desc())

Sorts the DataFrame. Use .asc() or .desc() on a col(). orderBy() and sort() are aliases.

df.join(df2, on="col", how="inner")

Joins two DataFrames. how options: "inner", "left", "right", "outer", "left_semi", "left_anti", "cross"

left_semi join

Returns rows from the LEFT DataFrame that have a match in the right. No columns from right table. Like SQL EXISTS.

left_anti join

Returns rows from the LEFT DataFrame that have NO match in the right. Like SQL NOT EXISTS.

df.distinct()

Removes duplicate rows across all columns. Equivalent to SQL SELECT DISTINCT *.

df.dropDuplicates(["col1", "col2"])

Removes duplicates based on specific columns. More targeted than distinct().

df.limit(n)

Returns a new DataFrame with the first n rows. Lazy — only executes on action.

df.union(df2)

Combines two DataFrames with the same schema by position. Keeps duplicates.

df.unionByName(df2)

Combines two DataFrames matching columns by name, not position. Safer for reordered schemas.

df.alias("a")

Assigns an alias to a DataFrame. Used to disambiguate columns in self-joins: df.alias("a").join(df.alias("b"), …)

col("x").cast("integer")

Casts a column to a new type inside an expression. df.withColumn("x", col("x").cast("integer"))

df.na.drop(how, subset)

Drops rows with NULLs. how="any" (default) drops if any null; how="all" drops only if all nulls. subset limits to specific columns.

df.na.fill(value, subset)

Fills NULLs. df.na.fill(0) for numeric, or df.na.fill({"col1": 0, "col2": "N/A"}) for per-column.

df.repartition(n, col)

Shuffles data into n partitions (full shuffle). Use to increase partitions or repartition by a key column.

df.coalesce(n)

Reduces partitions without a full shuffle. Efficient for decreasing partition count before writing.

df.explode(col)

Expands an array or map column into multiple rows — one row per element.

df.pivot("col")

Rotates row values into columns. df.groupBy("year").pivot("month").agg(F.sum("sales"))

df.sample(fraction, seed)

Returns a random sample of rows. df.sample(0.1, seed=42) = 10% sample.

Window.partitionBy("col").orderBy("col")

Defines a window specification for window functions. w = Window.partitionBy("dept").orderBy("sal")

df.withColumn("rn", F.row_number().over(w))

Applies a window function. Must use .over(windowSpec). Requires importing Window from pyspark.sql.window.