Spark interviewers stopped caring about API syntax years ago. At the senior and mid-level bar, the questions that separate candidates are almost all the same shape: "Your job is slow. Why, and what do you do?" Knowing how to write PySpark gets you past the screening; knowing how to tune it gets you the job. This post is the revision guide I'd want the night before — grounded in the official Spark tuning documentation, focused on the topics that actually come up (shuffle, partitioning, caching, joins, AQE, memory, Spark UI), with production-scenario questions at the end.
The mindset: know why and when, not just what
Every interview-prep list on this topic repeats the same concepts: narrow vs wide transformations, repartition vs coalesce, cache vs persist, broadcast vs shuffle join. Memorizing definitions is the minimum. What interviewers probe for is the second-order reasoning: when would you choose one over the other, and what does the trade-off cost you?
That framing matters because Spark tuning is about trade-offs, not silver bullets. Kryo serialization is faster than Java's but requires class registration. MEMORY_ONLY_SER halves memory usage but adds deserialization cost. Broadcast joins save a shuffle but only pay off under the threshold. If you can explain the trade-off for every technique, you've answered the deeper question underneath most interview questions.
1. Data serialization — the first thing to tune
The official tuning guide is blunt: serialization "will often be the first thing you should tune to optimize a Spark application." Serialization is what moves data between executors and spills RDDs to disk, so a slow or bloated format taxes your network and your memory at once.
Spark ships two serializers:
Java serialization (default) — flexible, works with any java.io.Serializable class, but slow and produces large payloads for many classes.
Kryo serialization — "significantly faster and more compact than Java serialization (often as much as 10x)." The catch: you must register the classes you use for best performance, which is exactly why it's not the default.
spark = SparkSession.builder \
.appName("tuned") \
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
.config("spark.kryoserializer.buffer", "64m") \
.getOrCreate()
Interview point: Kryo is not about syntax — it's about why it's not on by default (the registration requirement) and when you'd bother (network-intensive jobs, large object graphs). Since Spark 2.0, Spark already uses Kryo internally for shuffles of simple types, arrays, and strings.
2. Memory tuning — the unified memory model
Memory is where tuning questions get deep fast. Spark splits the JVM heap between execution (shuffles, joins, sorts, aggregations) and storage (cached data). Since Spark 1.6 they share a single unified region M, with three properties worth stating in an interview:
When execution isn't using its share, storage can use the whole region and vice versa.
Execution can evict storage blocks, but only down to a guaranteed floor R (storageFraction) where cached blocks are never evicted.
Storage can never evict execution memory.
The two configs you should be able to name:
spark.memory.fraction — fraction of heap (minus 300MiB reserved) that forms M. Default 0.6; the remaining 40% is for user data structures, Spark internal metadata, and OOM headroom.
spark.memory.storageFraction — fraction of M that's the guaranteed storage floor R. Default 0.5.
In an interview, don't just recite the numbers — explain the design intent. The unified model means an app that never caches gets the whole region for execution (fewer spills), while an app that caches gets a protected floor where its blocks can't be evicted.
GC tuning is the follow-up that separates the prepared from the rest. The goal is to keep long-lived RDDs in the Old (tenured) generation and keep the Young generation big enough to hold short-lived task objects, so you avoid full GCs mid-task. The classic moves, straight from the docs:
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps to measure GC before tuning.
Too many minor GCs → grow Eden; set -Xmn to roughly 4/3 × Eden estimate (survivor regions account for the extra third).
OldGen near full → lower spark.memory.fraction; it's better to cache fewer objects than to stall task execution.
Spark 4.x runs on JDK 17 by default, which makes G1GC the default — with big executor heaps you may need to bump -XX:G1HeapRegionSize.
A useful practical heuristic from the docs for sizing: a decompressed HDFS block is often 2–3× the stored size, so budget working space accordingly.
3. Parallelism and partitioning
"Clusters will not be fully utilized unless you set the level of parallelism for each operation high enough." That's the thesis of the parallelism section of the tuning guide. The headline number to know: 2–3 tasks per CPU core is the recommended parallelism. Spark can support tasks as short as 200ms efficiently because it reuses executor JVMs across tasks, so don't be afraid to raise parallelism past your core count.
The interview staples:
repartition() — changes partition count via a full shuffle. Use it to increase parallelism or fix a skewed partition count.
coalesce(n, shuffle=False) — merges partitions with minimal (or zero) shuffle. Use it to shrink partitions before writing to avoid small files — it can't usually increase parallelism.
Data skew is the other half. One executor running far longer than its peers is the classic symptom — visible in the Spark UI as a stage where one task's duration dwarfs the others. Skew happens because data (keys) is unevenly distributed; you fix it by salting keys, re-partitioning, or letting AQE do it for you.
File-based details worth knowing: for jobs reading many directories from object stores like S3, tune spark.sql.sources.parallelPartitionDiscovery.parallelism and .threshold to parallelize the directory listing — single-threaded listing on huge paths is a classic hidden slowdown.
4. Shuffle — the most expensive thing Spark does
Shuffles are the single biggest performance lever in most jobs. A wide transformation (groupBy, join, repartition, distinct) forces data movement across the cluster: each task writes shuffle output to disk and fetches other tasks' output over the network. Narrow transformations (map, filter, union) process partitions independently, so they never shuffle and never create a stage boundary.
Why it's expensive, as an interview answer: a shuffle means (a) serializing and writing intermediate data to disk, (b) moving it over the network, and (c) deserializing and merging on the receiving side. Every wide transformation also splits your job into stages, so the Spark UI shows you exactly where they are — a huge shuffle read/write spike marks the stage boundary.
Reducing shuffle, in practice:
Push filters down and filter before joins/aggregations, not after.
Prefer reduceByKey/aggregateByKey over groupByKey (combining on the map side shrinks what gets shuffled).
Broadcast small tables instead of shuffling a big one (next section).
Right-size spark.sql.shuffle.partitions — the default 200 is a guess, not a law. Too few partitions → big, slow, memory-heavy tasks; too many → scheduling and small-file overhead.
Rely on AQE to coalesce post-shuffle partitions automatically when you're not sure.
5. Join optimization
Joins are where the "can you run Spark at scale" questions concentrate. Three join strategies you must be able to explain and contrast:
Broadcast join — copies a small table to every executor, avoiding a shuffle entirely. The default auto-broadcast threshold (spark.sql.autoBroadcastJoinThreshold, 10MB by default) makes this the best-performing join when the small side fits. Know that AQE can convert shuffle joins to broadcast joins at runtime if a table turns out smaller than expected.
Shuffle hash join — hash-partitions both sides by key. Fast for medium tables but needs enough memory to hold one side's buckets in memory per task.
Sort merge join — the default for large tables, and the workhorse. Both sides are sorted and partitioned by the join key, then merged. Scale-safe but sorting both sides costs time and shuffle.
Interview framing: broadcast for small lookup tables, sort-merge for big-to-big, and mention that hash joins sit in the middle where memory allows. Add the skew-aware note: a heavily skewed join key can turn sort-merge into a straggler disaster, which is exactly the case AQE's skewed-join optimization targets.
6. Caching: cache() vs persist()
cache() is shorthand for persist(MEMORY_AND_DISK) — it stores a DataFrame across executors so repeated actions don't recompute the lineage. persist() generalizes this to a set of storage levels: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER, and disk-only variants.
The interview answer has three parts:
When to cache: a DataFrame reused across multiple actions (especially in iterative jobs — ML training loops, repeated aggregations on the same snapshot) or when recomputation is far more expensive than storage.
When not to: single-use data, data smaller than the query overhead, or data that changes between actions — a cached DataFrame does not auto-invalidate on updates, and a cached-but-useless DataFrame is just pressure on your storage region (and a classic "cache didn't help" production scenario).
SER levels: MEMORY_ONLY_SER / MEMORY_AND_DISK_SER store each partition as one byte array — the docs' first recommendation when objects are too big — trading deserialization CPU for dramatically lower memory (and the docs strongly recommend Kryo for serialized caching).
Don't forget lifecycle hygiene: unpersist() when done, because cached blocks compete with execution memory until evicted or removed.
File format answers are easy wins. Parquet is the default recommendation for analytical workloads because it's columnar, compressed, and splittable — and, crucially for interviews, because columnar formats unlock two optimizations that reduce I/O:
Predicate pushdown — filters (row selection) are pushed into the scan so Spark reads only the rows that match.
Column pruning — Spark reads only the columns referenced by the query, not the whole file.
Those two are the reason a Parquet-backed SELECT COUNT(*) can be dramatically faster than the equivalent CSV read. ORC is the same idea from the Hive/Hadoop lineage (stronger in Hive-ecosystem shops); Avro is row-oriented, better for write-heavy or schema-evolving pipelines. The interview-ready cheat: Parquet for analytics reads, Avro for transactional/write-heavy, CSV/JSON only when you have no choice. And mention the small-files problem: thousands of tiny files turn a scan into metadata and scheduling overhead — fix with coalesce()/repartition() on write or file compaction via Delta Lake/Iceberg.
8. Catalyst and Tungsten — the optimizer story
"DataFrame is faster than RDD" is a correct but unsatisfying answer. The satisfying one names the machinery:
Catalyst is Spark SQL's query optimizer. It converts your logical plan into an optimized physical plan via analysis, logical optimizations, physical planning, and code generation, applying 100+ rules including predicate pushdown, column pruning, constant folding, and join reordering. It also explains why UDFs are slow: they're a black box Catalyst can't push through or reorder — a favorite follow-up question.
Tungsten is the execution engine underneath: off-heap memory management, cache-aware data structures, and whole-stage code generation that compiles queries into tight JVM bytecode. This is where the "up to 100x faster than RDDs" claims come from.
9. AQE — the one feature to mention early
Adaptive Query Execution (enabled by default since Spark 3.2) re-optimizes the physical plan at runtime using statistics from completed stages — "runtime query optimization." Three things it does, all of which are interview gold because they replace manual tuning:
Dynamic coalescing of shuffle partitions — shrinks the post-shuffle partition count based on actual data size, killing the "200 default partitions is wrong" problem.
Dynamic join strategy switching — promotes shuffle joins to broadcast joins when a side is small enough.
Skew join handling — splits skewed partitions and rebalances work so one huge key doesn't straggle the whole stage.
Mentioning AQE and its limits (it only helps where stages have already completed to observe; it won't fix a bad schema, a UDF that blocks optimization, or a too-small cluster) shows you've actually run Spark, not just read about it.
10. Spark UI — your debugging superpower
Every tuning conversation should end at the Spark UI, because that's what interviewers will simulate in a scenario question. Know what each tab is for:
Jobs tab — which action, its DAG, overall timing.
Stages tab — stage boundaries (every wide transformation splits one); per-stage shuffle read/write and task durations. A slow stage with one long task = skew; a slow stage across all tasks = parallelism/memory problem.
Executors tab — per-executor memory, GC time, and cores; spot memory pressure and GC storms.
SQL tab — the physical plan per query; check whether a join became a broadcast or stayed a shuffle join.
Storage tab — what's actually cached and how much memory it uses (SizeEstimator and this tab are the docs' recommended way to size RDD memory usage).
The diagnosis ladder for "my job is slow": read the job → find the dominant stage → check shuffle metrics → check task-time distribution → check executors for GC → then pick the tuning lever that matches (more parallelism, better join, broadcast, serialization, caching, or file layout).
Production scenario questions — the final mile
These are the ones that feel like actual work, and a couple always surface. Walk them with the UI-first mindset above:
Job got 4x slower after a deployment. Diff the data volume and cluster sizing first, then check the UI: did a shuffle explode, did the query plan change (join strategy flipped to shuffle), or is GC dominating on the executors tab?
Excessive shuffle read/write. Look for wide transformations you don't need — groupByKey where reduceByKey works, un-broadcasted small joins, or an accidental repartition before a filter.
Thousands of small output files. coalesce() before write or compact the table; the write is I/O-bound on metadata, not data.
One executor always slower. Data skew — check task time distribution in the stage, then salt keys or let AQE handle it.
Driver OOM. Likely collect() on a large DataFrame — the driver pulling all data to one node. Replace with take(), show(), or an aggregation, and note that driver memory is sized separately from executors.
A practical tuning checklist
Enable Kryo and register your classes for network-heavy jobs.
Cache only reused data, prefer MEMORY_AND_DISK (or _SER + Kryo for large objects), and unpersist() when done.
Keep parallelism at 2–3× cores; right-size spark.sql.shuffle.partitions instead of trusting 200.
Broadcast small tables; let AQE handle the rest (it's on by default — say so).
Store analytical data in Parquet so pushdown and pruning actually work; fix small files on write.
Read the Spark UI before touching any config — tuning blind is guessing.
Measure first (GC logs, stage timings, storage tab), change one thing at a time, and re-measure.
Writing correct Spark code is table stakes. What interviewers pay for is the ability to look at a running job, find the bottleneck, and justify a change with a trade-off — and that's a skill you can practice on any of your existing pipelines. Take one production job, identify five optimizations from this list, apply them one at a time, and measure. That's the practical preparation that shows up in the room.
Sources
