The Execution Model (Everything Else Builds On This)
Almost every hard PySpark question reduces to 'do you know what happens across the network'. Get this right and the rest follows.
- Driver vs executors — where your Python code actually runs
- Job → stage → task: a stage boundary is a shuffle
- Lazy evaluation: transformations build a plan, actions trigger execution
- Catalyst optimiser and why your filter placement matters less than you think
- Partitions are the unit of parallelism — too few starves the cluster, too many adds overhead
Narrow vs Wide Transformations
This is the concept interviewers use to check whether you understand cost, and it comes up in almost every loop.
- Narrow: map, filter, select, withColumn — each output partition depends on one input partition, no network movement
- Wide: groupBy, join, distinct, repartition — data must move across executors (a shuffle)
- Shuffles write to disk and cross the network, which is why they dominate runtime
- Reducing the number of shuffles is usually a bigger win than any single micro-optimisation
Data Skew — The Most Asked Performance Question
Skew is when one key holds a disproportionate share of the rows, so one task runs for an hour while 199 finish in seconds.
- Symptom: in the Spark UI, one task's duration is orders of magnitude above the stage median
- Cause: a hot key — a null placeholder, a default customer id, one huge tenant
- Fix 1 — Broadcast join: if one side is small, avoid the shuffle entirely
- Fix 2 — Salting: append a random suffix to the hot key, join, then aggregate again
- Fix 3 — Adaptive Query Execution (AQE) skew join handling, available by default in Spark 3.x
- Fix 4 — Filter the hot key out, process it separately, union the results
Joins and Broadcast
Join strategy is where most real Spark performance is won or lost.
- Sort-merge join: the default for two large datasets, requires a shuffle on both sides
- Broadcast hash join: the small side is shipped to every executor, no shuffle on the large side
- spark.sql.autoBroadcastJoinThreshold controls the automatic cut-over (default 10 MB)
- A job that got slower 'for no reason' is often a dimension that outgrew the broadcast threshold
- Bucketing pre-shuffles data on disk so repeated joins on the same key skip the shuffle
Caching, Partitioning and File Layout
The unglamorous parts that determine whether your pipeline costs ₹5,000 or ₹50,000 a month.
- cache() only pays off if the DataFrame is reused — caching a single-use DataFrame is pure overhead
- repartition() is a full shuffle; coalesce() only merges partitions and avoids one
- The small-files problem: thousands of tiny Parquet files destroy read performance
- Partition columns should be low-cardinality (date, region) — never a user id
- Columnar formats (Parquet, ORC) plus predicate pushdown mean you read less from disk
Common Interview Questions & Answers
Q1. Explain lazy evaluation and one way it has caused you a problem.
Transformations only build a logical plan; nothing executes until an action like count() or write(). The practical consequence is that errors surface far from where you wrote them — a malformed cast on line 10 throws during the write on line 200. It also means that if you call an action three times without caching, the whole lineage recomputes three times, which is the most common accidental 3x cost in a pipeline.
Pairing the concept with a real consequence is what makes this answer stand out.
Q2. One task in your stage takes 45 minutes, the other 200 finish in under a minute. What's happening?
That's textbook data skew — one partition holds far more rows than the rest, almost always because of a hot join or groupBy key. I'd first identify the key by aggregating counts on the join column. If the hot key is a null or a placeholder, I'd filter it and handle it separately. If it's a genuinely large tenant, I'd salt the key: add a random 0-N suffix on both sides, join, then re-aggregate. If the other side of the join is small enough, a broadcast join removes the shuffle entirely.
Lead with the diagnosis, then give more than one fix — it shows you have hit this in production.
Q3. When would you NOT use cache()?
When the DataFrame is used exactly once — you pay serialisation and memory cost for no reuse. Also when the dataset doesn't comfortably fit in the executor memory fraction, because it will spill to disk or evict other cached data and end up slower than recomputing. And caching before a narrow chain of filters is usually wrong; cache after the expensive shuffle, not before it.
Being able to argue against a technique is a stronger signal than being able to argue for it.
Q4. repartition(200) vs coalesce(200) — which do you use and why?
coalesce is cheaper because it merges existing partitions without a full shuffle, so it's the right choice when reducing partition count, typically just before a write. repartition triggers a full shuffle but produces evenly sized partitions, so it's the right choice when I'm increasing parallelism or fixing an unbalanced distribution. Using coalesce to go from 2000 to 10 partitions can also backfire by reducing upstream parallelism, since the narrowing propagates back up the stage.
That last caveat about upstream parallelism is a detail most candidates miss.
Common Mistakes to Avoid
Explaining Spark purely through DataFrame API syntax with no mention of the cluster
Saying 'I'd just increase the cluster size' as the first answer to any performance question
Calling cache() reflexively without asking whether the data is reused
Not knowing that a shuffle writes to disk — describing it as only network movement
Claiming Spark experience that turns out to be a single notebook on a 10 MB CSV
Expert Tips
Learn to read the Spark UI — 'I'd look at the stage timeline' is a phrase that earns trust
Know your numbers: how many GB, how many partitions, how long the job runs
Practise saying the skew answer out loud until it takes 60 seconds, not three minutes
Mention AQE — many interviewers still expect the manual fixes and are impressed you know the modern default
Pre-Interview Checklist
6 itemsFrequently Asked Questions
Do I need Scala for a PySpark role?
No. PySpark is the standard for data engineering and analytics roles in India. Scala matters mainly for platform teams maintaining Spark internals or legacy codebases.
How do I practise Spark without a cluster?
Local mode reproduces the execution model faithfully enough for interview purposes — you can see stages, shuffles and the Spark UI on a laptop. Free Databricks Community Edition gives you a real cluster if you want the operational feel.
Ready to ace your next interview?
Practice with SpeakWell AI. Upload your resume → get resume-based questions → practice with AI interviewers → improve communication → track progress → get instant AI feedback.