When persisting a Spark DataFrame to storage for Delta Lake use, which format should you write in?

When persisting a Spark DataFrame to storage for Delta Lake use, which format should you write in?

• A. CSV
• B. PARQUET
• C. DELTA
• D. JSON

Answer: (C)

Delta Lake relies on a transaction log that sits alongside the data to provide ACID guarantees, time travel, and reliable upserts and schema management. Writing in the Delta format ensures Spark creates both the Parquet data files and the Delta transaction log, enabling features like MERGE/UPDATE/DELETE and concurrent writes with consistent reads. If you wrote as Parquet (or as CSV/JSON), you’d store data efficiently but miss the Delta transaction log and the associated capabilities. Therefore, using the Delta format is the correct approach.

Delta Lake relies on a transaction log that sits alongside the data to provide ACID guarantees, time travel, and reliable upserts and schema management. Writing in the Delta format ensures Spark creates both the Parquet data files and the Delta transaction log, enabling features like MERGE/UPDATE/DELETE and concurrent writes with consistent reads. If you wrote as Parquet (or as CSV/JSON), you’d store data efficiently but miss the Delta transaction log and the associated capabilities. Therefore, using the Delta format is the correct approach.