Exam Certified Data Engineer Professional topic 1 question 130 discussion

Actual exam question from Databricks's Certified Data Engineer Professional

Question #: 130
Topic #: 1

[All Certified Data Engineer Professional Questions]

The following table consists of items found in user carts within an e-commerce website.

The following MERGE statement is used to update this table using an updates view, with schema evolution enabled on this table.

How would the following update be handled?

A. The update throws an error because changes to existing columns in the target schema are not supported.
B. The new nested Field is added to the target schema, and dynamically read as NULL for existing unmatched records.
C. The update is moved to a separate "rescued" column because it is missing a column expected in the target schema.
D. The new nested field is added to the target schema, and files underlying existing records are updated to include NULL values for the new field.

Show Suggested Answer

Suggested Answer: B 🗳️

by MDWPartners at May 29, 2024, 6:55 p.m.

Comments

Submit Cancel

Freyr

Highly Voted 1 year, 5 months ago

Selected Answer: B

Correct Answer: B. In the question it is mentioned that the schema evolution is enabled. This option states that the new nested field is added to the target schema, and existing records not matching the new schema format are populated with NULL for the newly added columns. This behavior aligns with how schema evolution functions in Delta Lake, dynamically adapting the schema to include new fields.

upvoted 5 times

...

_Lukas_

Most Recent 1 day, 9 hours ago

Selected Answer: B

Correct: B - With schema evolution enabled (spark.databricks.delta.schema.autoMerge.enabled), Delta Lake detects the new coupon field inside the items array of the source data. It automatically updates the target table's schema to include this new nested field. For all existing records (like 1002 and 1003) that were not updated in this transaction and reside in old files, the new column simply doesn't exist physically. Delta Lake handles this at read time by returning NULL for the missing field, without needing to rewrite the underlying files.

upvoted 1 times

...

KadELbied

6 months, 2 weeks ago

Selected Answer: B

Suretly B

upvoted 1 times

...

HairyTorso

10 months, 4 weeks ago

Selected Answer: B

Seems like B is correct if below config is enabled: SET spark.databricks.delta.schema.autoMerge.enabled = true;

upvoted 1 times

...

pppppppppie

1 year, 2 months ago

because B says for unmatched records. but my new records has 1001 which is existing record. we should give ans what will happen with this new record. it will update the record in target with NULL as coupon

upvoted 1 times

...

pppppppppie

1 year, 2 months ago

It has to be D

upvoted 1 times

...

MDWPartners

1 year, 6 months ago

Selected Answer: B

schema evolution is enabled, so B.

upvoted 4 times

...