Databricks Databricks-Certified-Professional-Data-Engineer 問題集

質問 1：
A Spark job is taking longer than expected. Using the Spark UI, a data engineer notes that the Min, Median, and Max Durations for tasks in a particular stage show the minimum and median time to complete a task as roughly the same, but the max duration for a task to be roughly 100 times as long as the minimum.
Which situation is causing increased duration of the overall job?
A. Network latency due to some cluster nodes being in different regions from the source data
B. Skew caused by more data being assigned to a subset of spark-partitions.
C. Credential validation errors while pulling data from an external system.
D. Task queueing resulting from improper thread pool assignment.
E. Spill resulting from attached volume storage being too small.
正解：B
解説: (Topexam メンバーにのみ表示されます)

質問 2：
A data pipeline uses Structured Streaming to ingest data from kafka to Delta Lake. Data is being stored in a bronze table, and includes the Kafka_generated timesamp, key, and value. Three months after the pipeline is deployed the data engineering team has noticed some latency issued during certain times of the day.
A senior data engineer updates the Delta Table's schema and ingestion logic to include the current timestamp (as recoded by Apache Spark) as well the Kafka topic and partition. The team plans to use the additional metadata fields to diagnose the transient processing delays:
Which limitation will the team face while diagnosing this problem?
A. New fields not be computed for historic records.
B. Spark cannot capture the topic partition fields from the kafka source.
C. Updating the table schema will invalidate the Delta transaction log metadata.
D. Updating the table schema requires a default value provided for each file added.
正解：A
解説: (Topexam メンバーにのみ表示されます)

質問 3：
A table is registered with the following code:

Bothusersandordersare Delta Lake tables. Which statement describes the results of queryingrecent_orders?
A. All logic will execute when the table is defined and store the result of joining tables to the DBFS; this stored data will be returned when the table is queried.
B. Results will be computed and cached when the table is defined; these cached results will incrementally update as new records are inserted into source tables.
C. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query finishes.
D. The versions of each source table will be stored in the table transaction log; query results will be saved to DBFS with each query.
E. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query began.
正解：A

質問 4：
In order to facilitate near real-time workloads, a data engineer is creating a helper function to leverage the schema detection and evolution functionality of Databricks Auto Loader. The desired function will automatically detect the schema of the source directly, incrementally process JSON files as they arrive in a source directory, and automatically evolve the schema of the table when new fields are detected.
The function is displayed below with a blank:

Which response correctly fills in the blank to meet the specified requirements?

A. Option E
B. Option A
C. Option C
D. Option D
E. Option B
正解：E
解説: (Topexam メンバーにのみ表示されます)

質問 5：
In order to prevent accidental commits to production data, a senior data engineer has instituted a policy that all development work will reference clones of Delta Lake tables. After testing both deep and shallow clone, development tables are created using shallow clone.
A few weeks after initial table creation, the cloned versions of several tables implemented as Type 1 Slowly Changing Dimension (SCD) stop working. The transaction logs for the source tables show that vacuum was run the day before.
Why are the cloned tables no longer working?
A. The metadata created by the clone operation is referencing data files that were purged as invalid by the vacuum command
B. Because Type 1 changes overwrite existing records, Delta Lake cannot guarantee data consistency for cloned tables.
C. Running vacuum automatically invalidates any shallow clones of a table; deep clone should always be used when a cloned table will be repeatedly queried.
D. The data files compacted by vacuum are not tracked by the cloned metadata; running refresh on the cloned table will pull in recent changes.
正解：A
解説: (Topexam メンバーにのみ表示されます)

質問 6：
A data architect has designed a system in which two Structured Streaming jobs will concurrently write to a single bronze Delta table. Each job is subscribing to a different topic from an Apache Kafka source, but they will write data with the same schema. To keep the directory structure simple, a data engineer has decided to nest a checkpoint directory to be shared by both streams.
The proposed directory structure is displayed below:

Which statement describes whether this checkpoint directory structure is valid for the given scenario and why?
A. Yes; Delta Lake supports infinite concurrent writers.
B. Yes; both of the streams can share a single checkpoint directory.
C. No; Delta Lake manages streaming checkpoints in the transaction log.
D. No; only one stream can write to a Delta Lake table.
E. No; each of the streams needs to have its own checkpoint directory.
正解：E
解説: (Topexam メンバーにのみ表示されます)