Exam AWS Certified Data Engineer - Associate DEA-C01 topic 1 question 166 discussion

A = primary mechanism (event-driven incremental updates) B = fallback mechanism (periodic incremental updates)

A,B. Answer A and B. A) S3 event-based Glue crawler: configure S3 to send object events to SQS, and set the crawler to “Crawl based on events” using that queue. The crawler ingests only changes (incremental) and avoids full listings. B) Time-based schedule. Event-based crawlers still run on a schedule to poll SQS. If there are events, they update the Catalog.

C has higher operational effort: you’d need to write/maintain Lambda code for schema inference, catalog updates, and error handling. Glue already provides managed crawlers.

• A leverages the event-driven capability of Glue Crawlers. • C uses AWS Lambda for direct and real-time updates to the Data Catalog. • This combination ensures incremental updates are made only when changes occur, reducing costs and operational complexity.

(A) S3 Event-Based Crawler: Automatically triggers incremental catalog updates whenever new data arrives in the S3 bucket, reducing the need for custom code and manual intervention. (B) Time-Based Schedule: Periodically runs the crawler to catch any missed events and keep the data catalog accurate and up to date. Using both methods minimizes operational overhead while ensuring comprehensive and reliable incremental updates.

Check out the design pattern documentation for this case. There's no need for Lambda here, so option C should be excluded. Option B seems viable, along with option A (A is the obvious choice for me). https://aws.amazon.com/blogs/big-data/run-aws-glue-crawlers-using-amazon-s3-event-notifications/

B and D are wrong due too "Manually" and "Scheduling". E is too much for this use case

- Option A suggests creating an S3 event-based AWS Glue crawler to consume events from the SQS queue. This option is appropriate as it allows the crawler to automatically respond to events, thereby reducing manual intervention and ensuring timely updates to the Data Catalog - Option C involves using an AWS Lambda function to directly update the Data Catalog based on S3 events received from the SQS queue. This is a strong candidate as it automates the update process without the need for manual scheduling or intervention, thus minimizing operational overhead. AWS Glue Crawlers can consume events from an SQS queue: https://docs.aws.amazon.com/glue/latest/dg/crawler-s3-event-notifications.html

Based on this article (Option 1 for the architecture) it should be AB: 1. Run the crawler on a schedule. 2. Crawler polls for object create events in the SQS queue 3a. If there are events, crawler updates the Data Catalog 3b. If not, crawler stops

AWS Glue Crawlers can not consupe events from an SQS queue D introduce a manual operation E introduce more complexity so BC