#36. A few moments later
Topics: Architecture, data quality, dbt, Apache Spark, streaming
Simplify PySpark testing with DataFrame equality functions — Haejoon Lee, Allison Wang and Amanda Liu @ Databricks Engineering Blog
Finally we have PySpark functions for testing! Starting from Spark 3.5. No more additional libraries for testing. Or, maybe…
Data Quality Score: How We Evolved the Data Quality Strategy at Airbnb — Clark Wright @ Netflix Data Engineering Open Forum 2024
Some time ago, we published an article from AirBnb about the Data Quality framework that they had built. Now, you may watch the video for more details. Inspired by the level of platform solutions that they’re building.
Accelerating and Scaling dbt for the Enterprise — Dakota Kelley @ phData blog
This article highlights problems that you have to solve when preparing DBT as part of the Platform in your organization.
Course Events and Event Streaming — Adam Bellemare @ Confluent Developer
If you, as a data engineer, have discussions about proper event modeling in your company, then this video course is for you (and for your software engineers).
InfoQ Software Architecture and Design Trends Report - April 2024 — InfoQ
Have you already heard about architecture as a team sport? About cell-based architecture? Review the latest innovations and trends in the design trends report.
