Simplify PySpark testing with DataFrame equality functions — Haejoon Lee, Allison Wang and Amanda Liu @ Databricks Engineering Blog

Finally we have PySpark functions for testing! Starting from Spark 3.5. No more additional libraries for testing. Or, maybe…

Data Quality Score: How We Evolved the Data Quality Strategy at Airbnb — Clark Wright @ Netflix Data Engineering Open Forum 2024

Some time ago, we published an article from AirBnb about the Data Quality framework that they had built. Now, you may watch the video for more details. Inspired by the level of platform solutions that they’re building.

Accelerating and Scaling dbt for the Enterprise — Dakota Kelley @ phData blog

This article highlights problems that you have to solve when preparing DBT as part of the Platform in your organization.

Course Events and Event Streaming — Adam Bellemare @ Confluent Developer

If you, as a data engineer, have discussions about proper event modeling in your company, then this video course is for you (and for your software engineers).

InfoQ Software Architecture and Design Trends Report - April 2024 — InfoQ

Have you already heard about architecture as a team sport? About cell-based architecture? Review the latest innovations and trends in the design trends report.

Seamlessly Migrate Your Apache Parquet Data Lake to Delta Lake — Dipankar Kushari, Uday Satapathy @ Databricks Engineering Blog

Databricks is a company behind the Delta Lake format. The paper explains some drawbacks of building a data lake using Apache Parquet and explains how Delta Lake can solve such problems and how to migrate.

How Meta built the infrastructure for Threads — Laine Campbell, Chunqiang (CQ) Tang @ Engineering at Meta

It’s always interesting to read/watch real system’s design with explanation. Especially when you are trying to design them in mind.

Python 3.13 gets a JIT — Anthony Shaw

It’s not only data engineering, but probably will change the whole development landscape: Python gets JIT, which opens a door to massive performance improvements in future!

How we built our customer data warehouse all on Postgres — Adam Hendel @ Tembo

Very interesting experience of usage of Postgres as a warehouse. I’m saying not only about storage, but the whole system including orchestration 🤯. Of course authors had to write some code in Rust, but the concept looks very interesting and maybe even promising.

Data Domains — Where do I start? — Piethein Strengholt

Good and practical article about data domains. Not only from a data team perspective but also from a development perspective.

Is Kimball Still Relevant? — Joe Reis Blog

Quite a vital question! Links inside the post also have vital questions. If you’re thinking of dropping a detailed layer on DWH, you should read it.

The Zoo of Consistency Models — The Educative Team

Consistency models are something that you have constantly refresh in your memory.

Seven Principles of Cloud-Native Architecture — Alibaba Cloud Native Community Blog

Clear explanation of cloud-native architecture principles.

Streaming SQL in Data Mesh — Netflix Blog

If you’re writing a Data Platform in your company, there are not too many insights here. But at least one is guaranteed: how the Data Platform’s UI can look.

Data Quality Score: The next chapter of data quality at Airbnb — Airbnb blog

Absolutely brilliant approach of identifying data quality for all company data assets. Quite realistic approach.

Dynamic Filtering: a Critical Performance Optimization in Analytical Engines — Vladimir Ozerov @ Querify Labs Blog

Let’s continue getting acquainted with the query engine optimization techniques with developers of these query engines. Now it’s dynamic filtering time.

Data Mesh in practice: Product thinking and development (Part III) — Ammara Gafoor, Ian Murdoch, Kiran Prakash @ Thoughtworks Blog

There are 4 articles in this series, but I want to share this one with you because Data Product itself is closer to data engineers (from my point of view). But we’re often not aware of what it truly is and how is it built. So let’s fill the gap.

Other articles in the series: Part I, Part II, Part IV.

Privacy Enhancing Technologies: An Introduction for Technologists — Katharine Jarmul

A new article about Privacy Enhancing Technologies (PETs) in Martin Fowler’s blog. In the era of systems like ChatGPT or Stable Diffusion personal data privacy is especially important. This article is a great intro to PETs and provides some use cases on how to protect your data in the modern world.

The State of Data Engineering 2023 — Einat Orr @ lakeFS blog

It’s time to reveal new State of Data Engineering with lakeFS!

Why Uber Engineering Switched from Postgres to MySQL — Uber Engineering Blog

Uber started with a monolithic backend application that used Postgres. As the company evolved and grew, it moved to microservices and changed its approach to working with data. The paper explains why this migration happened and what benefits the company gets from it. There is nothing about OLAP in the paper but still a very interesting story.

Spark SQL Query Engine Deep Dive – Adaptive Query Execution: part I & part II – Linxiao Ma

It’s absolutely brilliant blog. I adore it. These articles declares Spark Adaptive Query Execution very deeply.

5 Helpful Extract & Load Practices for High-Quality Raw Data — Sven Balnojan

What layer do you focus on when creating data architecture? If it’s not Raw, then this article is for you. 5 points that will save your time n future.

Following a database read to the metal — Hussein Nasser

Exciting adventure into the query path to data.

Spark Connect Available in Apache Spark 3.4 — Databricks Blog

Just to keep in touch with a new Spark features. Give it up for Spark Connect.

Functional Data Engineering — a modern paradigm for batch data processing — Maxime Beauchemin

It’s very old. But quite conceptual.

Software Architecture and Design InfoQ Trends Report - April 2023 — InfoQ

Data engineers are dealing with code as much as with data. It means that software architecture and design are not alien to us.

Introducing Entity-Centric Data Modeling for Analytics — Maxime Beauchemin @ Preset Blog

Entity-centric data modeling (ECM) introduction from Maxime Beauchemin, CEO Preset and creator and lead committer on Apache Airflow and Apache Superset. Just imagine that metrics can also live in the dimension tables and go to read.

The end of a myth: Distributed transactions can scale — Murat Demirbas Blog

Analysis of the paper of the same name. If you want to switch your mind, then remote direct memory access and snapshot isolation are waiting for you.

Using Metrics Layer to Standardize and Scale Experimentation at DoorDash — Arun Balasubramani @ DoorDash engineering blog

It’s not the first article about metric’s layer in our blog, but again it worth to be seen. Motivation and challenges are waiting for you.

Your Data Catalog Shouldn’t Be Just One More UI — Mahdi Karabiben

When a data catalog is not only for business users.

Concurrency is not Parallelism by Rob Pike (slides) — Rob Pike

Rob Pike is a software engineer, best known for his work on the Go programming language. Quite old but still relevant talk discussing the crucial distinctions between concurrency and parallelism. Such foundational engineering concepts are cross disciplines and I would say must-have knowledge for all engineers.

Setting Uber’s Transactional Data Lake in Motion with Incremental ETL Using Apache Hudi — Uber Engineering Blog

As usual, Uber’s blog is fantastic resource for new ideas. This fresh article explains how to build streaming ETL framework and how to address some well-known issue on this way.

Apache Spark — Job monitoring — Hareesha Dandamudi

Short and understandable article about SparkListener. Quite a useful thing for creating monitoring and lineage handling (as OpenLineage does). Just use it as a start for your own articles research :)

The Question That Every Data Engineer Should Ask — Xinran Waibel @ Data Engineer Things Blog

Vital and clickbait.

Pushdown — Trino Query optimizer docs

Pushdown is a powerful query optimization that moves predicates in the WHERE clause closer to the tables they refer to. This is Trino docs, but you can go through all types and find them in other query engines. It can help you in reading query plans a lot.

Solving Advent of Code with DuckDB and DBT — Graham Wetzler

Have you ever thought about solving Advent of Code using SQL? While ones dream others do.

Bonus: Do you know about Advent of Data? It is a set of articles on various data topics designed in the advent calendar style.

Distinct aggregation optimization in Apache Calcite and Trino — Querify Labs Blog

Good and detailed description of how DISTINCT is implemented in Calcite and Trino engines.

The Chaos Data-Engineering Manifesto — Shane Murray @ Towards Data Science Blog

Very vital and easy-to-read article about the adaptation of chaos engineering to meet the needs of data engineers. “Nothing acts like prod, but prod. And nothing fails like prod, but prod.”

Essential Snowflake Cost Reduction Strategies — Niall Woodward @ select.dev Blog

Cloud can help you to release MVP as soon as possible. But also in final you can be bare as a bone. How to prevent it?

The Data Engineer’s Roadmap — James Phoenix

You will say: “Oh, another one article about how to be a data engineer.” But! Firstly, there are new data engineers who want to grow. Secondly, we have team leads here so that they can share this article with colleagues. Additional materials included <3

BIG DATA IS DEAD — Jordan Tigani

When size doesn’t matter.

Title: 5 Ways to Use Column Level Data Lineage — Montecarlo Data Blog

Have you ever thought of implementing column level lineage? If not, just read why it can be extremely helpful to have.

Why your mock doesn’t work — Ned Batchelder

I often come across the situations where my Mock didn’t work as I expected. I hate Python for that. This pretty short article showed me that I should hate someone else [most likely underinvestment into language]. I just misunderstood the basics. Must-read article which provides a brief explanation of how exactly variable assignment in import statements works in Python and what we should keep in mind working with Mock.

Driving efficiency and developer productivity at Facebook scale, Asynchronous computing at Meta: Overview and learnings — Engineering at Meta Blog

By the name you could think it is something from other world. You are right but only partially. Actually these are 2 articles of how to build a mix of scalable ETL and distributed computation in-house.

How Amazon RDS Replication Works and Why the FAA’s Database Problem Won’t Happen in AWS — Bohan Zhang, Andy Pavlo @ OtterTune Blog

Basic overview of how Amazon RDS and Amazon Aurora replication works. A good starting point to better understand what database replication is.

Databases in 2022: A Year in Review — Andy Pavlo @ OtterTune blog

Interesting review by Andy Pavlo about databases’ state by the end of 2022. Andy touched on database companies’ funding situation, blockchain, new database systems which getting popularity in 2022, and a few other topics.

Cost Efficiency @ Scale in Big Data File Format — Uber Engineering blog

If you need to choose compression type for parquet files in you data lake, this article is good starting point.

A deep dive into Spanner’s query optimizer — Campbell Fraser, Vlad Lifliand @ Google Cloud Blog

A good introduction to how the query optimizer works on a simple example. In this article, you will find what types of optimizations are used in Spanner, what optimizer statistics are collected, and how to deal with different optimizer versions.

Micropipelines: A Microservice Approach for DAG Authoring in Apache Airflow — Vikram Koka @ Astronomer Blog

No more monolithic pipelines. Please put your hands together for micropipelines!

Learn to Efficiently Test ETL Pipelines — Jacqueline Bilston

This is an absolutely amazing presentation about data pipelines unit testing. Hadn’t seen any resource that was so specific about data pipeline testing so far.