#25. Back to basics
Topics: Apache Arrow, culture, data lineage, data warehouse, Apache Kafka
From Eager to Smarter in Apache Kafka Consumer Rebalances — Sophie Blee-Goldman @ Confluent Blog.
We continue the Kafka topic. New rebalancing protocol from consumer client perspective.
Open Sourcing Amundsen: A Data Discovery And Metadata Platform — Lyft Engineering Blog
Great overview and how-to guide about building a data discovery platform. Additionally, I’ve never heard about Databuilder framework.
Write Better Commits, Build Better Projects — Github Engineering Blog
The article on how to rethink writing commits. At first glance, it may seem that investing time in writing meaningful commits wastes time. But everyone can remember being stuck trying to catch a feature or a bug. In the commits like “fix bug A” or “add feature X” without adding any context. Authors offer to be more meaningful and add understanding with the help of commits, thereby helping the reviewer and developers.
BigQuery vs Synapse — who is currently winning the battle? — Datasparq Technology Blog
Even if you aren’t interested in BigQuery and Synapse, it’s worth watching for great meme about infrastructure costs :)
Introducing the Gandiva Initiative for Apache Arrow — Dremio Blog
Let’s break from our articles about Data Mesh and Data Lakehouses and go back to 2018. To Arrow and Gandiva.
Smart Data 2022 on October 17-18 (online) and 29th (offline + online), 2022.
SmartData is a data engineering conference. It’s full of tech talks related to all the topics that are important for a data engineer, ranging from reliability to MLOps. The conference is held by JUG Ru Group.
Do not miss the Community Day. Get the free ticket for the last day (18th) of Online conference.
Trino Summit on November 10th, 2022.
Trino Summit is an event that convenes engineers, analysts, and superfans of the Trino project. This year, Trino Summit will be hosted as a hybrid event at the Commonwealth Club in San Francisco, CA on November 10th. For those who may not have heard, Trino is the new moniker for the project formerly known as PrestoSQL. We are thrilled to pull together the Trino community again to share our experiences, meet with professionals in the big data and analytics space, and grow your skills and learn about new possibilities.