Articles tagged with pipelines

How we deal with Data Quality using Circuit Breakers — Sandeep Uttamchandani @ Wrong AI.

This is an interesting way to use circuit breaker patterns inside the pipeline processes to prevent low-quality data from propagating to downstream processes.

level:advanced topic:data-quality topic:pipelines


Incrementally Adopting Dagster at Mapbox — Ben Pleasanton @ Dagster Blog.

We all know that Dagster may be (and maybe already is) the Next Big Thing after Airflow. But how many success stories did you read? Yes, adoption grows, but something impossible with Airflow? Well, folks from Mapbox hit the wall of complexity with Airflow, and they choose Dagster to cope with this complexity.

level:medium topic:dagster topic:pipelines


Data Fusion: A Code-Free Pipeline for Your Google Cloud Data Warehouse — Guillaume Dupont @ Towards Data Science.

Everybody loves to get results without writing a code. The most popular solution for building pipelines without code is Apache NiFi. But what if one can build the whole warehouse with all processes inside without code? Looks like Google Data Fusion may be used for it!

level:medium topic:pipelines