This week's issue includes articles on Pinterest’s time series database, deploying Elasticsearch on Kubernetes, data & ML engineering at Slack, Airbnb's job queuing system, and the architecture of the Cliqz search engine. There are also a slew of posts on other topics from optimizing Kafka consumer to JSON processing in Golang.

A look at the inner workings of the Kafka consumers, with some real world recommendations for deploying them when there's high latency in talking to the Kafka cluster and/or a large number of partitions. There are tips on important metrics to monitor, configurations, garbage collector settings, and changing the partition.class to improve unbalanced consumers.

https://www.jesseyates.com/2019/12/04/vertically-scaling-kafka-consumers.html

This article looks at why hash maps (unsorted) are popular for in-memory indexes whereas b-tree (sorted) are common in databases. It describes the trade-offs of the two approaches, and how those best fit in-memory/db use cases.

https://www.evanjones.ca/ordered-vs-unordered-indexes.html

This post describes a large scale conversion of data into JSON format in order to load it into BigQuery. To meet the naming requirements of BigQuery, they had to remap field names on every JSON document. Their tool, which is written in Golang, uses a producer/consumer job queue to parallelize processing and partition the data before writing it out. They processed data both from Kafka and S3, and the post talks a bit about how they optimized interaction with S3.

https://itnext.io/parsing-18-billion-lines-json-with-go-738be6ee5ed2

Pinterest writes about how they've extended their time series data store, Goku, to support querying of historical data. They tier data by compacting data through rebucketing and downsampling. For serving, they load data from S3 into RocksDB. The post goes into the details of the design of their RocksDB setup, cluster management functions, and the query processing framework.

https://medium.com/pinterest-engineering/gokul-extending-time-series-data-storage-to-serve-beyond-one-day-52264307364d

An overview of Kubernetes and how to deploy Elasticsearch on Kubernetes. This is a great introduction to many of the core concepts of Kubernetes (e.g. Deployment, Pod), including those that are important for running a stateful service (e.g. StatefulSet, PersistentVolumeClaim). It also shows how to configure accounts for ES using Kubernetes RBAC and use the helm package manager for deploying to Kubernetes.

https://sematext.com/blog/kubernetes-elasticsearch/

Available both in audio form and as a transcript, InfoQ has a podcast with Josh Wills that covers the evolution of data engineering and machine learning at Slack. The interview covers their data pipeline, which feeds into big data systems for BI/data warehousing and ML products. The interview also covers the kinds of products they build with machine learning and some thoughts on the future of observability for ML pipelines.

https://www.infoq.com/podcasts/slack-building-resilient-data-engineering/

Airbnb has open sourced Dynein, their job queuing system that they use for offloading tasks from the main request path and performing other asynchronous operations. It uses DynamoDB as a scheduler for future jobs and SQS for queuing—the post describes how this is built in a highly scalable way.

https://medium.com/airbnb-engineering/dynein-building-a-distributed-delayed-job-queueing-system-93ab10f05f99

When you're running a high throughput system in Java, issues with garbage collection are inevitable. This post provides details on how to enable GC logs, how to interpret the details from the Concurrent Mark Sweep and the G1GC collectors, and some tools for visualizing the output. This is one of the most comprehensive guides that I've seen on the topic.

https://sematext.com/blog/java-garbage-collection-logs/

Cliqz, makers of a web search engine, have a comprehensive post on their architecture. Their near real-time indexing system is built with Apache Kafka, Apache Cassandra, and RocksDB while their batch indexing system is built on MapReduce and Spark with Luigi for managing workflows. The post also describes how they manage Kubernetes clusters, use Helm/Helmfile for package management, and leverage Tilt and K9s for local development. They also share on how they optimize costs and describe their machine learning pipelines.

https://www.0x65.dev/blog/2019-12-14/the-architecture-of-a-large-scale-web-search-engine-circa-2019.html

Events

Curated by Datadog ( http://www.datadog.com )

California 

Building a Best-in-Class Data Lake on AWS and Azure (Santa Clara) - Tuesday, December 17

https://www.meetup.com/datariders/events/266951424/

North Carolina

Zero to Observability with Apache Kafka (Raleigh) - Tuesday, December 17

https://www.meetup.com/Raleigh-Apache-Kafka-Meetup-by-Confluent/events/266917829/

CANADA

Introduction to Kafka (Montreal) - Wednesday, December 18

https://www.meetup.com/montreal-jug/events/266729844/

BRAZIL

Data Engineering Meetup (Belo Horizonte) - Wednesday, December 18

https://www.meetup.com/engenharia-de-dados/events/267072117/

SERBIA

Event Deduplication in Kafka + Navigation in a 3D Environment with RL (Novi Sad) - Wednesday, December 18

https://www.meetup.com/Big-Data-Novi-Sad/events/267060354/

ISRAEL

Spark Advanced Topics (Tel Aviv-Yafo) - Thursday, December 19

https://www.meetup.com/Women-in-Big-Data-Israel/events/266728256/

RUSSIA

Real-Time Data: Streaming and Collecting Data in Real Time (Moscow) - Wednesday, December 18

https://www.meetup.com/Data-People/events/266992802/

INDIA

Designing ETL Pipelines with Structured Streaming and Delta Lake (Bengaluru) - Wednesday, December 18

https://www.meetup.com/Bangalore-Apache-Spark-Meetup/events/266970481/

Apache Kafka and Stream Processing Meetup @ Walmart (Bengaluru) - Sunday, December 22

https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/266777028/

SRI LANKA

Insights into HDInsights (Colombo) - Wednesday, December 18

https://www.meetup.com/sldatacommunity/events/267042058/

SOUTH KOREA

Flink Meetup (Seoul) - Tuesday, December 17

https://www.meetup.com/Seoul-Apache-Flink-Meetup/events/266824815/

Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.

After a few weeks off (hopefully folks in the US had a nice Thanksgiving!), we're back with your weekly fix of data engineering articles. Apache Kafka and Apache Airflow are covered from several angels in this issue, and there are posts on the future of data engineering, columnar file formats, bloom filters, and Cruise's platform for data pipelines. Lots of great posts from folks building large scale data platforms!

This article provides an overview of a talk at the recent QCon San Francisco on the future of data engineering. The talk covers six stages of data engineering and what it takes to evolve from one stage to the next through the lense of the data architecture a WePay. The talk also covers what's ahead in the field. If you want to dive in more, the article links out to the slides for the presentation.

https://www.infoq.com/news/2019/11/data-engineering-future-qconsf/

Zeebee is a workflow engine that can be used to execute and/or monitor workflows that span multiple microservices. This post looks at how it can integrate with Apache Kafka—as a source of data for monitoring or as a sink for publishing information about the state of the workflow. There are several good diagrams in the post to illustrate the key concepts.

https://blog.bernd-ruecker.com/zeebe-loves-kafka-d82516030f99

Zulily writes about how they have evolved their Apache Airflow architecture—moving from celery executors to the Kubernetes executor, leveraging AWS RDS for metadata, and using AWS EFS for the DAGs. The post also describes their CI/CD workflow, and more.

https://zulily-tech.com/2019/11/19/evolution-of-zulilys-airflow-infrastructure/

A good introduction to the Apache Kafka Client Consumer's PartitionAssignor strategies. The post covers the three builtin strategies (Range, RoundRobin, StickyAssignor), the StreamsPartitionAssignor from Kafka Streams, and how to implement a custom strategy. As an example, the post walks through building a FailoverAssignor that could be used for an active/passive setup.

https://medium.com/streamthoughts/understanding-kafka-partition-assignment-strategies-and-how-to-write-your-own-custom-assignor-ebeda1fc06f3

This article provides a good introduction to columnar file formats—describing how they physically store data (with an example of translating a CSV to a columnar CSV format), the benefits of columnar formats, and some of the trade-offs.

https://blog.matthewrathbone.com/2019/11/21/guide-to-columnar-file-formats.html

Pinterest writes about the service they've built to support large amounts of offline updates to their sharded MySQL cluster. This service, which exposes APIs for batch write operations, groups writes/updates based on operation type and shard. It also uses Kafka as a buffer—consumers fetch batch operation details and write to MySQL at a configured rate limit to keep the load of offline operations from impacting user-interactive queries. The post dives into technical details, including how they handle hot shards, variation in write operations, and the improvements they've seen from this new system.

https://medium.com/pinterest-engineering/using-kafka-to-throttle-qps-on-mysql-shards-in-bulk-write-apis-a326ae0f1ac1

A look at several techniques for monitoring Apache Kafka and related components. The post describes Quantyca's approach using MetricBeat with Burrow and Elastalert. They have an example of sending an alert to Slack.

https://medium.com/quantyca/how-to-monitor-your-kafka-cluster-efficiently-d45ce37c02f1

Azkarra Streams is a new framework for building Apache Kafka Streams applications. It provides a library that eliminates a lot of the boilerplate of a typical streams application, and it has a built in HTTP server to monitor the state of your application(s), a simple DAG visualizer, and a builtin HTTP request endpoint to query your Kafka Streams stores (along with a web UI to look at results).

https://medium.com/streamthoughts/introducing-azkarra-streams-the-first-micro-framework-for-apache-kafka-streams-e13605f3a3a6

Cruise writes about Terra, their platform built on the Apache Beam SDK for data pipelines. Terra supplements Beam's features by adding permissions management, job submission (including pulling python/C++ dependencies), lineage, and more. The post has some sample code that shows how it all fits together.

https://medium.com/cruise/introducing-terra-cruises-data-processing-platform-c6a476bb5b72

Airtunnel is a new open source project that provides blueprints for building Apache Airflow DAGs. The project is designed for several design principles: consistency (e.g. in naming of data sets, scripts, and workflows), declarative first (Airtunnel uses YAML to declare data assets), and metadata driven. Airtunnel, which includes custom operators, metadata extensions to collect data asset lineage, and more, is available on github.

https://medium.com/bcggamma/airtunnel-a-blueprint-for-workflow-orchestration-using-airflow-173054b458c3

Bloom Filters are ubiquitous in distributed data stores because they can eliminate certain expensive operations. This post dives into the features of a bloom filter, how it works, and contains a basic implementation in Python.

https://diogodanielsoaresferreira.github.io/bloom-filter/

Events

Curated by Datadog ( http://www.datadog.com )

California

Off the Ground w/ Apache Airflow + Ordinary People w/ Ability for Extraordinary (Santa Monica) - Thursday, December 12

https://www.meetup.com/LA-HUG/events/264629190/

Minnesota

Apache Kafka Committer & Co-Founder Jun Rao on Why Kafka Needs No Keeper (Minneapolis) - Monday, December 9

https://www.meetup.com/TwinCities-Apache-Kafka/events/266301147/

Illinois

HCSC Big Data Hadoop Meetup (Chicago) - Wednesday, December 11

https://www.meetup.com/HCSC-Technology-Group/events/266496994/

Georgia

Running Apache Airflow at Kabbage (Atlanta) - Tuesday, December 10

https://www.meetup.com/BigDataATL/events/266664587/

North Carolina

Building a Stream Processing Architecture with Apache Kafka (Charlotte) - Wednesday, December 11

https://www.meetup.com/Charlotte-Java-Developers-Meetup/events/266370651/

New York

An Introduction to Kafka Streams and KSQL (Webster) - Tuesday, December 10

https://www.meetup.com/RTG-Rochester-Technology-Group/events/266463037/

Massachusetts

December Apache Spark Meetup (Cambridge) - Tuesday, December 10

https://www.meetup.com/Boston-Apache-Spark-User-Group/events/265735754/

IRELAND

Data Science and Engineering Club (Dublin) - Wednesday, December 11

https://www.meetup.com/Data-Science-and-Engineering-Club/events/266969502/

PORTUGAL

Apache Kafka: Metamorphosis (Lisboa) - Thursday, December 12

https://www.meetup.com/Tech-Mate/events/266757230/

NETHERLANDS

From Hadoop to NoSQL & Graph to Translytical (Middelharnis) - Tuesday, December 10

https://www.meetup.com/Code-in-the-Middel/events/266260609/

GERMANY

The Learnings of Karate Kid Applied to Apache Kafka (Berlin) - Wednesday, December 11

https://www.meetup.com/Berlin-Apache-Kafka-Meetup-by-Confluent/events/266567425/

Apache NiFi + Hacking Around the IoTree (Frankfurt) - Wednesday, December 11

https://www.meetup.com/IoT-Hessen/events/265584374/

Kubernetes with Kafka Flavor (Berlin) - Thursday, December 12

https://www.meetup.com/DigitalOceanBerlin/events/266915196/

ITALY

Using PySpark with Google Colab + Spark 3.0 Preview (Milano) - Wednesday, December 11

https://www.meetup.com/Spark-More-Milano/events/266753286/

It's a Streamer World! A Journey Through Processing Flows of Data (Milano) - Wednesday, December 11

https://www.meetup.com/Milano-Kafka-meetup/events/266750779/

POLAND

First Warsaw Apache Airflow Workshop (Warsaw) - Friday, December 13

https://www.meetup.com/Warsaw-Airflow-Meetup/events/266996789/

INDIA

Building Consciousness on Real Time Events: ksqlDB Recipes (Chennai) - Wednesday, December 11

https://www.meetup.com/Chennai-Kafka/events/266140963/

Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.

A lot of breadth in this week's coverage of data engineering topics—from Delta to event processing at Spotify to the design goals of TileDB to understanding delivery guarantees in Apache Kafka. Also, a few articles on distributed systems—Yelp's autoscaling service, techniques for building reliable systems, and Facebook’s global routing infrastructure.

A look at how to use Delta's version tracking to "time travel" (read the contents of a table as it was at some previous point in time) and to produce audit details to track changes to a table. The post embeds several IPython notebooks with sample Scala code.

https://medium.com/@aravinthR/delta-time-travel-for-data-lake-part-2-b23879a4bc6d

Yelp writes about their newly open-sourced autoscaler for Kubernetes, Clusterman. Compared to the Kubernetes Autoscaler, there are some interesting features like support for spot instances and the ability to simulate production workloads.

https://engineeringblog.yelp.com/2019/11/open-source-clusterman.html

With the complexity that comes with a distributed system, it can be overwhelming to start the process of identifying and classifying failure modes. This post provides a great high level blueprint for how to approach this problem—including bucketing types of failure through Failure Modes and Effects Analysis and calculating a Risk Priority Number for each item in your analysis. Lots of great stuff (and spreadsheets to help!) for understanding how to build a reliable system.

https://medium.com/@adrianco/failure-modes-and-continuous-resilience-6553078caad5

Spotify shares lessons learned from operating a large-scale event delivery system, which powers some key business functions like tracking track plays to calculate royalties. They describe some of the principles (e.g. segregating data by event type and choosing liveness over lateness), how they think about Service Level Objectives for event delivery, and how they are exposing their event platform to other internal teams.

https://labs.spotify.com/2019/11/12/spotifys-event-delivery-life-in-the-cloud/

Coupang, an e-commerce company from South Korea, writes about the evolution of their data platform over the past several years. They've been through several phases: storing all data in a relational database, using Hadoop, Hive, & a MPP database, and migrating to the cloud. The post dives into some of the other important features of their pipeline, like how they track data quality, detect data abnormalities, and make data discoverable to users.

https://medium.com/coupang-tech/evolving-the-coupang-data-platform-308e305a9c45

TileDB is a new storage engine for multi-dimensional data like that commonly used for machine learning and genetics analysis. It supports both sparse and dense arrays, and it can use blob storage (such as S3) as the storage backend. The post describes the motivation for a new storage system, and how they've optimized the implementation for efficiency and cross language support.

https://medium.com/tiledb/tiledb-a-database-for-data-scientists-ddf4ca122176

Criteo writes about how they enforce data quality for the 450 PB that they have in their Hadoop clusters. Since Hadoop is very flexible/lenient in the data it supports, they run over 7,000 data quality checks per day using statistics from Hive tables and custom queries.

https://medium.com/criteo-labs/big-data-quality-at-criteo-66c6bd0d42d8

The morning paper covers Taiji, Facebook's system for routing and managing global traffic. Taiji takes advantage of knowledge about a user and their social connections to efficiently route traffic from the edge to a data center. By better taking advantage of caches, changes in routing materially reduce load on the database systems.

https://blog.acolyer.org/2019/11/15/facebook-taiji/

An in-depth look at how to the various Apache Kafka configurations for Producer and Consumers can be used to avoid duplicate data in your pipeline. This post describes the various types of delivery guarantees, how they map to Kafka settings, and discusses how to put them all together. The post has lots of great diagrams, and there's a bonus section on other strategies for deduplicating data in a stream.

https://medium.com/@andy.bryant/processing-guarantees-in-kafka-12dd2e30be0e

Events

Curated by Datadog ( http://www.datadog.com )

California

Apache Kafka and Immutable Infrastructure + An Introductory Kafka Talk! (Santa Monica) - Tuesday, November 19

https://www.meetup.com/Los-Angeles-Apache-Kafka-Meetup/events/265903100/

Learning How to Build-Event Streaming Applications with Pac-Man (San Diego) - Tuesday, November 19

https://www.meetup.com/San-Diego-Java-Users-Group/events/266167833/

Bay Area Apache Flink Meetup @ Cloudera (Palo Alto) - Wednesday, November 20

https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/266226960/

Building Data Lineage, Data Orchestration, and Data Mesh (Mountain View) - Thursday, November 21

https://www.meetup.com/SF-Big-Analytics/events/265730920/

Washington

The Rise of Apache Flink and Stream Processing (Bellevue) - Wednesday, November 20

https://www.meetup.com/Big-Data-Bellevue-BDB/events/262650434/

Texas

Foreign-Key Joins with Kafka Streams (Plano) - Thursday, November 21

https://www.meetup.com/Dallas-Kafka/events/265880083/

Wisconsin

Introduction to Apache Spark (Madison) - Monday, November 18

https://www.meetup.com/Women-in-Big-Data-Wisconsin-Chapter/events/265857508/

IRELAND

Data Science and Engineering Club @ Zalando (Dublin) - Thursday, November 21

https://www.meetup.com/Data-Science-and-Engineering-Club/events/265937165/

FRANCE

Fine-Tuning Kafka: Let's Look under the Hood! (Paris) - Wednesday, November 20

https://www.meetup.com/PerfUG/events/254607871/

BELGIUM

Apache Beam Intro and Use Cases (Antwerpen) - Friday, November 22

https://www.meetup.com/Belgium-Apache-Beam-Meetup/events/264933301/

DENMARK

Kafka Is More ACID Than Your Database (Kongens Lyngby) - Tuesday, November 19

https://www.meetup.com/Copenhagen-Javagruppen-Meetup/events/266441150/

SWITZERLAND

Apache Beam @ Ricardo.ch + Portable Schema + BeamSQL (Zurich) - Tuesday, November 19

https://www.meetup.com/Zurich-Apache-Beam-Meetup/events/265529665/

CZECH REPUBLIC

Apache Kafka as a Database (Brno-stred) - Wednesday, November 20

https://www.meetup.com/Brno-Java-Meetup/events/265476406/

SINGAPORE

Event-Driven Microservices with CQRS Using Axon: Excitingly Boring (Singapore) - Wednesday, November 20

https://www.meetup.com/singajug/events/266381397/

AUSTRALIA

Melbourne Data Engineering Meetup (Docklands) - Wednesday, November 20

https://www.meetup.com/Melbourne-Data-Engineering-Meetup/events/265892291/

Some great stuff in this week’s issue—Lyft’s open source metadata and data discovery platform, how Netflix uses GraphQL to build search indexes, an open source backup tool for Apache Cassandra, coverage of the Ceph distributed file systems evolution, and several other posts about Apache Airflow, Apache Spark, CockroachDB, event-driven microservices, and more!

Lyft writes about Amudsen, their open-source tool for metadata management and data discovery. The post covers the architecture of the system—the metadata, search, and frontend services as well as the databuilder (data ingestion framework). Since open sourcing earlier this year, they've had a number of contributions, such as one to make the datastore more extensible (supporting Apache Atlas in addition to Neo4j).

https://eng.lyft.com/open-sourcing-amundsen-a-data-discovery-and-metadata-platform-2282bb436234

A look at how to use Docker Swarm to scale out Apache Airflow using a custom Airflow operator.

https://medium.com/analytics-vidhya/orchestrating-airflow-tasks-with-docker-swarm-69b5fb2723a7

One data engineer's learnings after a year on the job. There are some good reflections on tools, automation (after a workflow of a certain size, a tool like Airflow is important), monitoring, and metadata/documentation housekeeping.

https://medium.com/@lohmengxin/one-year-as-a-data-engineer-d713b511f0d4

Netflix describes how they use GraphQL to build indexes ofrom data stored across multiple services. They key idea is to do issue a (batch) GraphQL query to return a full denormalized record (e.g. a show and its episodes, etc.) and store the results in Elasticsearch. Next, they can listen to changes on Kafka, and follow the relationships from the GraphQL schema to invalidate/reindex data.

https://medium.com/netflix-techblog/graphql-search-indexing-334c92e0d8d5

Medusa is a new open source backup tool for Apache Cassandra. It stores backup data in cloud storage (e.g. Amazon S3 or Google Cloud Storage), and it creates smart incremental backups by taking advantage of the immutable nature of SSTables. There's much more details about the tools features and how to use it in this post on The Last Pickle blog.

https://thelastpickle.com/blog/2019/11/05/cassandra-medusa-backup-tool-is-open-source.html

An in-depth look at the Apache Kafka consumer rebalance protocol. The post describes the pieces of the protocol like JoinGroup, SyncGroup, Heartbeat, and LeaveGroup. It also looks at the recent additions of static membership and incremental cooperative rebalancing. There are lots of great diagrams to illustrate the key concepts.

https://medium.com/streamthoughts/apache-kafka-rebalance-protocol-or-the-magic-behind-your-streams-applications-e94baf68e4f2

A look at how to detect skew in your Apache Spark jobs, and several ways to fix a job with skew (hints, randomizing the join key, writing a custom partitioner). Which solution is best/fastest depends a bit on the inputs to your job.

https://www.davidmcginnis.net/post/spark-job-optimization-dealing-with-data-skew

The morning paper covers a paper on Ceph, the open-source distributed file system. Over the past few years, Ceph implemented a new store that bypasses a filesystemto better take advantage of SSD and HDD disks. The post describes the motivation of the changes, some of the other options they explored (including rocksdb), and the performance improvements they see with the new storage backend.

https://blog.acolyer.org/2019/11/06/ceph-evolution/

Cockroach Labs writes about how they've sped up distributed transactions with parallel commits, which avoids certain round trips across the WAN. The post describes the solution, including how failure handling works, and it shows that experimentally latency is cut in half.

https://www.cockroachlabs.com/blog/parallel-commits

This post describes why you should consider a relational database and consider taking advantage of advanced features (such as triggers and stored procedures). The author motivates based on experience working with jupyter notebooks and comparing the complexity of a NoSQL database like Mongo or Elasticsearch.

https://tselai.com/modern-data-practice-and-the-sql-tradition.html

This article describes an event-driven architecture for maintaining a CRM and realtime database. An interesting component of the post details how to implement an audit system to ensure that all microservices consume the events. The basic idea is tag events with a unique ID and each microservice generates an audit event with that ID as it processes the event. Alerts are generated when (after aggregating based on a time window) the number of audit events for a particular ID is too low.

https://medium.com/@vladimir.elvov/why-event-driven-is-businesss-best-friend-982749561024

Events

Curated by Datadog ( http://www.datadog.com )

California

Data-Driven Development in Autonomous Driving + Spark Performance Tuning (Mountain View) - Tuesday, November 12

https://www.meetup.com/SF-Big-Analytics/events/265162755/

Data Engineering Meetup (San Diego) - Thursday, November 14

https://www.meetup.com/Data-Engineering-San-Diego/events/266078459/

Texas

Mirror Maker 2.0 (Austin) - Tuesday, November 12

https://www.meetup.com/Austin-Apache-Kafka-Meetup-Stream-Data-Platform/events/266214573/

Virginia

NOVA Data Engineering: First Meetup! (Herndon) - Thursday, November 14

https://www.meetup.com/NOVA-Data-Engineering/events/265820975/

BRAZIL

Data Meetup (Sao Carlos) - Wednesday, November 13

https://www.meetup.com/opensanca/events/265658143/

SWEDEN

Streaming Processing with Hazelcast Jet and Kafka (Stockholm) - Tuesday, November 12

https://www.meetup.com/Knock-Data-Stockholm/events/266041648/

SPAIN

Everything You Need to Know about Kafka Streams (A Coruna) - Thursday, November 14

https://www.meetup.com/CorunaJUG/events/266199913/

FRANCE

Airflow @ SchoolMouv: Build, Schedule, and Monitor Pipelines at Scale (Toulouse) - Wednesday, November 13

https://www.meetup.com/Toulouse-Data-Engineering/events/264518964/

CZECH REPUBLIC

Making Apache Spark Better with Delta Lake (Prague) - Thursday, November 14

https://www.meetup.com/CS-HUG/events/266200104/

POLAND

QA in Beam + Beam Use Case + More! (Warsaw) - Thursday, November 14

https://www.meetup.com/Warsaw-Apache-Beam-Meetup/events/265212437/

GREECE

Timeseries Forecasting as a Service + Run Spark and Flink Jobs on Kubernetes (Athens) - Thursday, November 14

https://www.meetup.com/Athens-Big-Data/events/265957761/

ISRAEL

Airflow Demystified + Big Data Demystified (Tel Aviv-Yafo) - Sunday, November 17

https://www.meetup.com/Big-Data-Demystified/events/263781635/

CHINA

Kafka Beijing Meetup (Beijing) - Saturday, November 16

https://www.meetup.com/Beijing-Kafka-Meetup/events/266223400/

AUSTRALIA

Kafka Is More ACID Than Your Database (Sydney) - Wednesday, November 13

https://www.meetup.com/apache-kafka-sydney/events/266137580/

K8s Meetup with Instaclustr & Google! (Pyrmont) - Wednesday, November 13

https://www.meetup.com/Sydney-Kubernetes-User-Group/events/265679988/

Sydney Data Engineering Meetup (Sydney) - Thursday, November 14

https://www.meetup.com/Sydney-Data-Engineering-Meetup/events/264743457/

Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.

Short and sweet issue this week, with several new open source tools—Beekeeper for cleaning up unused data, the Mantis project for real-time operations, and pg_flame's flame graphs for analyzing postgres queries—as well as implementation articles covering Apache Airflow, Rust for Kafka, and using bloom filters to optimize GDPR data deletion.

Beekeeper is a new open-source tool from Expedia for cleaning up unreferenced data in Apache Hive tables. It listens for metastore changes (like changes to table or partition locations) and periodically deletes abandoned data in S3. The github repo has instructions for trying it out using docker.

https://medium.com/expedia-group-tech/introducing-beekeeper-35ab8b770f23

Netflix has open sourced Mantis, its operations-focused event analysis system. Mantis aims to provide cost-effective, event-based analysis of a live system. It does so by allowing applications to publish lots of events but only incur costs of those events if there's a subscriber of the event stream. At Netflix, they use Mantis for things like monitoring video streaming health, contextual alerting, measuring Cassandra health, and alerting on log events. The Mantis github page has lots more details about the system, including an overview of its architecture and details on how to try it out.

https://medium.com/netflix-techblog/open-sourcing-mantis-a-platform-for-building-cost-effective-realtime-operations-focused-5b8ff387813a

https://netflix.github.io/mantis/

A look at how the Devoted Health team uses Apache Airflow. They deploy on Kubernetes (using Terraform and Helm), which allows developers to get their own instance of the stack for development/testing. They've built a tool for standardized DAG development (using YAML definitions) and a dev tool for synchronizing code changes to Kubernetes. The post also describes how they write integration tests and validations, deploy, and monitor their deployment.

https://adam.boscarino.me/posts/airflow-at-devoted-health/

Another post on Airflow deployments, Lyft writes about how they've implemented fine-grained secure access to the Apache Airflow UI. While Airflow has built-in RBAC, they built a custom security manager that adds DAG-level access permissions (defined alongside the DAG). At Lyft, each team has its own RBAC role and can decide which teams have access to the DAGs they publish.

https://eng.lyft.com/securing-apache-airflow-ui-with-dag-level-access-a7bc649a2821

A post on the Confluent blog describes porting a non-trivial Kafka application from Clojure to Rust. The author describes the tradeoffs between the various Rust libraries for Kafka, how to extend a client to support Avro records/Schema Registry, and shares some benchmarks comparing Clojure and Rust performance/memory usage.

https://www.confluent.io/blog/getting-started-with-rust-and-kafka

The Adobe Experience team writes about how they use bloom filters to speed up deletions of user data, for requests like those in GDPR. By maintaining a bloom filter for each partition of a data set, they can prune large swaths of data that definitely don't contain records for a particular user when deleting.

https://medium.com/adobetech/search-optimization-for-large-data-sets-for-gdpr-7c2f52d4ea1f

`pg_flame` looks like a handy tool for visualizing the output of postgres' `EXPLAIN ANALYZE`. Mousing over an entry in the flame graph reveals more details about that step of the `EXPLAIN`.

https://github.com/mgartner/pg_flame

Events

Curated by Datadog ( http://www.datadog.com )

California

Deploying Kafka SSL at Uber Scale + What’s the Time & Why? (Palo Alto) - Tuesday, October 29

https://www.meetup.com/KafkaBayArea/events/265692862/

Arizona

Planning for the Varied Levels of Once-Data-Processing with Kafka (Scottsdale) - Tuesday, October 29

https://www.meetup.com/Kafka-Phoenix/events/265328130/

BRAZIL

Everything You Want to Know about Apache Kafka, but Were Afraid to Ask (Sao Paulo) - Wednesday, October 30

https://www.meetup.com/Sao-Paulo-Apache-Kafka-Meetup-by-Confluent/events/265512308/

UNITED KINGDOM

Building Data Products in Production (London) - Tuesday, October 29

https://www.meetup.com/Data-something/events/265560240/

FRANCE

2 Years in Prod with a Hadoop Cluster (Villeurbanne) - Tuesday, October 29

https://www.meetup.com/lyonjug/events/265614417/

GERMANY

Kafka at Sky Deutschland + Kafka on Kubernetes  (Unterfohring) - Tuesday, October 29

https://www.meetup.com/Apache-Kafka-Germany-Munich/events/265114406/

POLAND

Kafka Connect (Torun) - Wednesday, October 30

https://www.meetup.com/Torun-JUG/events/265934291/

ROMANIA

Cloud Engineering at GoPro (Bucharest) - Tuesday, October 29

https://www.meetup.com/GoPro-Software-Development-in-Bucharest/events/265377428/

RUSSIA

KSQL (Saint Petersburg) - Monday, October 28

https://www.meetup.com/St-Petersburg-Kafka/events/265846058/

INDIA

Stream Processing with Apache Kafka, Flink, and Spark (Bengaluru) - Saturday, November 2

https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/265285812/

AUSTRALIA

ETL, Kafka, and Elasticsearch Combined! (Sydney) - Tuesday, October 29

https://www.meetup.com/Sydney-Alt-Net/events/264970509/